Improve crash debugging, suggestions needed

laanwj commented at 6:45 am on April 23, 2013: member

Once in a while there is someone with a unexplained crash in bitcoin-qt/bitcoind. This seems to happen especially on MacOSX and Windows.

Most we can do right now is ask for the debug.log, but most of the time it doesn’t help a lot. The OS-specific crashlogs that people send contain raw addresses, or at most function+offset. I’d like to improve the diagnosis ability. We can’t just ask people “to run it in gdb” anymore. I’d like to get as much as possible information without having to ask the user to do anything.

Things I’ve considered

I don’t think proposing something such as Google Breakpad would fly here, due to the obvious private key leak risk. It would in principle be possible to build something that just uploads a stack trace and debug.log and not any private key data. That would be very useful instead of asking for people to paste/upload the debug.log and stacktraces.
Including the debug information would increase the executable size enormously. However it would be possible to change the Gitian build (and Mac build) to store the debug information before stripping it off; this would allow us to at least resolve stack traces and such with main.o+0x2398 to a a line number at our side.

Diapolo commented at 8:13 am on April 23, 2013: none

I’m not sure what Pidgin is using, but they have a crash reporter for Windows AFAIK. Same for Firefox, do you know what they are using?

laanwj commented at 9:48 am on April 23, 2013: member

Yes – as i mentioned in the OP there are ready to use solutions for that but this is a more difficult problem than in their case because we don’t want to leak private keys and such.

ag346 commented at 5:02 pm on June 4, 2013: none

How about creating a beta version for volunteers like firefox does or allowing people to allow the program to optionally collect extra information.

jgarzik commented at 5:43 pm on June 4, 2013: contributor

@ag346 Not a bad idea, actually…

laanwj commented at 6:20 am on June 6, 2013: member

Yeah, not a bad idea.

But what I’d like here is a list of information that we can safely collect on a crash and that would help diagnostics without running the risk of collecting (parts of) private keys. As mentioned in the OP one of those is the stack trace, as long as only the function addresses and not the rest of the stack contents is dumped. But the register values would be off limits as they can contain parts of (or derived of) private keys.

Maybe that’s enough… but maybe some other things would be useful as well:

Current configuration (except RPC key and maybe others)
Last RPC commands (ok, this gets scary, as there may be importprivkeys and such)
Last NN bytes of debug.log
Statistics about memory usage and leveldb
…

ag346 commented at 2:11 pm on June 6, 2013: none

Maybe we could help brainstorm or improve the following idea:

One possible way to allow sharing of all information might be if a volunteer mode could be enabled. This would create an address that would be used for collecting and sharing test data only and would be limited in how it could be used. For example: An address is generated for the exclusive use of gathering and sharing diagnostic data Once the address is used for volunteering information, it can no longer be used for normal transactions The address can only send/receive small amounts of bitcoin. The address cannot accumulate more than xyz maximum amount of bitcoin The volunteer mode sends/receives transactions only to other volunteer addresses continuously and at random.

Some other possibilities: The addresses are “owned” by the developer team and loaned out to the volunteer computers. If the volunteer computer does not have any activity over the course of xyz days/months/etc. its address is returned to the developer team for reuse. To minimize the potential for abuse, the volunteer program is limited and requires registration/approval from a moderator.

I know this is not perfect, but maybe it’s a starting point for a general idea that could be improved with feedback from others?

laanwj commented at 6:15 am on July 10, 2013: member

If we do this, we should at least encrypt the crash dumps with the PGP keys of the developers.

neilneyman commented at 7:30 am on August 31, 2013: none

@ag346 it seems that the big issue might be the participant’s other actual private keys getting (all or partially) leaked in a crash report. This might be over-complicating things, but brainstorming here: could there be a suggestion or even some utility for the tester-client to immediately generate new addresses and spend all of the potentially-compromised coins to them? Could be optional but recommended upon the next startup. I don’t know if this would necessitate sending the crashdump on some sort of delay until the wallet is secured again. One glaring problem I see with this is transaction fees.

laanwj closed this on Oct 30, 2015

MarcoFalke locked this on Sep 8, 2021

Improve crash debugging, suggestions needed #2551