mutex debugging routines: LocksHeld() and AssertLockHeld() #3329

Inspired by a Mike Hearn comment on #3276 -- adds a couple of mutex-related routines:

AssertLockHeld("cs_foo") : if compiled -DDEBUG_LOCKORDER, walks the lock stack and makes sure the specified lock is held. If not, a message and the list of locks that ARE held are LogPrintf'ed and then assert(0) is called. Does nothing in normal builds.

LocksHeld() : -DDEBUG_LOCKORDER-only routine: returns a human-readable string with all of the locks held. This can be very handy to call from within an interactive debugging session (and it is used by AssertLockHeld).

I added AssertLockHeld("cs_main") to ProcessBlock as proof-of-concept; if we like this, adding AssertLockHeld calls to more routines can be done in subsequent pulls.

PS: compiling with -DDEBUG_LOCKORDER in the new autotools-world means a 'make clean' and then something like: CXXFLAGS='-DDEBUG_LOCKORDER' ./configure ...

ACK, sounds extremely useful for debugging.

Many of the wallet functions also assume that the caller holds the lock, would be good to add assertions for cs_wallet there.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/c649637b6ccb270bd0160163e6e92c1456774286 for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

Looks good to me!

Hmm, why are we replicating the functionality in clang's static analyser at runtime? Use the macros in threadsafety.h if we want to do this.

Static analysis is useful but limited, especially when doing whole program analysis. There's no harm in having both.