Assert inside leveldbwrapper to avoid continuing after failure. #5621

Database failure due to faulty hardware could leave users on a fork where they are vulnerable to opportunistic attack because they are rejecting the longest chain.

If there is a software fault in the OS or LevelDB that causes many nodes to experience database failure at once its also important for overall consensus stability that the failing nodes shut down and not continue operating.

In theory, the exceptions in leveldbwrapper should cause shutdowns but in practice the software calls into that database from many places, some of which throw away all exceptions.

A more organized approach to errors should be used in the long-term, but this works for the moment.

Tested by running on a faulty host that was reliably getting LevelDB checksum errors and continuing operation afterwards.

ut ACK

So, #5619 or this one? Or both?

Not both. This one will cause an assert before #5619 has a chance to catch anything anyway.

I'm testing 5619 on my faulty host. So far it successfully has failed one of one attempts, another one is ongoing. (5621 was at 3 of 3 shortly after opening the PR) Sorry, the host isn't fast.

I would accept 5619 if, after a more testing we're not surprised by any additional places where an exception here could escape. Though, ... I'm not sure we should be using exceptions for critical failures while we are still using them for normal conditions. Eliminating them from normal conditions would require a substantial rewrite of the network deserialization code, so asserting on the critical failures is the thing to do if you buy into my argument that we shouldn't be mixing them.

#5619 was merged.