Fix OOM when deserializing UTXO entries with invalid length #7933

Thanks to @pstratem for finding this.

The normal vector deserializer reads data in chunks of at most 5 MB, preventing OOM when insane vector lengths are encoded. This protection is not present in CScriptCompressor's specialized deserializer, however, resulting in a potential OOM when very large length descriptors exist, as the target CScript is resized before attempting to read that much data.

However, CScripts have a maximum length above which they're always invalid. We can treat scriptPubKeys with such lengths as unspendable, preventing them from going into the UTXO set even, and skipping them when deserializing.

Note that none of this is exposed to the network, as the P2P code uses normal (pre)vectors, which do have this OOM protection directly in serialize.h.

I think this needs a test (that fails without this, and succeeds with it). Or are we talking about 'insane lengths' of such magnitude that would result in a very long running test case?

The test (without fix) would use 2 GB+ RAM, and segfault. With fix, it will work fine. I'll add one.

The test (without fix) would use 2 GB+ RAM, and segfault.

Maybe we can cap it off before that happens? In any case a test that OOMs and crashes without this, but runs quickly with it, would be great too, it effectively prevents regression.

Added a test, and included #7936 (the test fails without).

utACK https://github.com/bitcoin/bitcoin/pull/7933/commits/1e44169f0e0c334a86b14a22ebc8fec45cec7354

Can confirm the test fails:

$ src/test/test_bitcoin -t coins_tests
Running 3 test cases...
unknown location(0): fatal error in "ccoins_serialization": memory access violation at address: 0x...: no mapping at fault address
test/coins_tests.cpp(411): last checkpoint

*** 1 failure detected in test suite "Bitcoin Test Suite"

utACK 1e44169