master: `gettxoutsetinfo` -> crash #7890

Current master (73fc922ed64333d45f18d8a448f30cfa2ae0281e) crashes at the end of processing RPC gettxoutsetinfo on OS X:

$ ./bitcoind 
Assertion failed: (ok), function Next, file txdb.cpp, line 139.
Abort trap: 6
$ 

Works ok on testnet. Hmm.

Before GetKey() keyTmp = (c, c9e3a8404fee8c8fe3406f86dec2989d7534a3b0c3548f139698a6f85affffff)!
GetKey() returned not OK, keyTmp = (d, c9e3a8404fee8c8fe3406f86dec2989d7534a3b0c3548f139698a6f85affffff)!

Notice the 'd'...

Keeping the copy of chainstate directory for later investigation and doing -reindex...

That's this code

    if (pcursor->Valid()) {
        bool ok = pcursor->GetKey(keyTmp);
        assert(ok); // If GetKey fails here something must be wrong with underlying database, we cannot handle that here
    } ...

I don't get it - why would a GetKey fail when the cursor is valid?

I extended dbwrapper.h's template<typename K> bool GetKey(K& key) { and the exception there is:

PJ: e = CDataStream::read(): end of data: unspecified iostream_category error

End of file?

I think the leveldb operation succeeds, but it is unable to deserialize the key:

    template<typename K> bool GetKey(K& key) {
        leveldb::Slice slKey = piter->key();
        try {
            CDataStream ssKey(slKey.data(), slKey.data() + slKey.size(), SER_DISK, CLIENT_VERSION);
            ssKey >> key;
        } catch (const std::exception&) {
            return false;
        }
        return true;
    }

Not sure why this would be the case but unless other people are experiencing this too, I'd say it's database corruption.

Can you put a LogPrintf("Failed key: %s\n", HexStr(slKey.data(), slKey.data() + slKey.size())); in the catch() of GetKey to see what key can't be parsed?

Yup. Once the reindex finishes. db.log is empty.

The db.log was transferred from the other system, where there were some lines about full disk space 8)

Edit: the chainstate (and other directories) was transferred from other machine. db.log from there is deleted. The file on the new machine is empty.

Will investigate further once the reindex locally is finished.

db.log is about bdb

I think the corruption in chainstate happened on the source machine.

Reindex helped to solve this issue for me.

I added @laanwj 's debug line and it is printed even on the start of bitcoind:

2016-04-17 05:48:23 Using obfuscation key for /Users/test/Library/Application Support/Bitcoin/chainstate: 0000000000000000
2016-04-17 05:48:25 Failed key: 6600000000

But it continues there. The crash happens in gettxoutsetinfo:

Failed key: 6400000003f13983c3a719c42b25eadf94446075b3

Closing, the real source of the problem was database corruption.

On second thought: is there a guarantee that keys always deserialize into the (char,uint256) stanza?

Failed key: 6600000000

'f' → DB_BLOCK_FILES

Failed key: 6400000003f13983c3a719c42b25eadf94446075b3

'd' → not listed

No, doesn't seem to be the case, see for example WriteBatchSync:

batch.Write(make_pair(DB_BLOCK_FILES, it->first), *it->second);

This serializes a char and integer into the key. So: GetKey can legitimately return false, it just means that we've reached a different key type. My interpretation of the meaning of the false return value in this comment was wrong:

// If GetKey fails here something must be wrong with underlying database, we cannot handle that here

Labeling this as bug!

I still have the chainstate contents. 3.3GB...

@paveljanik see #7904

#7094 was merged, this was solved