p2p-fullblocktest fails on BSD derivates #8605

On OpenBSD:

Initializing test directory /tmp/test8vty0436/25006
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11408
...
Unexpected exception caught during testing: error('cannot add item to database',)
  File "/home/user/bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 150, in main
    self.run_test()
  File "qa/rpc-tests/p2p-fullblocktest.py", line 73, in run_test
    self.test.run()
  File "/home/user/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 336, in run
    self.block_store.add_block(block)
  File "/home/user/bitcoin/qa/rpc-tests/test_framework/blockstore.py", line 80, in add_block
    self.blockDB[repr(block.sha256)] = bytes(block.serialize())
Stopping nodes

Could be a difference between ndb back-end libraries used on OpenBSD and Ubuntu. I vaguely remember some ndb variants can only handle fairly small databases.

On Ubuntu:

>>> import _dbm
>>> _dbm.library
'Berkeley DB'

On OpenBSD:

>>> import _dbm
>>> _dbm.library                                                                                                                                                                               
'GNU gdbm'

I believe even that is misreporting: OpenBSD does not install gdbm by default. This must be some ancient dbm variant. Just checked: It's taking the symbol from the libc(!) off all places: https://github.com/robertbachmann/openbsd-libc/tree/master/db

4.4BSD (and its derivatives such as NetBSD) contains Berkeley DB 1 in libc.

Ugh.

Anyhow, seems to work if I swap dbm.ndbm with dbm.dumb in test_framework/blockstore.py (be sure to replace all three occurences). This is not a good idea by default, as the performance of the dumb database is rumored to be abysmal (I haven't benchmarked though), but we could add a workaround specifically for OpenBSD.

diff --git a/qa/rpc-tests/test_framework/blockstore.py b/qa/rpc-tests/test_framework/blockstore.py
index 1e2bbb2..74cf125 100644
--- a/qa/rpc-tests/test_framework/blockstore.py
+++ b/qa/rpc-tests/test_framework/blockstore.py
@@ -9,11 +9,11 @@

 from .mininode import *
 from io import BytesIO
-import dbm.ndbm
+import dbm.dumb

 class BlockStore(object):
     def __init__(self, datadir):
-        self.blockDB = dbm.ndbm.open(datadir + "/blocks", 'c')
+        self.blockDB = dbm.dumb.open(datadir + "/blocks", 'c')
         self.currentBlock = 0
         self.headers_map = dict()

@@ -123,7 +123,7 @@ class BlockStore(object):

 class TxStore(object):
     def __init__(self, datadir):
-        self.txDB = dbm.ndbm.open(datadir + "/transactions", 'c')
+        self.txDB = dbm.dumb.open(datadir + "/transactions", 'c')

     def close(self):
         self.txDB.close()

Apparently this is a problem on other BSD derivatives as well, but slightly different: On FreeBSD and OSX, the file grows to a huge size (>100GB) fills the disk and the same error happens.

I confirm the issue on OSX (+64GB).