[qa] blockstore: Switch to dumb dbm #8834

pull MarcoFalke wants to merge 1 commits into bitcoin:master from MarcoFalke:Mf1610-qaBlockstoreDumb changing 1 files +3 −3
  1. MarcoFalke commented at 10:24 AM on September 29, 2016: member

    Closes #8605

    I could not see any performance degradation by this commit:

    $ time qa/pull-tester/rpc-tests.py p2p-fullblocktest

    Before:

    TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 297 s

ALL                  | True   | 297 s (accumulated)

Runtime: 297 s

real    4m57.633s
user    4m1.588s
sys 0m11.419s

    After:

    p2p-fullblocktest.py:
Pass: True, Duration: 288 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 288 s

ALL                  | True   | 288 s (accumulated)

Runtime: 288 s

real    4m48.573s
user    3m56.126s
sys 0m11.823s
  2. [qa] blockstore: Switch to dumb dbm fa9cd25ed0
  3. MarcoFalke added the label Tests on Sep 29, 2016
  4. MarcoFalke added the label Needs backport on Sep 29, 2016
  5. MarcoFalke added this to the milestone 0.13.1 on Sep 29, 2016
  6. laanwj commented at 11:02 AM on September 29, 2016: member

    Concept ACK, this is by far the most straightforward solution, if there is no need for platform-specific exceptions then let's avoid them.

  7. laanwj commented at 11:53 AM on September 29, 2016: member

    Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

    stderr:
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1be0>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/blocks.dir'
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1d68>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/transactions.dir'

    I can reproduce this locally on FreeBSD. Are perhaps the database files wiped before closing it? This only affects nulldummy.py not the other tests involving block stores.

  8. jonasschnelli commented at 12:05 PM on September 29, 2016: contributor

    Failed on my OSX machine:

    time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Initializing test directory /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11000
Test 1: PASS [1]
Test 2: PASS [100]
Test 3: PASS [101]
Test 4: PASS [102]
Test 5: PASS [102]
Test 6: PASS [103]
Test 7: PASS [103]
Test 8: PASS [104]
Test 9: PASS [104]
Test 10: PASS [104]
Test 11: PASS [104]
Test 12: PASS [104]
Test 13: PASS [104]
Test 14: PASS [104]
Test 15: PASS [104]
Test 16: PASS [105]
Test 17: PASS [106]
Test 18: PASS [106]
Test 19: PASS [106]
Test 20: PASS [106]
Test 21: PASS [106]
Test 22: PASS [106]
Test 23: PASS [106]
Test 24: PASS [106]
Test 25: PASS [107]
Test 26: PASS [107]
Test 27: PASS [107]
Test 28: PASS [107]
Test 29: PASS [107]
Test 30: PASS [107]
Test 31: PASS [107]
Test 32: PASS [108]
Test 33: PASS [109]
Test 34: PASS [109]
Test 35: PASS [110]
Test 36: PASS [110]
Test 37: PASS [111]
Test 38: PASS [111]
Test 39: PASS [111]
Test 40: PASS [111]
Test 41: PASS [112]
Test 42: PASS [112]
Test 43: PASS [113]
Test 44: PASS [113]
Test 45: PASS [114]
Test 46: PASS [115]
Test 47: PASS [115]
Test 48: PASS [115]
Test 49: PASS [115]
Test 50: PASS [115]
Test 51: PASS [115]
Test 52: PASS [115]
Test 53: PASS [115]
Test 54: PASS [115]
Test 55: PASS [115]
Test 56: PASS [115]
Test 57: PASS [116]
Test 58: PASS [116]
Test 59: PASS [116]
Test 60: PASS [117]
Test 61: PASS [117]
Test 62: PASS [117]
Test 63: PASS [117]
Test 64: PASS [118]
Test 65: PASS [118]
Test 66: PASS [118]
Test 67: PASS [118]
Test 68: PASS [118]
Test 69: PASS [119]
Test 70: PASS [120]
Test 71: PASS [120]
Test 72: PASS [120]
Test 73: PASS [120]
Test 74: PASS [121]
Test 75: PASS [121]
Test 76: PASS [121]
Test 77: PASS [122]
Test 78: PASS [122]
Test 79: PASS [122]
Test 80: PASS [123]
Test 81: PASS [124]
Test 82: PASS [125]
Test 83: PASS [126]
Test 84: PASS [127]
Test 85: PASS [127]
Test 86: PASS [127]
Test 87: PASS [128]
Test 88: PASS [129]
Test 89: PASS [130]
Test 90: PASS [130]
Test 91: PASS [131]
Test 92: PASS [131]
Test 93: PASS [132]
Test 94: PASS [132]
Test 95: PASS [1220]
Test 96: PASS [1220]
Test 97: PASS [1221]
Test 98: PASS [1221]
Unexpected exception caught during testing: ConnectionResetError(54, 'Connection reset by peer')
Stopping nodes
Not cleaning up dir /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
Failed

stderr:
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 151, in main
    self.run_test()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/p2p-fullblocktest.py", line 73, in run_test
    self.test.run()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in run
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in <listcomp>
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/coverage.py", line 49, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 148, in __call__
    response = self._request('POST', self.__url.path, postdata.encode('utf-8'))
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 125, in _request
    return self._get_response()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 163, in _get_response
    http_response = self.__conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)

Pass: False, Duration: 928 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | False  | 928 s

ALL                  | False  | 928 s (accumulated)

Runtime: 928 s

real    15m28.427s
user    14m47.171s
sys 0m31.176s

    Trying again....

  9. laanwj commented at 12:19 PM on September 29, 2016: member

    "Connection reset by peer" is a different issue, unrelated to the database. I have the same on FreeBSD. @jonasschnelli can you try:

    diff --git a/qa/rpc-tests/test_framework/authproxy.py b/qa/rpc-tests/test_framework/authproxy.py
index f5e0be2..fd7f32b 100644
--- a/qa/rpc-tests/test_framework/authproxy.py
+++ b/qa/rpc-tests/test_framework/authproxy.py
@@ -130,8 +130,9 @@ class AuthServiceProxy(object):
                 return self._get_response()
             else:
                 raise
-        except BrokenPipeError:
-            # Python 3.5+ raises this instead of BadStatusLine when the connection was reset
+        except (BrokenPipeError,ConnectionResetError):
+            # Python 3.5+ raises BrokenPipeError instead of BadStatusLine when the connection was reset
+            # ConnectionResetError happens on FreeBSD with Python 3.4
             self.__conn.close()
             self.__conn.request(method, path, postdata, headers)
             return self._get_response()
  10. jonasschnelli commented at 12:53 PM on September 29, 2016: contributor

    Passed after adding ConnectionResetError.

    time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Pass: True, Duration: 969 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 969 s

ALL                  | True   | 969 s (accumulated)

Runtime: 969 s

real    16m9.820s
user    15m22.178s
sys 0m34.593s
  11. MarcoFalke commented at 1:37 PM on September 29, 2016: member

    Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

    Unrelated to this pull, but needs travis retriggered after #8835

  12. laanwj commented at 2:46 PM on September 29, 2016: member

    Unrelated to this pull, but needs travis retriggered after #8835

    Done

  13. laanwj referenced this in commit 1d28faf9e9 on Sep 29, 2016
  15. laanwj merged this on Sep 29, 2016
  16. laanwj closed this on Sep 29, 2016
  17. laanwj referenced this in commit c9d7b0de2f on Sep 29, 2016
  18. MarcoFalke deleted the branch on Sep 29, 2016
  19. MarcoFalke referenced this in commit 1dd1783873 on Oct 3, 2016
  20. MarcoFalke referenced this in commit 31ab2f862a on Oct 3, 2016
  21. MarcoFalke removed the label Needs backport on Oct 13, 2016
  22. str4d referenced this in commit e93a3e1945 on Mar 24, 2017
  23. dagurval referenced this in commit 621894d14a on Aug 28, 2017
  24. dagurval referenced this in commit 7e2843856c on Aug 28, 2017
  25. dagurval referenced this in commit 4e3c29e1cb on Aug 29, 2017
  26. dagurval referenced this in commit 9b7555e2ef on Sep 18, 2017
  27. codablock referenced this in commit 9bfa39e58c on Sep 19, 2017
  28. OlegGirko referenced this in commit 6f55396f8e on Nov 7, 2017
  29. OlegGirko referenced this in commit c50ed3f6f0 on Nov 7, 2017
  30. codablock referenced this in commit e6af218659 on Jan 12, 2018
  31. andvgal referenced this in commit c91e199110 on Jan 6, 2019
  32. MarcoFalke locked this on Sep 8, 2021
Contributors
MarcoFalkelaanwjjonasschnelli
Labels
Tests

Milestone
0.13.1

Linked (view graph)
#8605 p2p-fullblocktest fails on BSD derivates#8839 test: Avoid ConnectionResetErrors during RPC tests
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 18:15 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me