bitcoind hits corrupted double-linked list error when running multiple wallet_multiwallet.py tests in parallel #12362

issue jnewbery openend this issue on February 6, 2018
  1. jnewbery commented at 3:52 am on February 6, 2018: member

    I’m still debugging this, but I’m sharing what I’ve got so far.

    If I run several instances of the wallet_multiwallet.py test in parallel, then bitcoind hits an error, but the test still passes:

      0./test_runner.py wallet_multiwallet.py  wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py
  1...*** Error in `/home/ubuntu/bitcoin/src/bitcoind': corrupted double-linked list: 0x0000562c19b928a0 ***
  2======= Backtrace: =========
  3/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f750c3547e5]
  4/lib/x86_64-linux-gnu/libc.so.6(+0x82970)[0x7f750c35f970]
  5/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f750c361184]
  6/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7f750cc53e78]
  7/home/ubuntu/bitcoin/src/bitcoind(+0x772b9)[0x562c179e92b9]
  8/home/ubuntu/bitcoin/src/bitcoind(+0x2b8187)[0x562c17c2a187]
  9/home/ubuntu/bitcoin/src/bitcoind(+0x2aeb36)[0x562c17c20b36]
 10/home/ubuntu/bitcoin/src/bitcoind(+0x68b2c)[0x562c179dab2c]
 11/home/ubuntu/bitcoin/src/bitcoind(+0x3f2db)[0x562c179b12db]
 12/home/ubuntu/bitcoin/src/bitcoind(+0x3222f)[0x562c179a422f]
 13/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f750c2fd830]
 14/home/ubuntu/bitcoin/src/bitcoind(+0x3def9)[0x562c179afef9]
 15======= Memory map: ========
 16562c17972000-562c17f23000 r-xp 00000000 08:01 516003                     /home/ubuntu/bitcoin/src/bitcoind
 17562c18123000-562c1812e000 r--p 005b1000 08:01 516003                     /home/ubuntu/bitcoin/src/bitcoind
 18562c1812e000-562c18136000 rw-p 005bc000 08:01 516003                     /home/ubuntu/bitcoin/src/bitcoind
 19562c18136000-562c1813c000 rw-p 00000000 00:00 0 
 20562c19a1d000-562c19baf000 rw-p 00000000 00:00 0                          [heap]
 217f74f8000000-7f74f8023000 rw-p 00000000 00:00 0 
 227f74f8023000-7f74fc000000 ---p 00000000 00:00 0 
 237f7500000000-7f7500021000 rw-p 00000000 00:00 0 
 247f7500021000-7f7504000000 ---p 00000000 00:00 0 
 257f750613f000-7f7506140000 ---p 00000000 00:00 0 
 267f7506140000-7f7506940000 rw-p 00000000 00:00 0 
 277f7506940000-7f7506941000 ---p 00000000 00:00 0 
 287f7506941000-7f7507141000 rw-p 00000000 00:00 0 
 297f7507141000-7f7507142000 ---p 00000000 00:00 0 
 307f7507142000-7f7507942000 rw-p 00000000 00:00 0 
 317f7507942000-7f7507943000 ---p 00000000 00:00 0 
 327f7507943000-7f7508143000 rw-p 00000000 00:00 0 
 337f7508143000-7f7508144000 ---p 00000000 00:00 0 
 347f7508144000-7f7508944000 rw-p 00000000 00:00 0 
 357f7508944000-7f7508945000 ---p 00000000 00:00 0 
 367f7508945000-7f7509145000 rw-p 00000000 00:00 0 
 377f7509145000-7f7509146000 ---p 00000000 00:00 0 
 387f7509146000-7f750b948000 rw-p 00000000 00:00 0 
 397f750ba49000-7f750baa6000 r-xp 00000000 08:01 61325                      /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
 407f750baa6000-7f750bca5000 ---p 0005d000 08:01 61325                      /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
 417f750bca5000-7f750bca6000 r--p 0005c000 08:01 61325                      /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
 427f750bca6000-7f750bca7000 rw-p 0005d000 08:01 61325                      /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
 437f750bca7000-7f750bcd0000 r-xp 00000000 08:01 61253                      /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
 447f750bcd0000-7f750becf000 ---p 00029000 08:01 61253                      /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
 457f750becf000-7f750bed0000 r--p 00028000 08:01 61253                      /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
 467f750bed0000-7f750bed1000 rw-p 00029000 08:01 61253                      /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
 477f750bed1000-7f750bed4000 r-xp 00000000 08:01 63277                      /lib/x86_64-linux-gnu/libdl-2.23.so
 487f750bed4000-7f750c0d3000 ---p 00003000 08:01 63277                      /lib/x86_64-linux-gnu/libdl-2.23.so
 497f750c0d3000-7f750c0d4000 r--p 00002000 08:01 63277                      /lib/x86_64-linux-gnu/libdl-2.23.so
 507f750c0d4000-7f750c0d5000 rw-p 00003000 08:01 63277                      /lib/x86_64-linux-gnu/libdl-2.23.so
 517f750c0d5000-7f750c0dc000 r-xp 00000000 08:01 63311                      /lib/x86_64-linux-gnu/librt-2.23.so
 527f750c0dc000-7f750c2db000 ---p 00007000 08:01 63311                      /lib/x86_64-linux-gnu/librt-2.23.so
 537f750c2db000-7f750c2dc000 r--p 00006000 08:01 63311                      /lib/x86_64-linux-gnu/librt-2.23.so
 547f750c2dc000-7f750c2dd000 rw-p 00007000 08:01 63311                      /lib/x86_64-linux-gnu/librt-2.23.so
 557f750c2dd000-7f750c49d000 r-xp 00000000 08:01 63272                      /lib/x86_64-linux-gnu/libc-2.23.so
 567f750c49d000-7f750c69d000 ---p 001c0000 08:01 63272                      /lib/x86_64-linux-gnu/libc-2.23.so
 577f750c69d000-7f750c6a1000 r--p 001c0000 08:01 63272                      /lib/x86_64-linux-gnu/libc-2.23.so
 587f750c6a1000-7f750c6a3000 rw-p 001c4000 08:01 63272                      /lib/x86_64-linux-gnu/libc-2.23.so
 597f750c6a3000-7f750c6a7000 rw-p 00000000 00:00 0 
 607f750c6a7000-7f750c6bd000 r-xp 00000000 08:01 2039                       /lib/x86_64-linux-gnu/libgcc_s.so.1
 617f750c6bd000-7f750c8bc000 ---p 00016000 08:01 2039                       /lib/x86_64-linux-gnu/libgcc_s.so.1
 627f750c8bc000-7f750c8bd000 rw-p 00015000 08:01 2039                       /lib/x86_64-linux-gnu/libgcc_s.so.1
 637f750c8bd000-7f750c9c5000 r-xp 00000000 08:01 18864                      /lib/x86_64-linux-gnu/libm-2.23.so
 647f750c9c5000-7f750cbc4000 ---p 00108000 08:01 18864                      /lib/x86_64-linux-gnu/libm-2.23.so
 657f750cbc4000-7f750cbc5000 r--p 00107000 08:01 18864                      /lib/x86_64-linux-gnu/libm-2.23.so
 667f750cbc5000-7f750cbc6000 rw-p 00108000 08:01 18864                      /lib/x86_64-linux-gnu/libm-2.23.so
 677f750cbc6000-7f750cd38000 r-xp 00000000 08:01 8173                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 687f750cd38000-7f750cf38000 ---p 00172000 08:01 8173                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 697f750cf38000-7f750cf42000 r--p 00172000 08:01 8173                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 707f750cf42000-7f750cf44000 rw-p 0017c000 08:01 8173                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 717f750cf44000-7f750cf48000 rw-p 00000000 00:00 0 
 727f750cf48000-7f750cfaa000 r-xp 00000000 08:01 62885                      /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
 737f750cfaa000-7f750d1a9000 ---p 00062000 08:01 62885                      /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
 747f750d1a9000-7f750d1ad000 r--p 00061000 08:01 62885                      /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
 757f750d1ad000-7f750d1ae000 rw-p 00065000 08:01 62885                      /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
 767f750d1ae000-7f750d1f2000 r-xp 00000000 08:01 26071                      /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
 777f750d1f2000-7f750d3f2000 ---p 00044000 08:01 26071                      /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
 787f750d3f2000-7f750d3f3000 r--p 00044000 08:01 26071                      /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
 797f750d3f3000-7f750d3f4000 rw-p 00045000 08:01 26071                      /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
 807f750d3f4000-7f750d3f6000 r-xp 00000000 08:01 61265                      /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
 817f750d3f6000-7f750d5f5000 ---p 00002000 08:01 61265                      /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
 827f750d5f5000-7f750d5f6000 r--p 00001000 08:01 61265                      /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
 837f750d5f6000-7f750d5f7000 rw-p 00002000 08:01 61265                      /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
 847f750d5f7000-7f750d811000 r-xp 00000000 08:01 28216                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 857f750d811000-7f750da10000 ---p 0021a000 08:01 28216                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 867f750da10000-7f750da2c000 r--p 00219000 08:01 28216                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 877f750da2c000-7f750da38000 rw-p 00235000 08:01 28216                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 887f750da38000-7f750da3b000 rw-p 00000000 00:00 0 
 897f750da3b000-7f750da53000 r-xp 00000000 08:01 63271                      /lib/x86_64-linux-gnu/libpthread-2.23.so
 907f750da53000-7f750dc52000 ---p 00018000 08:01 63271                      /lib/x86_64-linux-gnu/libpthread-2.23.so
 917f750dc52000-7f750dc53000 r--p 00017000 08:01 63271                      /lib/x86_64-linux-gnu/libpthread-2.23.so
 927f750dc53000-7f750dc54000 rw-p 00018000 08:01 63271                      /lib/x86_64-linux-gnu/libpthread-2.23.so
 937f750dc54000-7f750dc58000 rw-p 00000000 00:00 0 
 947f750dc58000-7f750ddf2000 r-xp 00000000 08:01 552553                     /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
 957f750ddf2000-7f750dff2000 ---p 0019a000 08:01 552553                     /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
 967f750dff2000-7f750dff5000 r--p 0019a000 08:01 552553                     /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
 977f750dff5000-7f750dff8000 rw-p 0019d000 08:01 552553                     /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
 987f750dff8000-7f750dffe000 r-xp 00000000 08:01 60749                      /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
 997f750dffe000-7f750e1fe000 ---p 00006000 08:01 60749                      /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
1007f750e1fe000-7f750e1ff000 r--p 00006000 08:01 60749                      /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
1017f750e1ff000-7f750e200000 rw-p 00007000 08:01 60749                      /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
1027f750e200000-7f750e224000 r-xp 00000000 08:01 60765                      /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
1037f750e224000-7f750e423000 ---p 00024000 08:01 60765                      /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
1047f750e423000-7f750e425000 r--p 00023000 08:01 60765                      /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
1057f750e425000-7f750e426000 rw-p 00025000 08:01 60765                      /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
1067f750e426000-7f750e4a0000 r-xp 00000000 08:01 60742                      /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
1077f750e4a0000-7f750e6a0000 ---p 0007a000 08:01 60742                      /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
1087f750e6a0000-7f750e6a3000 r--p 0007a000 08:01 60742                      /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
1097f750e6a3000-7f750e6a4000 rw-p 0007d000 08:01 60742                      /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
1107f750e6a4000-7f750e6bb000 r-xp 00000000 08:01 60843                      /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
1117f750e6bb000-7f750e8ba000 ---p 00017000 08:01 60843                      /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
1127f750e8ba000-7f750e8bb000 r--p 00016000 08:01 60843                      /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
1137f750e8bb000-7f750e8bc000 rw-p 00017000 08:01 60843                      /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
1147f750e8bc000-7f750e8bf000 r-xp 00000000 08:01 60743                      /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
1157f750e8bf000-7f750eabe000 ---p 00003000 08:01 60743                      /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
1167f750eabe000-7f750eabf000 r--p 00002000 08:01 60743                      /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
1177f750eabf000-7f750eac0000 rw-p 00003000 08:01 60743                      /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
1187f750eac0000-7f750eae6000 r-xp 00000000 08:01 19651                      /lib/x86_64-linux-gnu/ld-2.23.so
1197f750eb1f000-7f750eb2f000 rw-p 00000000 00:00 0 
1207f750eb2f000-7f750ecc7000 r--p 00000000 08:01 7864                       /usr/lib/locale/locale-archive
1217f750ecc7000-7f750ecd6000 rw-p 00000000 00:00 0 
1227f750ecdd000-7f750ecde000 rw-p 00000000 00:00 0 
1237f750ecde000-7f750ece5000 r--s 00000000 08:01 63406                      /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
1247f750ece5000-7f750ece6000 r--p 00025000 08:01 19651                      /lib/x86_64-linux-gnu/ld-2.23.so
1257f750ece6000-7f750ece7000 rw-p 00026000 08:01 19651                      /lib/x86_64-linux-gnu/ld-2.23.so
1267f750ece7000-7f750ece8000 rw-p 00000000 00:00 0 
1277ffd2c413000-7ffd2c434000 rw-p 00000000 00:00 0                          [stack]
1287ffd2c475000-7ffd2c477000 r--p 00000000 00:00 0                          [vvar]
1297ffd2c477000-7ffd2c479000 r-xp 00000000 00:00 0                          [vdso]
130ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
131wallet_multiwallet.py passed, Duration: 5 s
132[...]

    I can only see this output if I run through the test_runner, not if I run the test directly (this might just be something to do with where the test_framework is sending stderr).

    I’ve bisected this to somewhere between commit 886a92f25f2e47a7ec9097d7c5a8d8a6173ef998 and 9bb59cf7baea0b33d97ef0c0eeee88a8b104be3a . Note that prior to 90600bc, this test was called multiwallet.py.

    This test is testing a lot of cases of bitcoind failing to start because of bad -wallet parameters being passed in, so the test is expecting the process to exit. However, we obviously shouldn’t be hitting memory corruption like I see in the output above.

    No-one else here has been able to reproduce this yet, but I can hit it reliably on master and v0.16.0rc2. One guess of somewhere to look is bdb since this is being tickled by the wallet code.

  2. jnewbery renamed this:
    bitcoind hits `corrupted double-linked list when running multiple `wallet_multiwallet.py` tests in parallel
    bitcoind hits corrupted double-linked list error when running multiple wallet_multiwallet.py tests in parallel
    on Feb 6, 2018
  3. fanquake added the label Tests on Feb 6, 2018
  4. jnewbery commented at 2:33 pm on February 6, 2018: member
    I’ve narrowed this down to commit 3448907a68e0f719be3bf82f4d321a30975d3539 (behaviour not seen in 7936446268e2f63baa855d694df0243924b7e1a9).
  5. ryanofsky commented at 3:17 pm on February 6, 2018: member
    It may be worth using addr2line to decode the backtrace, e.g. #9683 (comment)
  6. MarcoFalke added this to the milestone 0.16.0 on Feb 6, 2018
  7. MarcoFalke removed the label Tests on Feb 6, 2018
  8. jnewbery commented at 4:10 pm on February 6, 2018: member

    If I drop commit https://github.com/bitcoin/bitcoin/commit/3448907a68e0f719be3bf82f4d321a30975d3539 I no longer see this error.

    addr2line shows:

     0→ addr2line -f -e ../../src/bitcoind 0x773e9 0x2b9247 0x2afbf6 0x68c5c 0x3f3fb 0x3234f 0x3e019
 1_ZN9__gnu_cxx13new_allocatorIN5boost10filesystem4pathEE8allocateEmPKv
 2/usr/include/c++/5/ext/new_allocator.h:104
 3_ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE9push_backEOS5_
 4/usr/include/c++/5/bits/stl_vector.h:932
 5_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEaSEOS4_
 6/usr/include/c++/5/bits/basic_string.h:589
 7_Z8Shutdownv
 8/home/ubuntu/bitcoin/src/init.cpp:278 (discriminator 4)
 9_Z7AppInitiPPc
10/home/ubuntu/bitcoin/src/bitcoind.cpp:176
11main
12/home/ubuntu/bitcoin/src/bitcoind.cpp:186
13_start
14??:?
  9. laanwj commented at 4:26 pm on February 6, 2018: member

    /home/ubuntu/bitcoin/src/init.cpp:278 (discriminator 4)

    I don’t get it - is this the LogPrintf("%s: done\n", __func__); line?!

  10. TheBlueMatt commented at 4:28 pm on February 6, 2018: member
    Its memory corruption, that just is where libc is finding out about the corruption, not where its happening.
  11. jnewbery commented at 4:40 pm on February 6, 2018: member

    I’ve added print calls to the test case to narrow down where this is happening. So far I’ve observed it happening on the following lines:

    • self.assert_start_raises_init_error(0, ['-wallet=w11'], 'Error loading wallet w11. -wallet filename must be a regular file.')
    • self.assert_start_raises_init_error(0, ['-walletdir=wallets'], 'Error: Specified -walletdir "wallets" is a relative path', cwd=data_dir())
    • self.assert_start_raises_init_error(0, ['-walletdir=' + not_a_dir], 'Error: Specified -walletdir "' + not_a_dir + '" is not a directory')
    • self.assert_start_raises_init_error(0, ['-wallet=w1', '-wallet=w1'], 'Error loading wallet w1. Duplicate -wallet filename specified.')

    It seems like many (all?) of the failures in VerifyWallets() can trigger this

  12. laanwj commented at 4:48 pm on February 6, 2018: member
    Hm, so (from those) it looks like a shutdown race in case of early termination due to a wallet parameter related error.
  13. jnewbery commented at 5:41 pm on February 6, 2018: member

    @TheBlueMatt has provided the following patch. I don’t hit the error when I run with it:

     0diff --git a/src/httpserver.cpp b/src/httpserver.cpp
 1index a022d220e..3faafad1e 100644
 2--- a/src/httpserver.cpp
 3+++ b/src/httpserver.cpp
 4@@ -306,11 +306,13 @@ static void http_reject_request_cb(struct evhttp_request* req, void*)
 5     evhttp_send_error(req, HTTP_SERVUNAVAIL, nullptr);
 6 }
 7 
 8+std::atomic<int> threads_running(0);
 9 /** Event dispatcher thread */
10 static bool ThreadHTTP(struct event_base* base, struct evhttp* http)
11 {
12     RenameThread("bitcoin-http");
13     LogPrint(BCLog::HTTP, "Entering http event loop\n");
14+    threads_running++;
15     event_base_dispatch(base);
16     // Event loop will be interrupted by InterruptHTTPServer()
17     LogPrint(BCLog::HTTP, "Exited http event loop\n");
18@@ -358,6 +360,7 @@ static bool HTTPBindAddresses(struct evhttp* http)
19 /** Simple wrapper to set thread name and run work queue */
20 static void HTTPWorkQueueRun(WorkQueue<HTTPClosure>* queue)
21 {
22+    threads_running++;
23     RenameThread("bitcoin-httpworker");
24     queue->Run();
25 }
26@@ -468,6 +471,8 @@ bool StartHTTPServer()
27 
28 void InterruptHTTPServer()
29 {
30+    int rpcThreads = std::max((long)gArgs.GetArg("-rpcthreads", DEFAULT_HTTP_THREADS), 1L);
31+    while (threads_running != rpcThreads + 1) continue;
32     LogPrint(BCLog::HTTP, "Interrupting HTTP server\n");
33     if (eventHTTP) {
34         // Unlisten sockets
  14. laanwj referenced this in commit 2fefa3077e on Feb 6, 2018
  15. laanwj referenced this in commit b1c2370dde on Feb 6, 2018
  16. laanwj closed this on Feb 8, 2018
  17. laanwj referenced this in commit 6db4fa7ad3 on Feb 8, 2018
  18. laanwj referenced this in commit dd346cb262 on Feb 8, 2018
  19. Willtech referenced this in commit 934384fcd0 on Feb 11, 2018
  20. hkjn referenced this in commit b74db71329 on Feb 12, 2018
  21. HashUnlimited referenced this in commit 6219f45d29 on Mar 16, 2018
  22. laanwj referenced this in commit 612ba35ab1 on May 9, 2018
  23. ccebrecos referenced this in commit 05abb648f1 on Sep 14, 2018
  24. lionello referenced this in commit e0225b8c74 on Nov 7, 2018
  25. lateminer referenced this in commit afe2c55f1c on Jan 22, 2019
  26. codablock referenced this in commit ebf671aa76 on Oct 14, 2019
  27. codablock referenced this in commit 00cbfac3c8 on Oct 14, 2019
  28. barrystyle referenced this in commit f2fe2b3ecb on Jan 22, 2020
  29. furszy referenced this in commit 98eb626d6a on Feb 16, 2021
  30. furszy referenced this in commit 9c593ae9dc on Feb 16, 2021
  31. furszy referenced this in commit 950a849281 on Feb 16, 2021
  32. furszy referenced this in commit b8f7364917 on Feb 18, 2021
  33. TheArbitrator referenced this in commit 92bb93cef0 on Jun 7, 2021
  34. UdjinM6 referenced this in commit a57a82be1a on Jun 18, 2021
  35. UdjinM6 referenced this in commit 9e9cd7c035 on Jun 24, 2021
  36. UdjinM6 referenced this in commit 37b4656d23 on Jun 26, 2021
  37. UdjinM6 referenced this in commit 18e8ca4fa1 on Jun 26, 2021
  38. UdjinM6 referenced this in commit cc38451955 on Jun 28, 2021
  39. DrahtBot locked this on Sep 8, 2021


jnewbery ryanofsky laanwj TheBlueMatt

Milestone
0.16.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-04 22:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me