bitcoind broken on latest master #17523

issue mb300sd openend this issue on November 19, 2019
  1. mb300sd commented at 8:14 pm on November 19, 2019: contributor

    Bitcoind takes up 100% cpu, log file not written to. Even bitcoind –help isn’t working.

    Expected behavior

    Run normally

    Actual behavior

    100% cpu usage, no output

    To reproduce

    Compile current master and try to run bitcoind

    System information

    latest master

    Debian Buster VM with 28 Xeon Platinum cores, 64GB RAM, SSD storage

    Compiled from v0.19.0.1 tag and it’s working.

  2. mb300sd added the label Bug on Nov 19, 2019
  3. fanquake added the label Linux/Unix on Nov 19, 2019
  4. MarcoFalke commented at 8:22 pm on November 19, 2019: member
    Can you start in gdb and see where it is stalling?
  5. mb300sd commented at 8:24 pm on November 19, 2019: contributor

    Just did. Not very familiar with gdb, but seem to have found it.

     0gdb --args ./bitcoind --help
 1GNU gdb (Debian 8.2.1-2+b3) 8.2.1
 2Copyright (C) 2018 Free Software Foundation, Inc.
 3License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 4This is free software: you are free to change and redistribute it.
 5There is NO WARRANTY, to the extent permitted by law.
 6Type "show copying" and "show warranty" for details.
 7This GDB was configured as "x86_64-linux-gnu".
 8Type "show configuration" for configuration details.
 9For bug reporting instructions, please see:
10<http://www.gnu.org/software/gdb/bugs/>.
11Find the GDB manual and other documentation resources online at:
12    <http://www.gnu.org/software/gdb/documentation/>.
13
14For help, type "help".
15Type "apropos word" to search for commands related to "word"...
16Reading symbols from ./bitcoind...done.
17(gdb) run
18Starting program: /root/src/bitcoin/src/bitcoind --help
19[Thread debugging using libthread_db enabled]
20Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
21
22^C
23Program received signal SIGINT, Interrupt.
240x000055555596962e in GetCPUID (d=<optimized out>, c=<optimized out>, b=<optimized out>, a=<optimized out>, subleaf=<optimized out>,
25    leaf=<optimized out>) at randomenv.cpp:202
26202                 AddCPUID(hasher, leaf, subleaf, ax, bx, cx, dx);
  6. fanquake commented at 8:25 pm on November 19, 2019: member
    That will be related to #17270.
  7. MarcoFalke added the label Utils/log/libs on Nov 19, 2019
  8. MarcoFalke added this to the milestone 0.20.0 on Nov 19, 2019
  9. practicalswift commented at 8:32 pm on November 19, 2019: contributor

    @mb300sd Thanks a lot for reporting!

    What is the output of git rev-parse HEAD?

  10. mb300sd commented at 8:32 pm on November 19, 2019: contributor
    b4a1da9ef8e4b673c290d5b882527e627ae1b43a
  11. fanquake commented at 8:36 pm on November 19, 2019: member
    @mb300sd Can you configure with --enable-debug and post a full backtrace?
  12. sipa commented at 8:37 pm on November 19, 2019: member

    @mb300sd What’s the output of cat /proc/cpuinfo?

    My guess is that somehow the iteration over the cpuid data does not terminate.

  13. mb300sd commented at 8:40 pm on November 19, 2019: contributor 
     0(gdb) run
 1Starting program: /root/src/bitcoin/src/bitcoind --help
 2[Thread debugging using libthread_db enabled]
 3Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 4^C
 5Program received signal SIGINT, Interrupt.
 6__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232
 7232     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
 8(gdb) bt
 9[#0](/bitcoin-bitcoin/0/)  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232
10[#1](/bitcoin-bitcoin/1/)  0x0000555555cdc47c in CSHA512::Write (this=0x7fffffffe330, data=0x7fffffffde2c "", len=4) at crypto/sha512.cpp:179
11[#2](/bitcoin-bitcoin/2/)  0x0000555555c63995 in (anonymous namespace)::operator<< <unsigned int> (hasher=..., data=@0x7fffffffde2c: 0) at randomenv.cpp:119
12[#3](/bitcoin-bitcoin/3/)  0x0000555555c62aaa in (anonymous namespace)::AddCPUID (hasher=..., leaf=4, subleaf=2372698, ax=@0x7fffffffde24: 1006649344, bx=@0x7fffffffde28: 0,
13    cx=@0x7fffffffde2c: 0, dx=@0x7fffffffde30: 0) at randomenv.cpp:191
14[#4](/bitcoin-bitcoin/4/)  0x0000555555c62b69 in (anonymous namespace)::AddAllCPUID (hasher=...) at randomenv.cpp:202
15[#5](/bitcoin-bitcoin/5/)  0x0000555555c632a7 in RandAddStaticEnv (hasher=...) at randomenv.cpp:359
16[#6](/bitcoin-bitcoin/6/)  0x0000555555c603fd in SeedStartup (hasher=..., rng=...) at random.cpp:510
17[#7](/bitcoin-bitcoin/7/)  0x0000555555c60580 in ProcRand (
18    out=0x555556108780 <(anonymous namespace)::signatureCache> "\213\067\366\023Ss(\204\206\031\362\376\341\231F\030\351\362\t\aa\337\016Y\372\352;\217@?\341n", num=32, level=RNGLevel::FAST) at random.cpp:547
19[#8](/bitcoin-bitcoin/8/)  0x0000555555c60602 in GetRandBytes (
20    buf=0x555556108780 <(anonymous namespace)::signatureCache> "\213\067\366\023Ss(\204\206\031\362\376\341\231F\030\351\362\t\aa\337\016Y\372\352;\217@?\341n", num=32) at random.cpp:552
21[#9](/bitcoin-bitcoin/9/)  0x00005555557f5db9 in (anonymous namespace)::CSignatureCache::CSignatureCache (this=0x555556108780 <(anonymous namespace)::signatureCache>)
22    at script/sigcache.cpp:34
23[#10](/bitcoin-bitcoin/10/) 0x00005555557f66eb in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at script/sigcache.cpp:67
24[#11](/bitcoin-bitcoin/11/) 0x00005555557f6742 in _GLOBAL__sub_I__Z18InitSignatureCachev () at script/sigcache.cpp:93
25[#12](/bitcoin-bitcoin/12/) 0x0000555555d6ab85 in __libc_csu_init ()
26[#13](/bitcoin-bitcoin/13/) 0x00007ffff73a102a in __libc_start_main (main=0x55555558da8c <main(int, char**)>, argc=2, argv=0x7fffffffe5e8,
27    init=0x555555d6ab40 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5d8) at ../csu/libc-start.c:264
28[#14](/bitcoin-bitcoin/14/) 0x000055555558cc9a in _start ()

     0processor       : 0
 1vendor_id       : GenuineIntel
 2cpu family      : 6
 3model           : 85
 4model name      : Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz
 5stepping        : 5
 6microcode       : 0xffffffff
 7cpu MHz         : 2200.000
 8cache size      : 39424 KB
 9physical id     : 0
10siblings        : 28
11core id         : 0
12cpu cores       : 14
13apicid          : 0
14initial apicid  : 0
15fpu             : yes
16fpu_exception   : yes
17cpuid level     : 21
18wp              : yes
19flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
20bugs            : spectre_v1 spectre_v2 spec_store_bypass mds swapgs taa itlb_multihit
21bogomips        : 4400.00
22clflush size    : 64
23cache_alignment : 64
24address sizes   : 44 bits physical, 48 bits virtual
25power management:
26
27......
  14. sipa commented at 8:42 pm on November 19, 2019: member

    @mb300sd Can you compile and run this program?

     0#include <stdio.h>
 1#include <cpuid.h>
 2#include <stdint.h>
 3
 4char static inline ch(uint32_t v, int b) {
 5    int c = (v >> (b * 8)) & 0xFF;
 6    if (c >= 32 && c<= 126) return c;
 7    return '.';
 8}
 9
10void static inline GetCPUID(uint32_t leaf, uint32_t subleaf, uint32_t& a, uint32_t& b, uint32_t& c, uint32_t& d)
11{
12#ifdef __GNUC__
13    __cpuid_count(leaf, subleaf, a, b, c, d);
14#else
15  __asm__ ("cpuid" : "=a"(a), "=b"(b), "=c"(c), "=d"(d) : "0"(leaf), "2"(subleaf));
16#endif
17    printf("%lx.%lx: %08lx %08lx %08lx %08lx %c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c\n", (unsigned long)leaf, (unsigned long)subleaf, (unsigned long)a, (unsigned long)b, (unsigned long)c, (unsigned long)d, ch(a, 0), ch(a, 1), ch(a, 2), ch(a, 3), ch(b, 0), ch(b, 1), ch(b, 2), ch(b, 3), ch(c, 0), ch(c, 1), ch(c, 2), ch(c, 3), ch(d, 0), ch(d, 1), ch(d, 2), ch(d, 3));
18}
19
20void static walk(void) {
21    uint32_t ax, bx, cx, dx;
22    GetCPUID(0, 0, ax, bx, cx, dx);
23    uint32_t max = ax;
24    for (uint32_t leaf = 1; leaf <= max; ++leaf) {
25        for (uint32_t subleaf = 0;; ++subleaf) {
26            GetCPUID(leaf, subleaf, ax, bx, cx, dx);
27            // Iterate over subleaves for leaf 4, 11, 13
28            if (leaf != 4 && leaf != 11 && leaf != 13) break;
29            if ((leaf == 4 || leaf == 13) && ax == 0) break;
30            if (leaf == 11 && (cx & 0xFF00) == 0) break;
31        }
32    }
33    GetCPUID(0x80000000, 0, ax, bx, cx, dx);
34    uint32_t maxext = ax;
35    for (uint32_t leaf = 0x80000001; leaf <= maxext; ++leaf) {
36        GetCPUID(leaf, 0, ax, bx, cx, dx);
37    }
38}
39
40int main(void) {
41    walk();
42    return 0;
43}

    g++ walkcpuid.cpp -o walkcpuid && ./walkcpuid

  15. mb300sd commented at 8:42 pm on November 19, 2019: contributor

    Different backtrace a second time.

     0[#0](/bitcoin-bitcoin/0/)  0x0000555555c626c7 in GetCPUID (leaf=4, subleaf=4710365, a=@0x7fffffffde24: 1006649344, b=@0x7fffffffde28: 0, c=@0x7fffffffde2c: 0,
 1    d=@0x7fffffffde30: 0) at ./compat/cpuid.h:17
 2[#1](/bitcoin-bitcoin/1/)  0x0000555555c62a4f in (anonymous namespace)::AddCPUID (hasher=..., leaf=4, subleaf=4710365, ax=@0x7fffffffde24: 1006649344, bx=@0x7fffffffde28: 0,
 3    cx=@0x7fffffffde2c: 0, dx=@0x7fffffffde30: 0) at randomenv.cpp:190
 4[#2](/bitcoin-bitcoin/2/)  0x0000555555c62b69 in (anonymous namespace)::AddAllCPUID (hasher=...) at randomenv.cpp:202
 5[#3](/bitcoin-bitcoin/3/)  0x0000555555c632a7 in RandAddStaticEnv (hasher=...) at randomenv.cpp:359
 6[#4](/bitcoin-bitcoin/4/)  0x0000555555c603fd in SeedStartup (hasher=..., rng=...) at random.cpp:510
 7[#5](/bitcoin-bitcoin/5/)  0x0000555555c60580 in ProcRand (
 8    out=0x555556108780 <(anonymous namespace)::signatureCache> "\020\275\r\223\267\364֠\240\301\251\225\253\365\252P\265\266YA\237Φ\306\370\205)2\243\066\16h", num=32, level=RNGLevel::FAST) at random.cpp:547
 9[#6](/bitcoin-bitcoin/6/)  0x0000555555c60602 in GetRandBytes (
10    buf=0x555556108780 <(anonymous namespace)::signatureCache> "\020\275\r\223\267\364֠\240\301\251\225\253\365\252P\265\266YA\237Φ\306\370\205)2\243\066\16h", num=32) at random.cpp:552
11[#7](/bitcoin-bitcoin/7/)  0x00005555557f5db9 in (anonymous namespace)::CSignatureCache::CSignatureCache (this=0x555556108780 <(anonymous namespace)::signatureCache>)
12    at script/sigcache.cpp:34
13[#8](/bitcoin-bitcoin/8/)  0x00005555557f66eb in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at script/sigcache.cpp:67
14[#9](/bitcoin-bitcoin/9/)  0x00005555557f6742 in _GLOBAL__sub_I__Z18InitSignatureCachev () at script/sigcache.cpp:93
15[#10](/bitcoin-bitcoin/10/) 0x0000555555d6ab85 in __libc_csu_init ()
16[#11](/bitcoin-bitcoin/11/) 0x00007ffff73a102a in __libc_start_main (main=0x55555558da8c <main(int, char**)>, argc=2, argv=0x7fffffffe5e8,
17    init=0x555555d6ab40 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5d8) at ../csu/libc-start.c:264
18[#12](/bitcoin-bitcoin/12/) 0x000055555558cc9a in _start ()
  16. mb300sd commented at 8:45 pm on November 19, 2019: contributor

    First couple lines

     00.0: 00000015 756e6547 6c65746e 49656e69 ....GenuntelineI
 11.0: 00050655 19200800 feda3203 1f8bfbff U..... ..2......
 22.0: 76036301 00f0b5ff 00000000 00c30000 .c.v............
 33.0: 00000000 00000000 00000000 00000000 ................
 44.0: 3c004121 01c0003f 0000003f 00000000 !A.<?...?.......
 54.1: 3c004122 01c0003f 0000003f 00000000 "A.<?...?.......
 64.2: 3c004143 03c0003f 000003ff 00000000 CA.<?...........
 74.3: 3c07c163 0280003f 0000dfff 00000000 c..<?...........
 84.4: 3c004000 00000000 00000000 00000000 .@.<............
 94.5: 3c004000 00000000 00000000 00000000 .@.<............
104.6: 3c004000 00000000 00000000 00000000 .@.<............
114.7: 3c004000 00000000 00000000 00000000 .@.<............
124.8: 3c004000 00000000 00000000 00000000 .@.<............
134.9: 3c004000 00000000 00000000 00000000 .@.<............
144.a: 3c004000 00000000 00000000 00000000 .@.<............
154.b: 3c004000 00000000 00000000 00000000 .@.<............
164.c: 3c004000 00000000 00000000 00000000 .@.<............
174.d: 3c004000 00000000 00000000 00000000 .@.<............
184.e: 3c004000 00000000 00000000 00000000 .@.<............
194.f: 3c004000 00000000 00000000 00000000 .@.<............

    and then it scrolls the last one forever.

    4.108cc5: 3c004000 00000000 00000000 00000000 .@.<............

  17. sipa commented at 8:46 pm on November 19, 2019: member
    @mb300sd Thanks, that’s it!
  18. sipa commented at 10:21 pm on November 19, 2019: member

    @mb300sd Can you try again with the following walk function?

     0void static walk(void) {
 1    uint32_t ax, bx, cx, dx;
 2    GetCPUID(0, 0, ax, bx, cx, dx);
 3    uint32_t max = ax;
 4    for (uint32_t leaf = 1; leaf <= max; ++leaf) {
 5        uint32_t maxsub = 0;
 6        for (uint32_t subleaf = 0;; ++subleaf) {
 7            GetCPUID(leaf, subleaf, ax, bx, cx, dx);
 8            if (leaf == 4) {
 9                if ((ax & 0x1f) == 0) break;
10            } else if (leaf == 7) {
11                if (subleaf == 0) maxsub = ax;
12                if (subleaf == maxsub) break;
13            } else if (leaf == 11) {
14                if ((cx & 0xff00) == 0) break;
15            } else if (leaf == 13) {
16                if (ax == 0 && bx == 0 && cx == 0 && dx == 0) break;
17            } else {
18                break;
19            }
20        }
21    }
22    GetCPUID(0x80000000, 0, ax, bx, cx, dx);
23    uint32_t maxext = ax;
24    for (uint32_t leaf = 0x80000001; leaf <= maxext; ++leaf) {
25        GetCPUID(leaf, 0, ax, bx, cx, dx);
26    }
27}
  19. mb300sd commented at 10:28 pm on November 19, 2019: contributor

    Looks good!

     00.0: 00000015 756e6547 6c65746e 49656e69 ....GenuntelineI
 11.0: 00050655 18200800 feda3203 1f8bfbff U..... ..2......
 22.0: 76036301 00f0b5ff 00000000 00c30000 .c.v............
 33.0: 00000000 00000000 00000000 00000000 ................
 44.0: 3c004121 01c0003f 0000003f 00000000 !A.<?...?.......
 54.1: 3c004122 01c0003f 0000003f 00000000 "A.<?...?.......
 64.2: 3c004143 03c0003f 000003ff 00000000 CA.<?...........
 74.3: 3c07c163 0280003f 0000dfff 00000000 c..<?...........
 84.4: 3c004000 00000000 00000000 00000000 .@.<............
 95.0: 00000000 00000000 00000000 00000000 ................
106.0: 00000000 00000000 00000000 00000000 ................
117.0: 00000000 d19f6fb9 00000000 bc000000 .....o..........
128.0: 00000000 00000000 00000000 00000000 ................
139.0: 00000000 00000000 00000000 00000000 ................
14a.0: 00000000 00000000 00000000 00000000 ................
15b.0: 00000001 00000002 00000100 00000018 ................
16b.1: 00000005 00000020 00000201 00000018 .... ...........
17b.2: 00000000 00000000 00000002 00000018 ................
18c.0: 00000000 00000000 00000000 00000000 ................
19d.0: 000000ff 00000a80 00000a80 00000000 ................
20d.1: 0000000f 00000a00 00000000 00000000 ................
21d.2: 00000100 00000240 00000000 00000000 ....@...........
22d.3: 00000040 000003c0 00000000 00000000 @...............
23d.4: 00000040 00000400 00000000 00000000 @...............
24d.5: 00000040 00000440 00000000 00000000 @...@...........
25d.6: 00000200 00000480 00000000 00000000 ................
26d.7: 00000400 00000680 00000000 00000000 ................
27d.8: 00000000 00000000 00000000 00000000 ................
28e.0: 00000000 00000000 00000000 00000000 ................
29f.0: 00000000 00000000 00000000 00000000 ................
3010.0: 00000000 00000000 00000000 00000000 ................
3111.0: 00000000 00000000 00000000 00000000 ................
3212.0: 00000000 00000000 00000000 00000000 ................
3313.0: 00000000 00000000 00000000 00000000 ................
3414.0: 00000000 00000000 00000000 00000000 ................
3515.0: 00000000 00000000 00000000 00000000 ................
3680000000.0: 80000008 00000000 00000000 00000000 ................
3780000001.0: 00000000 00000000 00000121 2c100800 ........!......,
3880000002.0: 65746e49 2952286c 6f655820 2952286e Intel(R) Xeon(R)
3980000003.0: 616c5020 756e6974 3238206d 204c3637  Platinum 8276L
4080000004.0: 20555043 2e322040 48473032 0000007a CPU @ 2.20GHz...
4180000005.0: 00000000 00000000 00000000 00000000 ................
4280000006.0: 00000000 00000000 01006040 00000000 ........@`......
4380000007.0: 00000000 00000000 00000000 00000000 ................
4480000008.0: 0000302c 00000000 00000000 00000000 ,0..............
  20. sipa commented at 11:07 pm on November 19, 2019: member
    @mb300sd See #17527.
  21. laanwj closed this on Nov 24, 2019
  22. sidhujag referenced this in commit 7f69c42e6f on Nov 24, 2019
  23. sidhujag referenced this in commit c14e5d9cf9 on Nov 10, 2020
  24. DrahtBot locked this on Dec 16, 2021
  25. PastaPastaPasta referenced this in commit 0d18eae6d0 on Apr 25, 2022


mb300sd MarcoFalke fanquake practicalswift sipa

Labels
Bug Linux/Unix Utils/log/libs

Milestone
0.20.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me