bitcoind broken on latest master #17523

1. 
   mb300sd commented at 8:14 PM on November 19, 2019: contributor

   <!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

   <!-- Describe the issue -->

   Bitcoind takes up 100% cpu, log file not written to. Even bitcoind --help isn't working.

   Expected behavior

   Run normally

   <!--- What behavior did you expect? -->

   Actual behavior

   100% cpu usage, no output

   <!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

   To reproduce

   Compile current master and try to run bitcoind

   <!--- How reliably can you reproduce the issue, what are the steps to do so? -->

   System information

   <!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

   latest master

   <!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

   Debian Buster VM with 28 Xeon Platinum cores, 64GB RAM, SSD storage

   <!-- GUI-related issue? What is your operating system and its version? If Linux, what is your desktop environment and graphical shell? -->

   <!-- Any extra information that might be useful in the debugging process. -->

   <!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

   Compiled from v0.19.0.1 tag and it's working.

2. mb300sd added the label Bug on Nov 19, 2019
3. fanquake added the label Linux/Unix on Nov 19, 2019
4. 
   MarcoFalke commented at 8:22 PM on November 19, 2019: member

   Can you start in gdb and see where it is stalling?

5. 
   mb300sd commented at 8:24 PM on November 19, 2019: contributor

   Just did. Not very familiar with gdb, but seem to have found it.

   gdb --args ./bitcoind --help
   GNU gdb (Debian 8.2.1-2+b3) 8.2.1
   Copyright (C) 2018 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.
   Type "show copying" and "show warranty" for details.
   This GDB was configured as "x86_64-linux-gnu".
   Type "show configuration" for configuration details.
   For bug reporting instructions, please see:
   <http://www.gnu.org/software/gdb/bugs/>.
   Find the GDB manual and other documentation resources online at:
       <http://www.gnu.org/software/gdb/documentation/>.
   
   For help, type "help".
   Type "apropos word" to search for commands related to "word"...
   Reading symbols from ./bitcoind...done.
   (gdb) run
   Starting program: /root/src/bitcoin/src/bitcoind --help
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
   
   ^C
   Program received signal SIGINT, Interrupt.
   0x000055555596962e in GetCPUID (d=<optimized out>, c=<optimized out>, b=<optimized out>, a=<optimized out>, subleaf=<optimized out>,
       leaf=<optimized out>) at randomenv.cpp:202
   202                 AddCPUID(hasher, leaf, subleaf, ax, bx, cx, dx);
   

6. 
   fanquake commented at 8:25 PM on November 19, 2019: member

   That will be related to #17270.

7. MarcoFalke added the label Utils/log/libs on Nov 19, 2019
8. MarcoFalke added this to the milestone 0.20.0 on Nov 19, 2019
9. 
   practicalswift commented at 8:32 PM on November 19, 2019: contributor

   @mb300sd Thanks a lot for reporting!

   What is the output of git rev-parse HEAD?

10. 
    mb300sd commented at 8:32 PM on November 19, 2019: contributor

    b4a1da9ef8e4b673c290d5b882527e627ae1b43a

11. 
    fanquake commented at 8:36 PM on November 19, 2019: member

    @mb300sd Can you configure with --enable-debug and post a full backtrace?

12. 
    sipa commented at 8:37 PM on November 19, 2019: member

    @mb300sd What's the output of cat /proc/cpuinfo?

    My guess is that somehow the iteration over the cpuid data does not terminate.

13. 
    mb300sd commented at 8:40 PM on November 19, 2019: contributor

    (gdb) run
    Starting program: /root/src/bitcoin/src/bitcoind --help
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    ^C
    Program received signal SIGINT, Interrupt.
    __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232
    232     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
    (gdb) bt
    [#0](/bitcoin-bitcoin/0/)  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232
    [#1](/bitcoin-bitcoin/1/)  0x0000555555cdc47c in CSHA512::Write (this=0x7fffffffe330, data=0x7fffffffde2c "", len=4) at crypto/sha512.cpp:179
    [#2](/bitcoin-bitcoin/2/)  0x0000555555c63995 in (anonymous namespace)::operator<< <unsigned int> (hasher=..., data=@0x7fffffffde2c: 0) at randomenv.cpp:119
    [#3](/bitcoin-bitcoin/3/)  0x0000555555c62aaa in (anonymous namespace)::AddCPUID (hasher=..., leaf=4, subleaf=2372698, ax=@0x7fffffffde24: 1006649344, bx=@0x7fffffffde28: 0,
        cx=@0x7fffffffde2c: 0, dx=@0x7fffffffde30: 0) at randomenv.cpp:191
    [#4](/bitcoin-bitcoin/4/)  0x0000555555c62b69 in (anonymous namespace)::AddAllCPUID (hasher=...) at randomenv.cpp:202
    [#5](/bitcoin-bitcoin/5/)  0x0000555555c632a7 in RandAddStaticEnv (hasher=...) at randomenv.cpp:359
    [#6](/bitcoin-bitcoin/6/)  0x0000555555c603fd in SeedStartup (hasher=..., rng=...) at random.cpp:510
    [#7](/bitcoin-bitcoin/7/)  0x0000555555c60580 in ProcRand (
        out=0x555556108780 <(anonymous namespace)::signatureCache> "\213\067\366\023Ss(\204\206\031\362\376\341\231F\030\351\362\t\aa\337\016Y\372\352;\217@?\341n", num=32, level=RNGLevel::FAST) at random.cpp:547
    [#8](/bitcoin-bitcoin/8/)  0x0000555555c60602 in GetRandBytes (
        buf=0x555556108780 <(anonymous namespace)::signatureCache> "\213\067\366\023Ss(\204\206\031\362\376\341\231F\030\351\362\t\aa\337\016Y\372\352;\217@?\341n", num=32) at random.cpp:552
    [#9](/bitcoin-bitcoin/9/)  0x00005555557f5db9 in (anonymous namespace)::CSignatureCache::CSignatureCache (this=0x555556108780 <(anonymous namespace)::signatureCache>)
        at script/sigcache.cpp:34
    [#10](/bitcoin-bitcoin/10/) 0x00005555557f66eb in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at script/sigcache.cpp:67
    [#11](/bitcoin-bitcoin/11/) 0x00005555557f6742 in _GLOBAL__sub_I__Z18InitSignatureCachev () at script/sigcache.cpp:93
    [#12](/bitcoin-bitcoin/12/) 0x0000555555d6ab85 in __libc_csu_init ()
    [#13](/bitcoin-bitcoin/13/) 0x00007ffff73a102a in __libc_start_main (main=0x55555558da8c <main(int, char**)>, argc=2, argv=0x7fffffffe5e8,
        init=0x555555d6ab40 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5d8) at ../csu/libc-start.c:264
    [#14](/bitcoin-bitcoin/14/) 0x000055555558cc9a in _start ()
    

    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 85
    model name      : Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz
    stepping        : 5
    microcode       : 0xffffffff
    cpu MHz         : 2200.000
    cache size      : 39424 KB
    physical id     : 0
    siblings        : 28
    core id         : 0
    cpu cores       : 14
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 21
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
    bugs            : spectre_v1 spectre_v2 spec_store_bypass mds swapgs taa itlb_multihit
    bogomips        : 4400.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 44 bits physical, 48 bits virtual
    power management:
    
    ......
    

14. 
    sipa commented at 8:42 PM on November 19, 2019: member

    @mb300sd Can you compile and run this program?

    #include <stdio.h>
    #include <cpuid.h>
    #include <stdint.h>
    
    char static inline ch(uint32_t v, int b) {
        int c = (v >> (b * 8)) & 0xFF;
        if (c >= 32 && c<= 126) return c;
        return '.';
    }
    
    void static inline GetCPUID(uint32_t leaf, uint32_t subleaf, uint32_t& a, uint32_t& b, uint32_t& c, uint32_t& d)
    {
    #ifdef __GNUC__
        __cpuid_count(leaf, subleaf, a, b, c, d);
    #else
      __asm__ ("cpuid" : "=a"(a), "=b"(b), "=c"(c), "=d"(d) : "0"(leaf), "2"(subleaf));
    #endif
        printf("%lx.%lx: %08lx %08lx %08lx %08lx %c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c\n", (unsigned long)leaf, (unsigned long)subleaf, (unsigned long)a, (unsigned long)b, (unsigned long)c, (unsigned long)d, ch(a, 0), ch(a, 1), ch(a, 2), ch(a, 3), ch(b, 0), ch(b, 1), ch(b, 2), ch(b, 3), ch(c, 0), ch(c, 1), ch(c, 2), ch(c, 3), ch(d, 0), ch(d, 1), ch(d, 2), ch(d, 3));
    }
    
    void static walk(void) {
        uint32_t ax, bx, cx, dx;
        GetCPUID(0, 0, ax, bx, cx, dx);
        uint32_t max = ax;
        for (uint32_t leaf = 1; leaf <= max; ++leaf) {
            for (uint32_t subleaf = 0;; ++subleaf) {
                GetCPUID(leaf, subleaf, ax, bx, cx, dx);
                // Iterate over subleaves for leaf 4, 11, 13
                if (leaf != 4 && leaf != 11 && leaf != 13) break;
                if ((leaf == 4 || leaf == 13) && ax == 0) break;
                if (leaf == 11 && (cx & 0xFF00) == 0) break;
            }
        }
        GetCPUID(0x80000000, 0, ax, bx, cx, dx);
        uint32_t maxext = ax;
        for (uint32_t leaf = 0x80000001; leaf <= maxext; ++leaf) {
            GetCPUID(leaf, 0, ax, bx, cx, dx);
        }
    }
    
    int main(void) {
        walk();
        return 0;
    }
    

    g++ walkcpuid.cpp -o walkcpuid && ./walkcpuid

15. 
    mb300sd commented at 8:42 PM on November 19, 2019: contributor

    Different backtrace a second time.

    [#0](/bitcoin-bitcoin/0/)  0x0000555555c626c7 in GetCPUID (leaf=4, subleaf=4710365, a=@0x7fffffffde24: 1006649344, b=@0x7fffffffde28: 0, c=@0x7fffffffde2c: 0,
        d=@0x7fffffffde30: 0) at ./compat/cpuid.h:17
    [#1](/bitcoin-bitcoin/1/)  0x0000555555c62a4f in (anonymous namespace)::AddCPUID (hasher=..., leaf=4, subleaf=4710365, ax=@0x7fffffffde24: 1006649344, bx=@0x7fffffffde28: 0,
        cx=@0x7fffffffde2c: 0, dx=@0x7fffffffde30: 0) at randomenv.cpp:190
    [#2](/bitcoin-bitcoin/2/)  0x0000555555c62b69 in (anonymous namespace)::AddAllCPUID (hasher=...) at randomenv.cpp:202
    [#3](/bitcoin-bitcoin/3/)  0x0000555555c632a7 in RandAddStaticEnv (hasher=...) at randomenv.cpp:359
    [#4](/bitcoin-bitcoin/4/)  0x0000555555c603fd in SeedStartup (hasher=..., rng=...) at random.cpp:510
    [#5](/bitcoin-bitcoin/5/)  0x0000555555c60580 in ProcRand (
        out=0x555556108780 <(anonymous namespace)::signatureCache> "\020\275\r\223\267\364֠\240\301\251\225\253\365\252P\265\266YA\237Φ\306\370\205)2\243\066\16h", num=32, level=RNGLevel::FAST) at random.cpp:547
    [#6](/bitcoin-bitcoin/6/)  0x0000555555c60602 in GetRandBytes (
        buf=0x555556108780 <(anonymous namespace)::signatureCache> "\020\275\r\223\267\364֠\240\301\251\225\253\365\252P\265\266YA\237Φ\306\370\205)2\243\066\16h", num=32) at random.cpp:552
    [#7](/bitcoin-bitcoin/7/)  0x00005555557f5db9 in (anonymous namespace)::CSignatureCache::CSignatureCache (this=0x555556108780 <(anonymous namespace)::signatureCache>)
        at script/sigcache.cpp:34
    [#8](/bitcoin-bitcoin/8/)  0x00005555557f66eb in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at script/sigcache.cpp:67
    [#9](/bitcoin-bitcoin/9/)  0x00005555557f6742 in _GLOBAL__sub_I__Z18InitSignatureCachev () at script/sigcache.cpp:93
    [#10](/bitcoin-bitcoin/10/) 0x0000555555d6ab85 in __libc_csu_init ()
    [#11](/bitcoin-bitcoin/11/) 0x00007ffff73a102a in __libc_start_main (main=0x55555558da8c <main(int, char**)>, argc=2, argv=0x7fffffffe5e8,
        init=0x555555d6ab40 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5d8) at ../csu/libc-start.c:264
    [#12](/bitcoin-bitcoin/12/) 0x000055555558cc9a in _start ()
    

16. 
    mb300sd commented at 8:45 PM on November 19, 2019: contributor

    First couple lines

    0.0: 00000015 756e6547 6c65746e 49656e69 ....GenuntelineI
    1.0: 00050655 19200800 feda3203 1f8bfbff U..... ..2......
    2.0: 76036301 00f0b5ff 00000000 00c30000 .c.v............
    3.0: 00000000 00000000 00000000 00000000 ................
    4.0: 3c004121 01c0003f 0000003f 00000000 !A.<?...?.......
    4.1: 3c004122 01c0003f 0000003f 00000000 "A.<?...?.......
    4.2: 3c004143 03c0003f 000003ff 00000000 CA.<?...........
    4.3: 3c07c163 0280003f 0000dfff 00000000 c..<?...........
    4.4: 3c004000 00000000 00000000 00000000 .@.<............
    4.5: 3c004000 00000000 00000000 00000000 .@.<............
    4.6: 3c004000 00000000 00000000 00000000 .@.<............
    4.7: 3c004000 00000000 00000000 00000000 .@.<............
    4.8: 3c004000 00000000 00000000 00000000 .@.<............
    4.9: 3c004000 00000000 00000000 00000000 .@.<............
    4.a: 3c004000 00000000 00000000 00000000 .@.<............
    4.b: 3c004000 00000000 00000000 00000000 .@.<............
    4.c: 3c004000 00000000 00000000 00000000 .@.<............
    4.d: 3c004000 00000000 00000000 00000000 .@.<............
    4.e: 3c004000 00000000 00000000 00000000 .@.<............
    4.f: 3c004000 00000000 00000000 00000000 .@.<............
    

    and then it scrolls the last one forever.

    4.108cc5: 3c004000 00000000 00000000 00000000 .@.<............

17. 
    sipa commented at 8:46 PM on November 19, 2019: member

    @mb300sd Thanks, that's it!

18. 
    sipa commented at 10:21 PM on November 19, 2019: member

    @mb300sd Can you try again with the following walk function?

    void static walk(void) {
        uint32_t ax, bx, cx, dx;
        GetCPUID(0, 0, ax, bx, cx, dx);
        uint32_t max = ax;
        for (uint32_t leaf = 1; leaf <= max; ++leaf) {
            uint32_t maxsub = 0;
            for (uint32_t subleaf = 0;; ++subleaf) {
                GetCPUID(leaf, subleaf, ax, bx, cx, dx);
                if (leaf == 4) {
                    if ((ax & 0x1f) == 0) break;
                } else if (leaf == 7) {
                    if (subleaf == 0) maxsub = ax;
                    if (subleaf == maxsub) break;
                } else if (leaf == 11) {
                    if ((cx & 0xff00) == 0) break;
                } else if (leaf == 13) {
                    if (ax == 0 && bx == 0 && cx == 0 && dx == 0) break;
                } else {
                    break;
                }
            }
        }
        GetCPUID(0x80000000, 0, ax, bx, cx, dx);
        uint32_t maxext = ax;
        for (uint32_t leaf = 0x80000001; leaf <= maxext; ++leaf) {
            GetCPUID(leaf, 0, ax, bx, cx, dx);
        }
    }
    

19. 
    mb300sd commented at 10:28 PM on November 19, 2019: contributor

    Looks good!

    0.0: 00000015 756e6547 6c65746e 49656e69 ....GenuntelineI
    1.0: 00050655 18200800 feda3203 1f8bfbff U..... ..2......
    2.0: 76036301 00f0b5ff 00000000 00c30000 .c.v............
    3.0: 00000000 00000000 00000000 00000000 ................
    4.0: 3c004121 01c0003f 0000003f 00000000 !A.<?...?.......
    4.1: 3c004122 01c0003f 0000003f 00000000 "A.<?...?.......
    4.2: 3c004143 03c0003f 000003ff 00000000 CA.<?...........
    4.3: 3c07c163 0280003f 0000dfff 00000000 c..<?...........
    4.4: 3c004000 00000000 00000000 00000000 .@.<............
    5.0: 00000000 00000000 00000000 00000000 ................
    6.0: 00000000 00000000 00000000 00000000 ................
    7.0: 00000000 d19f6fb9 00000000 bc000000 .....o..........
    8.0: 00000000 00000000 00000000 00000000 ................
    9.0: 00000000 00000000 00000000 00000000 ................
    a.0: 00000000 00000000 00000000 00000000 ................
    b.0: 00000001 00000002 00000100 00000018 ................
    b.1: 00000005 00000020 00000201 00000018 .... ...........
    b.2: 00000000 00000000 00000002 00000018 ................
    c.0: 00000000 00000000 00000000 00000000 ................
    d.0: 000000ff 00000a80 00000a80 00000000 ................
    d.1: 0000000f 00000a00 00000000 00000000 ................
    d.2: 00000100 00000240 00000000 00000000 ....@...........
    d.3: 00000040 000003c0 00000000 00000000 @...............
    d.4: 00000040 00000400 00000000 00000000 @...............
    d.5: 00000040 00000440 00000000 00000000 @...@...........
    d.6: 00000200 00000480 00000000 00000000 ................
    d.7: 00000400 00000680 00000000 00000000 ................
    d.8: 00000000 00000000 00000000 00000000 ................
    e.0: 00000000 00000000 00000000 00000000 ................
    f.0: 00000000 00000000 00000000 00000000 ................
    10.0: 00000000 00000000 00000000 00000000 ................
    11.0: 00000000 00000000 00000000 00000000 ................
    12.0: 00000000 00000000 00000000 00000000 ................
    13.0: 00000000 00000000 00000000 00000000 ................
    14.0: 00000000 00000000 00000000 00000000 ................
    15.0: 00000000 00000000 00000000 00000000 ................
    80000000.0: 80000008 00000000 00000000 00000000 ................
    80000001.0: 00000000 00000000 00000121 2c100800 ........!......,
    80000002.0: 65746e49 2952286c 6f655820 2952286e Intel(R) Xeon(R)
    80000003.0: 616c5020 756e6974 3238206d 204c3637  Platinum 8276L
    80000004.0: 20555043 2e322040 48473032 0000007a CPU @ 2.20GHz...
    80000005.0: 00000000 00000000 00000000 00000000 ................
    80000006.0: 00000000 00000000 01006040 00000000 ........@`......
    80000007.0: 00000000 00000000 00000000 00000000 ................
    80000008.0: 0000302c 00000000 00000000 00000000 ,0..............
    

20. 
    sipa commented at 11:07 PM on November 19, 2019: member

    @mb300sd See #17527.

21. laanwj closed this on Nov 24, 2019

    ---

22. sidhujag referenced this in commit 7f69c42e6f on Nov 24, 2019
23. sidhujag referenced this in commit c14e5d9cf9 on Nov 10, 2020
24. DrahtBot locked this on Dec 16, 2021
25. PastaPastaPasta referenced this in commit 0d18eae6d0 on Apr 25, 2022

Milestone
0.20.0