'std::out_of_range' crash in I2P fuzz test #28665

1. 
   murchandamus commented at 1:46 PM on October 17, 2023: contributor

   Is there an existing issue for this?

   • I have searched the existing issues

   Current behaviour

   I got a std::out_of_range crash during merging fuzz outputs in the i2p target (see log below.) I was not able to reproduce the crash when re-running the seed with the fuzz executable in the regular build, but I figured I’d share it here if someone else wants to take a look. The binaries used for the merge and the reproduction were both built from the latest master: 738ef44abb6895dad016d8f32f7d7fa1c251b354.

   Expected behaviour

   If this issue can be reproduced, it may point at a bug in the I2P fuzzer or the I2P code.

   Steps to reproduce

   You can recreate the seed with:

   echo "wIA9ID0gUkVTVUxUPU9LClBSSVY9gD0gPSBSRVNVTFQ9T0sKUFJJVj0CAAD//13/GhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoAEBoaGhoaGhoaGhoaGhouGhoaGhoaGhoaGhoaGn4aGhoaGhoaGgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaABoaGhoaGhpXGhoAGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGurq6mrqUFBQUFBQUFBQUFBQUOrq6gAAAABbAAAAAAAAAAAAAAAAAAAAAgAAeHh4eHh4eHgpeHh4eHh4eHh4eHgaGhoaGhoaGhoaGho=" | base64 -d crash-946784c8f03d9aeeef70e22b346a069e6940e186

   Relevant log output

   Run i2p with args /home/murch/Workspace/qa-merge/src/test/fuzz/fuzz -set_cover_merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=0 /tmp/merge-all/i2p ../qa-assets/fuzz_seed_corpus/i2p ../qa-assets-active-fuzzing/fuzz_seed_corpus/i2p
   INFO: Running with entropic power schedule (0xFF, 100).
   INFO: Seed: 2619897554
   INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x55a029467ca0, 0x55a0294c4a37), 
   INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x55a0294c4a38,0x55a029a923a8), 
   MERGE-OUTER: 14141 files, 0 in the initial corpus, 0 processed earlier
   MERGE-OUTER: attempt 1
   INFO: Running with entropic power schedule (0xFF, 100).
   INFO: Seed: 3047975919
   INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x563efb209ca0, 0x563efb266a37), 
   INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x563efb266a38,0x563efb8343a8), 
   INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
   MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge17284.txt'
   MERGE-INNER: 14141 total files; 0 processed earlier; will process 14141 files now
   [#1](/bitcoin-bitcoin/1/)	pulse  cov: 244 exec/s: 0 rss: 88Mb
   [#2](/bitcoin-bitcoin/2/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
   [#4](/bitcoin-bitcoin/4/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
   [#8](/bitcoin-bitcoin/8/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
   [#16](/bitcoin-bitcoin/16/)	pulse  cov: 263 exec/s: 0 rss: 88Mb
   [#32](/bitcoin-bitcoin/32/)	pulse  cov: 300 exec/s: 0 rss: 88Mb
   [#64](/bitcoin-bitcoin/64/)	pulse  cov: 313 exec/s: 0 rss: 88Mb
   [#128](/bitcoin-bitcoin/128/)	pulse  cov: 336 exec/s: 0 rss: 88Mb
   [#256](/bitcoin-bitcoin/256/)	pulse  cov: 417 exec/s: 0 rss: 88Mb
   [#512](/bitcoin-bitcoin/512/)	pulse  cov: 435 exec/s: 0 rss: 88Mb
   [#1024](/bitcoin-bitcoin/1024/)	pulse  cov: 455 exec/s: 1024 rss: 88Mb
   [#2048](/bitcoin-bitcoin/2048/)	pulse  cov: 485 exec/s: 1024 rss: 88Mb
   [#4096](/bitcoin-bitcoin/4096/)	pulse  cov: 538 exec/s: 682 rss: 88Mb
   terminate called after throwing an instance of 'std::out_of_range'
     what():  vector::_M_range_check: __n (which is 385) >= this->size() (which is 0)
   ==17287== ERROR: libFuzzer: deadly signal
       [#0](/bitcoin-bitcoin/0/) 0x563ef9f7eda4 in __sanitizer_print_stack_trace (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x9bfda4) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#1](/bitcoin-bitcoin/1/) 0x563ef9f56248 in fuzzer::PrintStackTrace() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997248) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#2](/bitcoin-bitcoin/2/) 0x563ef9f3c2d3 in fuzzer::Fuzzer::CrashCallback() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97d2d3) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#3](/bitcoin-bitcoin/3/) 0x7fa19a83c45f  (/lib/x86_64-linux-gnu/libc.so.6+0x3c45f) (BuildId: ff2d8e707625b73b293961a4bc168e373d14a44a)
       [#4](/bitcoin-bitcoin/4/) 0x7fa19a89152a in __pthread_kill_implementation nptl/pthread_kill.c:43:17
       [#5](/bitcoin-bitcoin/5/) 0x7fa19a89152a in __pthread_kill_internal nptl/pthread_kill.c:78:10
       [#6](/bitcoin-bitcoin/6/) 0x7fa19a89152a in pthread_kill nptl/pthread_kill.c:89:10
       [#7](/bitcoin-bitcoin/7/) 0x7fa19a83c3b5 in raise signal/../sysdeps/posix/raise.c:26:13
       [#8](/bitcoin-bitcoin/8/) 0x7fa19a82287b in abort stdlib/abort.c:79:7
       [#9](/bitcoin-bitcoin/9/) 0x7fa19aca4ee5  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa4ee5) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
       [#10](/bitcoin-bitcoin/10/) 0x7fa19acb6e9b  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6e9b) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
       [#11](/bitcoin-bitcoin/11/) 0x7fa19acb6f06 in std::terminate() (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6f06) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
       [#12](/bitcoin-bitcoin/12/) 0x7fa19acb7167 in __cxa_throw (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb7167) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
       [#13](/bitcoin-bitcoin/13/) 0x7fa19aca82ba  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa82ba) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
       [#14](/bitcoin-bitcoin/14/) 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::_M_range_check(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1153:4
       [#15](/bitcoin-bitcoin/15/) 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::at(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1194:2
       [#16](/bitcoin-bitcoin/16/) 0x563efa269705 in i2p::sam::Session::MyDestination() const src/i2p.cpp:354:38
       [#17](/bitcoin-bitcoin/17/) 0x563efa269705 in i2p::sam::Session::CreateIfNotCreatedAlready() src/i2p.cpp:405:40
       [#18](/bitcoin-bitcoin/18/) 0x563efa268664 in i2p::sam::Session::Listen(i2p::Connection&) src/i2p.cpp:143:9
       [#19](/bitcoin-bitcoin/19/) 0x563efa07c088 in i2p_fuzz_target(Span<unsigned char const>) src/test/fuzz/i2p.cpp:38:14
       [#20](/bitcoin-bitcoin/20/) 0x563ef9f8269e in void std::__invoke_impl<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(std::__invoke_other, void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
       [#21](/bitcoin-bitcoin/21/) 0x563ef9f8269e in std::enable_if<is_invocable_r_v<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>, void>::type std::__invoke_r<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:111:2
       [#22](/bitcoin-bitcoin/22/) 0x563ef9f8269e in std::_Function_handler<void (Span<unsigned char const>), void (*)(Span<unsigned char const>)>::_M_invoke(std::_Any_data const&, Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:9
       [#23](/bitcoin-bitcoin/23/) 0x563efa1dd075 in std::function<void (Span<unsigned char const>)>::operator()(Span<unsigned char const>) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
       [#24](/bitcoin-bitcoin/24/) 0x563efa1dd075 in LLVMFuzzerTestOneInput src/test/fuzz/fuzz.cpp:178:5
       [#25](/bitcoin-bitcoin/25/) 0x563ef9f3d742 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97e742) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#26](/bitcoin-bitcoin/26/) 0x563ef9f47445 in fuzzer::Fuzzer::CrashResistantMergeInternalStep(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, bool) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x988445) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#27](/bitcoin-bitcoin/27/) 0x563ef9f2d5cd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x96e5cd) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#28](/bitcoin-bitcoin/28/) 0x563ef9f56a82 in main (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997a82) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
       [#29](/bitcoin-bitcoin/29/) 0x7fa19a823a8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
       [#30](/bitcoin-bitcoin/30/) 0x7fa19a823b48 in __libc_start_main csu/../csu/libc-start.c:360:3
       [#31](/bitcoin-bitcoin/31/) 0x563ef9f22074 in _start (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x963074) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
   
   NOTE: libFuzzer has rudimentary signal handlers.
         Combine libFuzzer with AddressSanitizer or similar for better crash reports.
   SUMMARY: libFuzzer: deadly signal
   MS: 0 ; base unit: 0000000000000000000000000000000000000000
   artifact_prefix='./'; Test unit written to ./crash-946784c8f03d9aeeef70e22b346a069e6940e186
   

   ➜  qa-merge git:(merge-fuzz) ✗ FUZZ=i2p src/test/fuzz/fuzz   crash-946784c8f03d9aeeef70e22b346a069e6940e186
   INFO: Running with entropic power schedule (0xFF, 100).
   INFO: Seed: 3335468885
   INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x5612fee10ca0, 0x5612fee6da37), 
   INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x5612fee6da38,0x5612ff43b3a8), 
   src/test/fuzz/fuzz: Running 1 inputs 1 time(s) each.
   Running: crash-946784c8f03d9aeeef70e22b346a069e6940e186
   Executed crash-946784c8f03d9aeeef70e22b346a069e6940e186 in 0 ms
   ***
   *** NOTE: fuzzing was not performed, you have only
   ***       executed the target code on a fixed set of inputs.
   ***
   

   How did you obtain Bitcoin Core

   Compiled from source

   What version of Bitcoin Core are you using?

   738ef44abb6895dad016d8f32f7d7fa1c251b354

   Operating system and version

   Ubuntu 23.04

   Machine specifications

   No response

2. 
   maflcko commented at 2:23 PM on October 17, 2023: member

   Is m_private_key accessed by more than one thread? Which threads are running in the target? What happens if the target is run in tsan?

3. 
   murchandamus commented at 2:51 PM on October 17, 2023: contributor

   As far as I can tell, the merging process was the only thing running that touched this data. I’m not sure how I would be running the target in tsan.

4. 
   darosior commented at 3:07 PM on October 17, 2023: member

   I’m not sure how I would be running the target in tsan.

   See the --with-sanitizers configure option. Available sanitizers are listed here.

   So if you are using ./configure --with-sanitizers=fuzzer right now, you want to compile using ./configure --with-sanitizers=fuzzer,thread for tsan.

5. 
   dergoegge commented at 3:10 PM on October 17, 2023: member

   I'm also not able to reproduce. Running with tsan also doesn't yield anything so far (I'm not even sure there are any threads involved?).

6. 
   maflcko commented at 5:28 PM on October 17, 2023: member

   Jup, could be a single thread, and given that tsan doesn't complain, seems good to close this for now.

7. maflcko closed this on Oct 17, 2023

   ---

8. maflcko reopened this on Oct 19, 2023

   ---

9. 
   maflcko commented at 11:00 AM on October 19, 2023: member

   Steps to reproduce:

   cd qa-assets
   git fetch origin 00c956e4a9c555da5640946ec28cdd5e501c0554
   git checkout 00c956e4a9c555da5640946ec28cdd5e501c0554
   
   FUZZ=i2p ../bitcoin-core/src/test/fuzz/fuzz -runs=1 ./fuzz_seed_corpus/i2p/
   

10. 
    maflcko commented at 11:00 AM on October 19, 2023: member

    cc @vasild

11. maflcko added this to the milestone 26.0 on Oct 19, 2023
12. maflcko added the label Bug on Oct 19, 2023
13. 
    maflcko commented at 3:21 PM on October 19, 2023: member

    549c82ad3a34a885ecca37a5f04c36dfbaa95d17 is the first bad commit
    commit 549c82ad3a34a885ecca37a5f04c36dfbaa95d17
    Author: Vasil Dimov <vd@FreeBSD.org>
    Date:   Wed Apr 7 11:40:59 2021 +0200
    
        fuzz: use ConsumeBool() instead of !ConsumeBool()
        
        The former is shorter and ends up with a "random" bool anyway.
    
     src/test/fuzz/util.cpp | 2 +-
     1 file changed, 1 insertion(+), 1 deletion(-)
    

14. 
    maflcko commented at 3:22 PM on October 19, 2023: member

    Tried a bisect, but I guess that didn't work, because the fuzz input format was changed at some point in 2021, so likely the fuzz inputs trigger something else when going back further than 549c82ad3a34a885ecca37a5f04c36dfbaa95d17

15. 
    mzumsande commented at 6:14 PM on October 19, 2023: contributor

    This cannot be reproduced by running a single seed, only by a combination of two seeds from the qa-assets mentioned in #28665 (comment): FUZZ=i2p ./src/test/fuzz/fuzz -runs=1 ~/qa-assets/fuzz_seed_corpus/i2p/41717eadbc1239897d9a03b261a2590aff0474cf ~/qa-assets/fuzz_seed_corpus/i2p/56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59

    I think the problem is that one seed (41717eadbc1239897d9a03b261a2590aff0474cf) attempts to create a i2p privkey, which will be empty because i2p is not running within the fuzz environment (within normal usage of bitcoind, this should be impossible because init would abort). This empty file is saved into the datadir, and then picked up by another seed (56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59), leading to the error. So I think that there are two things to fix:

    1. Have the fuzz test clean up after itself (removing any private key it may write into the data directory) so that fuzz seeds don't depend on each other
    2. Implement better checks in i2p.cpp to prevent parsing of empty private keys.
16. 
    maflcko commented at 11:38 AM on October 20, 2023: member

    On a general note I wonder how useful this target is at all, given that it can't even reach basic coverage in its own test code, despite years of fuzzing: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/77f0ceb7175dbd00/7c13430491b21dca/fuzz.coverage/src/test/fuzz/i2p.cpp.gcov.html

    Not sure whether the lack of coverage is due to this bug here, but that also makes me question why no one (including oss-fuzz) reported this bug earlier over the years?

17. fanquake closed this on Oct 20, 2023

    ---

18. Frank-GER referenced this in commit 5f33e14e52 on Oct 21, 2023
19. Julio-Rats referenced this in commit abfc8c901d on Jan 26, 2024
20. bitcoin locked this on Oct 19, 2024

Milestone
26.0