'std::out_of_range' crash in I2P fuzz test

murchandamus commented at 1:46 pm on October 17, 2023: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

I got a std::out_of_range crash during merging fuzz outputs in the i2p target (see log below.) I was not able to reproduce the crash when re-running the seed with the fuzz executable in the regular build, but I figured I’d share it here if someone else wants to take a look. The binaries used for the merge and the reproduction were both built from the latest master: 738ef44abb6895dad016d8f32f7d7fa1c251b354.

Expected behaviour

If this issue can be reproduced, it may point at a bug in the I2P fuzzer or the I2P code.

Steps to reproduce

You can recreate the seed with:

echo "wIA9ID0gUkVTVUxUPU9LClBSSVY9gD0gPSBSRVNVTFQ9T0sKUFJJVj0CAAD//13/GhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoAEBoaGhoaGhoaGhoaGhouGhoaGhoaGhoaGhoaGn4aGhoaGhoaGgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaABoaGhoaGhpXGhoAGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGurq6mrqUFBQUFBQUFBQUFBQUOrq6gAAAABbAAAAAAAAAAAAAAAAAAAAAgAAeHh4eHh4eHgpeHh4eHh4eHh4eHgaGhoaGhoaGhoaGho=" | base64 -d crash-946784c8f03d9aeeef70e22b346a069e6940e186

Relevant log output

 0Run i2p with args /home/murch/Workspace/qa-merge/src/test/fuzz/fuzz -set_cover_merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=0 /tmp/merge-all/i2p ../qa-assets/fuzz_seed_corpus/i2p ../qa-assets-active-fuzzing/fuzz_seed_corpus/i2p
 1INFO: Running with entropic power schedule (0xFF, 100).
 2INFO: Seed: 2619897554
 3INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x55a029467ca0, 0x55a0294c4a37), 
 4INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x55a0294c4a38,0x55a029a923a8), 
 5MERGE-OUTER: 14141 files, 0 in the initial corpus, 0 processed earlier
 6MERGE-OUTER: attempt 1
 7INFO: Running with entropic power schedule (0xFF, 100).
 8INFO: Seed: 3047975919
 9INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x563efb209ca0, 0x563efb266a37), 
10INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x563efb266a38,0x563efb8343a8), 
11INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
12MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge17284.txt'
13MERGE-INNER: 14141 total files; 0 processed earlier; will process 14141 files now
14[#1](/bitcoin-bitcoin/1/)	pulse  cov: 244 exec/s: 0 rss: 88Mb
15[#2](/bitcoin-bitcoin/2/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
16[#4](/bitcoin-bitcoin/4/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
17[#8](/bitcoin-bitcoin/8/)	pulse  cov: 245 exec/s: 0 rss: 88Mb
18[#16](/bitcoin-bitcoin/16/)	pulse  cov: 263 exec/s: 0 rss: 88Mb
19[#32](/bitcoin-bitcoin/32/)	pulse  cov: 300 exec/s: 0 rss: 88Mb
20[#64](/bitcoin-bitcoin/64/)	pulse  cov: 313 exec/s: 0 rss: 88Mb
21[#128](/bitcoin-bitcoin/128/)	pulse  cov: 336 exec/s: 0 rss: 88Mb
22[#256](/bitcoin-bitcoin/256/)	pulse  cov: 417 exec/s: 0 rss: 88Mb
23[#512](/bitcoin-bitcoin/512/)	pulse  cov: 435 exec/s: 0 rss: 88Mb
24[#1024](/bitcoin-bitcoin/1024/)	pulse  cov: 455 exec/s: 1024 rss: 88Mb
25[#2048](/bitcoin-bitcoin/2048/)	pulse  cov: 485 exec/s: 1024 rss: 88Mb
26[#4096](/bitcoin-bitcoin/4096/)	pulse  cov: 538 exec/s: 682 rss: 88Mb
27terminate called after throwing an instance of 'std::out_of_range'
28  what():  vector::_M_range_check: __n (which is 385) >= this->size() (which is 0)
29==17287== ERROR: libFuzzer: deadly signal
30    [#0](/bitcoin-bitcoin/0/) 0x563ef9f7eda4 in __sanitizer_print_stack_trace (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x9bfda4) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
31    [#1](/bitcoin-bitcoin/1/) 0x563ef9f56248 in fuzzer::PrintStackTrace() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997248) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
32    [#2](/bitcoin-bitcoin/2/) 0x563ef9f3c2d3 in fuzzer::Fuzzer::CrashCallback() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97d2d3) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
33    [#3](/bitcoin-bitcoin/3/) 0x7fa19a83c45f  (/lib/x86_64-linux-gnu/libc.so.6+0x3c45f) (BuildId: ff2d8e707625b73b293961a4bc168e373d14a44a)
34    [#4](/bitcoin-bitcoin/4/) 0x7fa19a89152a in __pthread_kill_implementation nptl/pthread_kill.c:43:17
35    [#5](/bitcoin-bitcoin/5/) 0x7fa19a89152a in __pthread_kill_internal nptl/pthread_kill.c:78:10
36    [#6](/bitcoin-bitcoin/6/) 0x7fa19a89152a in pthread_kill nptl/pthread_kill.c:89:10
37    [#7](/bitcoin-bitcoin/7/) 0x7fa19a83c3b5 in raise signal/../sysdeps/posix/raise.c:26:13
38    [#8](/bitcoin-bitcoin/8/) 0x7fa19a82287b in abort stdlib/abort.c:79:7
39    [#9](/bitcoin-bitcoin/9/) 0x7fa19aca4ee5  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa4ee5) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
40    [#10](/bitcoin-bitcoin/10/) 0x7fa19acb6e9b  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6e9b) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
41    [#11](/bitcoin-bitcoin/11/) 0x7fa19acb6f06 in std::terminate() (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6f06) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
42    [#12](/bitcoin-bitcoin/12/) 0x7fa19acb7167 in __cxa_throw (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb7167) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
43    [#13](/bitcoin-bitcoin/13/) 0x7fa19aca82ba  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa82ba) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
44    [#14](/bitcoin-bitcoin/14/) 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::_M_range_check(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1153:4
45    [#15](/bitcoin-bitcoin/15/) 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::at(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1194:2
46    [#16](/bitcoin-bitcoin/16/) 0x563efa269705 in i2p::sam::Session::MyDestination() const src/i2p.cpp:354:38
47    [#17](/bitcoin-bitcoin/17/) 0x563efa269705 in i2p::sam::Session::CreateIfNotCreatedAlready() src/i2p.cpp:405:40
48    [#18](/bitcoin-bitcoin/18/) 0x563efa268664 in i2p::sam::Session::Listen(i2p::Connection&) src/i2p.cpp:143:9
49    [#19](/bitcoin-bitcoin/19/) 0x563efa07c088 in i2p_fuzz_target(Span<unsigned char const>) src/test/fuzz/i2p.cpp:38:14
50    [#20](/bitcoin-bitcoin/20/) 0x563ef9f8269e in void std::__invoke_impl<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(std::__invoke_other, void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
51    [#21](/bitcoin-bitcoin/21/) 0x563ef9f8269e in std::enable_if<is_invocable_r_v<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>, void>::type std::__invoke_r<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:111:2
52    [#22](/bitcoin-bitcoin/22/) 0x563ef9f8269e in std::_Function_handler<void (Span<unsigned char const>), void (*)(Span<unsigned char const>)>::_M_invoke(std::_Any_data const&, Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:9
53    [#23](/bitcoin-bitcoin/23/) 0x563efa1dd075 in std::function<void (Span<unsigned char const>)>::operator()(Span<unsigned char const>) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
54    [#24](/bitcoin-bitcoin/24/) 0x563efa1dd075 in LLVMFuzzerTestOneInput src/test/fuzz/fuzz.cpp:178:5
55    [#25](/bitcoin-bitcoin/25/) 0x563ef9f3d742 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97e742) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
56    [#26](/bitcoin-bitcoin/26/) 0x563ef9f47445 in fuzzer::Fuzzer::CrashResistantMergeInternalStep(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, bool) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x988445) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
57    [#27](/bitcoin-bitcoin/27/) 0x563ef9f2d5cd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x96e5cd) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
58    [#28](/bitcoin-bitcoin/28/) 0x563ef9f56a82 in main (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997a82) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
59    [#29](/bitcoin-bitcoin/29/) 0x7fa19a823a8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
60    [#30](/bitcoin-bitcoin/30/) 0x7fa19a823b48 in __libc_start_main csu/../csu/libc-start.c:360:3
61    [#31](/bitcoin-bitcoin/31/) 0x563ef9f22074 in _start (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x963074) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
62
63NOTE: libFuzzer has rudimentary signal handlers.
64      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
65SUMMARY: libFuzzer: deadly signal
66MS: 0 ; base unit: 0000000000000000000000000000000000000000
67artifact_prefix='./'; Test unit written to ./crash-946784c8f03d9aeeef70e22b346a069e6940e186

 0➜  qa-merge git:(merge-fuzz) ✗ FUZZ=i2p src/test/fuzz/fuzz   crash-946784c8f03d9aeeef70e22b346a069e6940e186
 1INFO: Running with entropic power schedule (0xFF, 100).
 2INFO: Seed: 3335468885
 3INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x5612fee10ca0, 0x5612fee6da37), 
 4INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x5612fee6da38,0x5612ff43b3a8), 
 5src/test/fuzz/fuzz: Running 1 inputs 1 time(s) each.
 6Running: crash-946784c8f03d9aeeef70e22b346a069e6940e186
 7Executed crash-946784c8f03d9aeeef70e22b346a069e6940e186 in 0 ms
 8***
 9*** NOTE: fuzzing was not performed, you have only
10***       executed the target code on a fixed set of inputs.
11***

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

738ef44abb6895dad016d8f32f7d7fa1c251b354

Operating system and version

Ubuntu 23.04

Machine specifications

No response

maflcko commented at 2:23 pm on October 17, 2023: member

Is m_private_key accessed by more than one thread? Which threads are running in the target? What happens if the target is run in tsan?

murchandamus commented at 2:51 pm on October 17, 2023: contributor

As far as I can tell, the merging process was the only thing running that touched this data. I’m not sure how I would be running the target in tsan.

darosior commented at 3:07 pm on October 17, 2023: member

I’m not sure how I would be running the target in tsan.

See the --with-sanitizers configure option. Available sanitizers are listed here.

So if you are using ./configure --with-sanitizers=fuzzer right now, you want to compile using ./configure --with-sanitizers=fuzzer,thread for tsan.

dergoegge commented at 3:10 pm on October 17, 2023: member

I’m also not able to reproduce. Running with tsan also doesn’t yield anything so far (I’m not even sure there are any threads involved?).

maflcko commented at 5:28 pm on October 17, 2023: member

Jup, could be a single thread, and given that tsan doesn’t complain, seems good to close this for now.

maflcko closed this on Oct 17, 2023

maflcko reopened this on Oct 19, 2023

maflcko commented at 11:00 am on October 19, 2023: member

Steps to reproduce:

0cd qa-assets
1git fetch origin 00c956e4a9c555da5640946ec28cdd5e501c0554
2git checkout 00c956e4a9c555da5640946ec28cdd5e501c0554
3
4FUZZ=i2p ../bitcoin-core/src/test/fuzz/fuzz -runs=1 ./fuzz_seed_corpus/i2p/

maflcko commented at 11:00 am on October 19, 2023: member

cc @vasild

maflcko added this to the milestone 26.0 on Oct 19, 2023

maflcko added the label Bug on Oct 19, 2023

maflcko commented at 3:21 pm on October 19, 2023: member

 0549c82ad3a34a885ecca37a5f04c36dfbaa95d17 is the first bad commit
 1commit 549c82ad3a34a885ecca37a5f04c36dfbaa95d17
 2Author: Vasil Dimov <vd@FreeBSD.org>
 3Date:   Wed Apr 7 11:40:59 2021 +0200
 4
 5    fuzz: use ConsumeBool() instead of !ConsumeBool()
 6    
 7    The former is shorter and ends up with a "random" bool anyway.
 8
 9 src/test/fuzz/util.cpp | 2 +-
10 1 file changed, 1 insertion(+), 1 deletion(-)

maflcko commented at 3:22 pm on October 19, 2023: member

Tried a bisect, but I guess that didn’t work, because the fuzz input format was changed at some point in 2021, so likely the fuzz inputs trigger something else when going back further than 549c82ad3a34a885ecca37a5f04c36dfbaa95d17

mzumsande commented at 6:14 pm on October 19, 2023: contributor

This cannot be reproduced by running a single seed, only by a combination of two seeds from the qa-assets mentioned in #28665 (comment): FUZZ=i2p ./src/test/fuzz/fuzz -runs=1 ~/qa-assets/fuzz_seed_corpus/i2p/41717eadbc1239897d9a03b261a2590aff0474cf ~/qa-assets/fuzz_seed_corpus/i2p/56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59

I think the problem is that one seed (41717eadbc1239897d9a03b261a2590aff0474cf) attempts to create a i2p privkey, which will be empty because i2p is not running within the fuzz environment (within normal usage of bitcoind, this should be impossible because init would abort). This empty file is saved into the datadir, and then picked up by another seed (56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59), leading to the error. So I think that there are two things to fix:

Have the fuzz test clean up after itself (removing any private key it may write into the data directory) so that fuzz seeds don’t depend on each other
Implement better checks in i2p.cpp to prevent parsing of empty private keys.

maflcko commented at 11:38 am on October 20, 2023: member

On a general note I wonder how useful this target is at all, given that it can’t even reach basic coverage in its own test code, despite years of fuzzing: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/77f0ceb7175dbd00/7c13430491b21dca/fuzz.coverage/src/test/fuzz/i2p.cpp.gcov.html

Not sure whether the lack of coverage is due to this bug here, but that also makes me question why no one (including oss-fuzz) reported this bug earlier over the years?

fanquake closed this on Oct 20, 2023

‘std::out_of_range’ crash in I2P fuzz test #28665