fuzz: p2p: Detect peer deadlocks #29009

pull maflcko wants to merge 2 commits into bitcoin:master from maflcko:2312-fuzz-p2p-dead- changing 3 files +28 −12
  1. maflcko commented at 12:15 pm on December 6, 2023: member

    It may be possible that a peer connection will deadlock, due to software bugs such as #18808.

    Fix this by detecting them in the fuzz target.

    Can be tested by introducing a bug such as:

    0diff --git a/src/net_processing.cpp b/src/net_processing.cpp
    1index 1067341495..97495a13df 100644
    2--- a/src/net_processing.cpp
    3+++ b/src/net_processing.cpp
    4@@ -2436,3 +2436,3 @@ void PeerManagerImpl::ProcessGetData(CNode& pfrom, Peer& peer, const std::atomic
    5     if (it != peer.m_getdata_requests.end() && !pfrom.fPauseSend) {
    6-        const CInv &inv = *it++;
    7+        const CInv& inv = *it;
    8         if (inv.IsGenBlkMsg()) {
    

    Using a fuzz input such as:

    0$ base64 ./timeout-ada0fecaba2b8c46c6e970cf637d9625b01bf7e5 
    1kNptdNbW1tbWYghvXIpwb25vPQAA////////cwAjLv8AXAB2ZXJhY2sAQW5v/62tra3Pz///////
    2//////////////////////9c8GZpbHRlcmxvYWQAAAEAAwAAAABVYwC2XABmaWx0ZXJhZGQAAAAX
    3Fxdn/////2V0F861tcqvEmAAACEAAABjYXB0dXJldmUAAH4AgAA1PNfX11x0Z2V0ZGF0YQBDACOw
    4AQMAAAAGIm5GERoLWcqvEmBD61u/KMNPOl4zKh/HKLK3PPGIkQ9eE/////////8AAAAAAAAAAFtb
    5WyjDTzpeMSofx7K3PNfX11x0Z2V0ZGF0YQBDACMwAQMAAAAGIm5GERoLWcqvEmBD61u/KMNPOl4z
    6Kh/Hsrc88YiRD2/Nzc3Nzc3Nzc3NTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N
    7zWWj1NTUudTU1NTU1P///0j+P/9cdHR4AAAAAAAAy/4AAHR4AAAAAAAAP8v+AAD/+P//////////
    8AX55bJl8HWnz/////wAgXGF0YVPxY2RkAAAA
    

    And running the fuzz target:

    0$ FUZZ=process_messages ./src/test/fuzz/fuzz -runs=1 -timeout=18 ./timeout-ada0fecaba2b8c46c6e970cf637d9625b01bf7e5 
    1INFO: Running with entropic power schedule (0xFF, 100).
    2INFO: Seed: 3436516708
    3INFO: Loaded 1 modules   (390807 inline 8-bit counters): 390807 [0x55d0d6221e80, 0x55d0d6281517), 
    4INFO: Loaded 1 PC tables (390807 PCs): 390807 [0x55d0d6281518,0x55d0d6877e88), 
    5./src/test/fuzz/fuzz: Running 1 inputs 1 time(s) each.
    6Running: ./timeout-ada0fecaba2b8c46c6e970cf637d9625b01bf7e5
    7ALARM: working on the last Unit for 19 seconds
    8       and the timeout value is 18 (use -timeout=N to change)
    9==375014== ERROR: libFuzzer: timeout after 19 seconds
    
  2. fuzz: p2p: Detect peer deadlocks fae1e7e012
  3. DrahtBot commented at 12:15 pm on December 6, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK dergoegge, naumenkogs, brunoerg
    Concept ACK furszy

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28960 (kernel: Remove dependency on CScheduler by TheCharlatan)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. DrahtBot added the label Tests on Dec 6, 2023
  5. furszy commented at 12:28 pm on December 6, 2023: member
    Concept ACK
  6. dergoegge commented at 2:07 pm on December 6, 2023: member
    Should we do the same for process_message? would need to change it to use ProcessMessagesOnce as well.
  7. maflcko commented at 2:41 pm on December 6, 2023: member

    Should we do the same for process_message? would need to change it to use ProcessMessagesOnce as well.

    Happy to review a pull request, or happy to include any patch here, that compiles, if someone writes it.

  8. fuzz: Detect deadlocks in process_message 9f265d8825
  9. dergoegge commented at 4:12 pm on December 6, 2023: member

    Happy to review a pull request, or happy to include any patch here, that compiles, if someone writes it.

    feel free to pick https://github.com/dergoegge/bitcoin/commit/9f265d88253ed464413dea5614fa13dea0d8cfd5

  10. brunoerg commented at 6:39 pm on December 6, 2023: contributor
    Concept ACK
  11. naumenkogs commented at 8:24 am on December 7, 2023: member
    Concept ACK
  12. dergoegge approved
  13. dergoegge commented at 10:34 am on December 8, 2023: member
    ACK 9f265d88253ed464413dea5614fa13dea0d8cfd5
  14. DrahtBot requested review from furszy on Dec 8, 2023
  15. DrahtBot requested review from brunoerg on Dec 8, 2023
  16. DrahtBot requested review from naumenkogs on Dec 8, 2023
  17. fanquake commented at 11:10 am on December 8, 2023: member
    cc also @mzumsande @sipa
  18. fanquake closed this on Dec 8, 2023

  19. maflcko reopened this on Dec 8, 2023

  20. naumenkogs commented at 9:37 am on December 11, 2023: member
    ACK 9f265d88253ed464413dea5614fa13dea0d8cfd5
  21. DrahtBot removed review request from naumenkogs on Dec 11, 2023
  22. brunoerg approved
  23. brunoerg commented at 12:11 pm on December 11, 2023: contributor
    ACK 9f265d88253ed464413dea5614fa13dea0d8cfd5
  24. fanquake merged this on Dec 11, 2023
  25. fanquake closed this on Dec 11, 2023

  26. maflcko deleted the branch on Dec 11, 2023
  27. jamesob commented at 3:08 pm on December 12, 2023: contributor

    ACK 9f265d88253ed464413dea5614fa13dea0d8cfd5 (jamesob/ackr/29009.1.maflcko.fuzz_p2p_detect_peer_dea)

    Using the suggested buggy diff, I was able to reproduce a fuzz timeout locally:

     0SUMMARY: libFuzzer: timeout
     1================== Job 4 exited with exit code 70 ============
     2INFO: Running with entropic power schedule (0xFF, 100).
     3INFO: Seed: 3004856548
     4INFO: Loaded 1 modules   (549836 inline 8-bit counters): 549836 [0x55b1d6e36508, 0x55b1d6ebc8d4),
     5INFO: Loaded 1 PC tables (549836 PCs): 549836 [0x55b1d6ebc8d8,0x55b1d7720598),
     6INFO:     2124 files found in ../qa-assets/fuzz_seed_corpus/process_message
     7INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1047633 bytes
     8INFO: seed corpus: files: 2124 min: 1b max: 1047633b total: 78093906b rss: 247Mb
     9Warning: Please check that your computer's date and time are correct! If your clock is wrong, Bitcoin Core will not work properly.
    10ALARM: working on the last Unit for 1796 seconds
    11       and the timeout value is 1200 (use -timeout=N to change)
    12MS: 0 ; base unit: 0000000000000000000000000000000000000000
    13artifact_prefix='./'; Test unit written to ./timeout-3a21200cd77fa45164203d47b65985b623d18c8f
    14==1628346== ERROR: libFuzzer: timeout after 1796 seconds
    15    [#0](/bitcoin-bitcoin/0/) 0x55b1d2def21f  (/home/james/src/bitcoin/src/test/fuzz/fuzz+0x1db721f) (BuildId: 6352bec346d40adbe5b31bd18e73b640a40465c2)
    16
    17    [ ... snip ... ]
    18
    19    [#17](/bitcoin-bitcoin/17/) 0x7f8844188ccf  (/usr/lib/libc.so.6+0x27ccf) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
    20    [#18](/bitcoin-bitcoin/18/) 0x7f8844188d89  (/usr/lib/libc.so.6+0x27d89) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
    21    [#19](/bitcoin-bitcoin/19/) 0x55b1d2c9e304  (/home/james/src/bitcoin/src/test/fuzz/fuzz+0x1c66304) (BuildId: 6352bec346d40adbe5b31bd18e73b640a40465c2)
    22
    23SUMMARY: libFuzzer: timeout
    24FUZZ=process_message ./src/test/fuzz/fuzz  -rss_limit_mb=20000 -jobs=12  7716.06s user 734.55s system 463% cpu 30:21.38 total
    25
    26[1] 15:38:59 james@fido src/bitcoin (?± ackr/29009.1.maflcko.fuzz_p2p_detect_peer_dea 9f265d8) % base64 ./timeout-3a21200cd77fa45164203d47b65985b623d18c8f
    27
    28Z2V0ZGF0YQALkmNvbm7///r/+v///////////////2dldGEA//+JHgoBAABAAP//QTBloTBlMDp0
    29imRhdGEA/2VkdGFndGEA//+JHgoBAABAAAEwMP9lZ0FlMDp0imRhlZ4A/2VkdGFndGEA//+JHgIB
    30AABAAP9BKDAwMGV0ZGfL+SybqZYAFfABduEAD2dldGRhdGEBAABAZXRkYXRhAGP/c2VuZGFkZHJ2
    31MjB24f0BbH8AAGf///9x//////////8BAABA/x4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4e
    32Hh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4e////
    33//////////8D/+r/////BAIA//9B/9//MDAyMTM4NDAzMzYwZ2V0ZGF0YQB0AHRgBAD//4keAgD/
    34/2dljJthdGEA/////0H/AAAAAbNc/evr//8PAOvrrP///2VnbpYAFWH9////AQAAAAAAAAABZYsA
    35lP5kYXRhAGP/bH8DMDU1AGRhdH//WAj/ZAJldGRhYQB0dGD/f/94CIODg4ODg4OD////////////
    361f////////8AAAAAAAAAExMBAAAAAAAAAAAAAAAAhAAAAAAAAAAAAAAAAAAAAAAAOgAAAAD+//8a
    37AAAAZ2V0ZAAAAAAAAAB0YQ==
    
    0-----BEGIN PGP SIGNED MESSAGE-----
    1Hash: SHA512
    2
    3ACK 9f265d88253ed464413dea5614fa13dea0d8cfd5 ([`jamesob/ackr/29009.1.maflcko.fuzz_p2p_detect_peer_dea`](https://github.com/jamesob/bitcoin/tree/ackr/29009.1.maflcko.fuzz_p2p_detect_peer_dea))
    4
    5Using the suggested buggy diff, I was able to reproduce a fuzz timeout locally:
    

    SUMMARY: libFuzzer: timeout ================== Job 4 exited with exit code 70 ============ INFO: Running with entropic power schedule (0xFF, 100). INFO: Seed: 3004856548 INFO: Loaded 1 modules (549836 inline 8-bit counters): 549836 [0x55b1d6e36508, 0x55b1d6ebc8d4), INFO: Loaded 1 PC tables (549836 PCs): 549836 [0x55b1d6ebc8d8,0x55b1d7720598), INFO: 2124 files found in ../qa-assets/fuzz_seed_corpus/process_message INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1047633 bytes INFO: seed corpus: files: 2124 min: 1b max: 1047633b total: 78093906b rss: 247Mb Warning: Please check that your computer’s date and time are correct! If your clock is wrong, Bitcoin Core will not work properly. ALARM: working on the last Unit for 1796 seconds and the timeout value is 1200 (use -timeout=N to change) MS: 0 ; base unit: 0000000000000000000000000000000000000000 artifact_prefix=’./’; Test unit written to ./timeout-3a21200cd77fa45164203d47b65985b623d18c8f ==1628346== ERROR: libFuzzer: timeout after 1796 seconds #0 0x55b1d2def21f (/home/james/src/bitcoin/src/test/fuzz/fuzz+0x1db721f) (BuildId: 6352bec346d40adbe5b31bd18e73b640a40465c2)

    [ ... snip ... ]
    
    [#17](/bitcoin-bitcoin/17/) 0x7f8844188ccf  (/usr/lib/libc.so.6+0x27ccf) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
    [#18](/bitcoin-bitcoin/18/) 0x7f8844188d89  (/usr/lib/libc.so.6+0x27d89) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
    [#19](/bitcoin-bitcoin/19/) 0x55b1d2c9e304  (/home/james/src/bitcoin/src/test/fuzz/fuzz+0x1c66304) (BuildId: 6352bec346d40adbe5b31bd18e73b640a40465c2)
    

    SUMMARY: libFuzzer: timeout FUZZ=process_message ./src/test/fuzz/fuzz -rss_limit_mb=20000 -jobs=12 7716.06s user 734.55s system 463% cpu 30:21.38 total

    [1] 15:38:59 james@fido src/bitcoin (?± ackr/29009.1.maflcko.fuzz_p2p_detect_peer_dea 9f265d8) % base64 ./timeout-3a21200cd77fa45164203d47b65985b623d18c8f

    Z2V0ZGF0YQALkmNvbm7///r/+v///////////////2dldGEA//+JHgoBAABAAP//QTBloTBlMDp0 imRhdGEA/2VkdGFndGEA//+JHgoBAABAAAEwMP9lZ0FlMDp0imRhlZ4A/2VkdGFndGEA//+JHgIB AABAAP9BKDAwMGV0ZGfL+SybqZYAFfABduEAD2dldGRhdGEBAABAZXRkYXRhAGP/c2VuZGFkZHJ2 MjB24f0BbH8AAGf///9x//////////8BAABA/x4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4e Hh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4e//// //////////8D/+r/////BAIA//9B/9//MDAyMTM4NDAzMzYwZ2V0ZGF0YQB0AHRgBAD//4keAgD/ /2dljJthdGEA/////0H/AAAAAbNc/evr//8PAOvrrP///2VnbpYAFWH9////AQAAAAAAAAABZYsA lP5kYXRhAGP/bH8DMDU1AGRhdH//WAj/ZAJldGRhYQB0dGD/f/94CIODg4ODg4OD//////////// 1f////////8AAAAAAAAAExMBAAAAAAAAAAAAAAAAhAAAAAAAAAAAAAAAAAAAAAAAOgAAAAD+//8a AAAAZ2V0ZAAAAAAAAAB0YQ==

     0
     1
     2-----BEGIN PGP SIGNATURE-----
     3
     4iQIzBAEBCgAdFiEEGNRVI1NPYuZCSIrGepNdrbLETwUFAmV4d6oACgkQepNdrbLE
     5TwXLkBAAk/7T4AcyzzjRrAaKvF+wShsYO+MA/FbLY6OuwehUlKemRFCj3/CoYr25
     6u/K955m/RA9VOR/OZZH3lohNZMhA01yEbaIWzvYe2ZBeXvTNOBEQgPr61WBmY6tC
     7wTXRBa5kWwAc5JMfksKdvxgDrVRRQalwoqSS2r1fmfNVqF5zmgYwSVYClJuHfwuO
     8tTrZFYZwGOdpgRSHux2os51oGep7TJhA+xxcM0ZunsPHqCWm6DD7JiVJhoNk+qG+
     9YJSjXUJjH9qM9VJ8xOziEtESDGiMCHY5XVYzw6jhYwjUnk/NdGUsAWdM6x1mUx35
    10+wus5szjxkyb5ABEGtPggJSiEBxCoZ3i6fTzT3kK4QNqalp+Qho/3Q6oDLEhVnuk
    11q1440aw9BR+hQbDdlZjWxSjctAzF7b0BP4Km0VBYZjKbzgpLUKgUqo3rGCF89MZj
    12c9BoyY9FXIYKhFBhs0DMSArvpjH8B7h5e7HBMYxRQSTT6kldYwKwxJkXohOO5fvO
    13tDX0rQIFqP+vdQpBtNxxRVpkOsLof39quUVZwIFGfcf2OgwzdVn/vo70kR1IfUjh
    14LonnxfA37AQl8/PwI4Zztet2nna5Ao2BceB3xfzhuvkb8UH/H/MwSAkcH9yFoyUU
    15YvssqGIqGejYhMSNOMAfX57QI3WHAXHmTFEr/qcNJVtKvYIWgIg=
    16=5BSl
    17-----END PGP SIGNATURE-----
    
    0Tested on Linux-6.5.5-arch1-1-x86_64-with-glibc2.38
    1
    2Configured with ./configure 'BDB_LIBS=-L/home/james/src/bitcoin/db4/lib -ldb_cxx-4.8' BDB_CFLAGS=-I/home/james/src/bitcoin/db4/include 'CXXFLAGS=-fPIE -pipe -O2 -g -Wthread-safety-analysis -Wall -Werror=sign-compare ' --disable-gui-tests --enable-fuzz --with-sanitizers=address,fuzzer,undefined --enable-wallet --enable-debug --with-daemon --enable-natpmp-default
    3
    4Compiled with /usr/bin/ccache /usr/bin/clang++ -std=c++17 -mavx -mavx2 -mpclmul -fPIE -pipe -O2 -g -Wthread-safety-analysis -Wall -Werror=sign-compare  -O0 -g3 -ftrapv -fdebug-prefix-map=$(abs_top_srcdir)=.  -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -fsanitize=address,fuzzer,undefined -msse4.1 -msse4.2 -msse4 -msha  i
    5
    6Compiler version: clang version 16.0.6
    
  28. Julio-Rats referenced this in commit 00e0658e77 on Jan 26, 2024
  29. bitcoin locked this on Dec 11, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-22 06:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me