net: Make DNS lookup mockable, add fuzzing harness #19415

pull practicalswift wants to merge 2 commits into bitcoin:master from practicalswift:fuzzers-dns-lookup changing 4 files +154 −61
  1. practicalswift commented at 10:33 pm on June 29, 2020: contributor

    Make DNS lookup mockable/testable/fuzzable.

    Add fuzzing harness for Lookup(…)/LookupHost(…)/LookupNumeric(…)/LookupSubNet(…).

    See doc/fuzzing.md for information on how to fuzz Bitcoin Core. Don’t forget to contribute any coverage increasing inputs you find to the Bitcoin Core fuzzing corpus repo.

    Happy fuzzing :)

  2. DrahtBot added the label Build system on Jun 29, 2020
  3. DrahtBot added the label P2P on Jun 29, 2020
  4. DrahtBot added the label Tests on Jun 29, 2020
  5. MarcoFalke removed the label Build system on Jun 29, 2020
  6. MarcoFalke removed the label Tests on Jun 29, 2020
  7. MarcoFalke renamed this:
    tests: Make DNS lookup mockable/testable/fuzzable. Add fuzzing harness for Lookup(…)/LookupHost(…)/LookupNumeric(…)/LookupSubNet(…).
    net: Make DNS lookup mockable/testable/fuzzable. Add fuzzing harness for Lookup(…)/LookupHost(…)/LookupNumeric(…)/LookupSubNet(…).
    on Jun 29, 2020
  8. MarcoFalke renamed this:
    net: Make DNS lookup mockable/testable/fuzzable. Add fuzzing harness for Lookup(…)/LookupHost(…)/LookupNumeric(…)/LookupSubNet(…).
    net: Make DNS lookup mockable, add fuzzing harness
    on Jun 29, 2020
  9. DrahtBot commented at 2:48 am on June 30, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #21328 (net, refactor: pass uint16 CService::port as uint16 by jonatack)
    • #17160 (refactor: net: subnet lookup: use single-result LookupHost() by theStack)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  10. practicalswift force-pushed on Jun 30, 2020
  11. naumenkogs commented at 12:50 pm on June 30, 2020: member
    Concept ACK
  12. DrahtBot added the label Needs rebase on Jul 9, 2020
  13. practicalswift force-pushed on Jul 9, 2020
  14. DrahtBot removed the label Needs rebase on Jul 9, 2020
  15. practicalswift force-pushed on Jul 10, 2020
  16. DrahtBot added the label Needs rebase on Jul 11, 2020
  17. practicalswift force-pushed on Jul 11, 2020
  18. DrahtBot removed the label Needs rebase on Jul 11, 2020
  19. in src/netbase.cpp:65 in a9582c7dc1 outdated
    82-            vIP.push_back(addr);
    83-            return true;
    84-        }
    85-    }
    86-
    87+static Optional<std::vector<CNetAddr>> LookupInternGetAddrInfo(const std::string& name, bool fAllowLookup) {
    


    Crypt-iQ commented at 1:07 am on August 3, 2020:
    This is changed in bfc1455 to WrappedGetAddrInfo, why not do that here?

    practicalswift commented at 4:33 pm on August 3, 2020:

    I could have done it already in the earlier commit. I suggest keeping the current commit order in not invalidate the ACK. Makes sense? :)

    Thanks a lot for reviewing BTW ❤️


    Crypt-iQ commented at 6:54 pm on August 3, 2020:
    Yeah that makes sense. Fuzzing is a labor of love
  20. Crypt-iQ approved
  21. Crypt-iQ commented at 1:12 am on August 3, 2020: contributor
    Tested ACK 426617a. Coverage for ~48 hours of libfuzzer fuzzing here: https://crypt-iq.github.io/netbase_dns_cov_outs/src/netbase.cpp.gcov.html
  22. DrahtBot added the label Needs rebase on Aug 25, 2020
  23. practicalswift force-pushed on Aug 25, 2020
  24. DrahtBot removed the label Needs rebase on Aug 25, 2020
  25. jamesob commented at 11:26 am on August 27, 2020: member
    Concept ACK
  26. Crypt-iQ commented at 1:39 am on September 1, 2020: contributor

    ACK 954fbb74971a9056043f6f98e09fb8f175244031

    Code change is straightforward to reason about. Ubuntu 18:

    • ./configure --enable-fuzz --with-sanitizers=address,undefined,integer,fuzzer reports no errors
    • valgrind reports no errors

    macOS v10.15.4:

    • ./configure --enable-fuzz --with-sanitizers=address,fuzzer --disable-asm reports no errors
    • ./configure --enable-fuzz --with-sanitizers=undefined,integer,fuzzer reports the following:
     0netbase.cpp:240:37: runtime error: implicit conversion from type 'int' of value -1877995509 (32-bit, signed) to type 'uint16_t' (aka 'unsigned short') changed the value to 4107 (16-bit, unsigned)
     1    [#0](/bitcoin-bitcoin/0/) 0x109a6b73b in Lookup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<CService, std::__1::allocator<CService> >&, int, bool, unsigned int, std::__1::function<boost::optional<std::__1::vector<CNetAddr, std::__1::allocator<CNetAddr> > > (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)>)+0x7ab (netbase_dns_lookup:x86_64+0x101c0273b)
     2    [#1](/bitcoin-bitcoin/1/) 0x109a6cad8 in Lookup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, CService&, int, bool, std::__1::function<boost::optional<std::__1::vector<CNetAddr, std::__1::allocator<CNetAddr> > > (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)>)+0x338 (netbase_dns_lookup:x86_64+0x101c03ad8)
     3    [#2](/bitcoin-bitcoin/2/) 0x107e73228 in test_one_input(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&)+0x5d8 (netbase_dns_lookup:x86_64+0x10000a228)
     4    [#3](/bitcoin-bitcoin/3/) 0x10a304841 in LLVMFuzzerTestOneInput+0x151 (netbase_dns_lookup:x86_64+0x10249b841)
     5    [#4](/bitcoin-bitcoin/4/) 0x10a66fac0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:556
     6    [#5](/bitcoin-bitcoin/5/) 0x10a66f205 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:470
     7    [#6](/bitcoin-bitcoin/6/) 0x10a6718a7 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) FuzzerLoop.cpp:765
     8    [#7](/bitcoin-bitcoin/7/) 0x10a671c09 in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) FuzzerLoop.cpp:792
     9    [#8](/bitcoin-bitcoin/8/) 0x10a65f3ed in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:829
    10    [#9](/bitcoin-bitcoin/9/) 0x10a68a0b2 in main FuzzerMain.cpp:19
    11    [#10](/bitcoin-bitcoin/10/) 0x7fff73778cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
    12
    13SUMMARY: UndefinedBehaviorSanitizer: implicit-signed-integer-truncation netbase.cpp:240:37 in
    
  27. practicalswift force-pushed on Sep 17, 2020
  28. practicalswift commented at 1:15 pm on September 17, 2020: contributor

    Had to rebase after the merge of 15c27c44417ab77a660b53b8574f7eb5261b19f8.

    Changed:

     0diff --git a/src/Makefile.test.include b/src/Makefile.test.include
     1index 73eb9ad56..d9bf704bd 100644
     2--- a/src/Makefile.test.include
     3+++ b/src/Makefile.test.include
     4@@ -755,7 +755,7 @@ test_fuzz_netaddress_SOURCES = test/fuzz/netaddress.cpp
     5 test_fuzz_netbase_dns_lookup_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
     6 test_fuzz_netbase_dns_lookup_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
     7 test_fuzz_netbase_dns_lookup_LDADD = $(FUZZ_SUITE_LD_COMMON)
     8-test_fuzz_netbase_dns_lookup_LDFLAGS = $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS)
     9+test_fuzz_netbase_dns_lookup_LDFLAGS = $(FUZZ_SUITE_LDFLAGS_COMMON)
    10 test_fuzz_netbase_dns_lookup_SOURCES = test/fuzz/netbase_dns_lookup.cpp
    11
    12 test_fuzz_out_point_deserialize_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES) -DOUT_POINT_DESERIALIZE=1
    

    @Crypt-iQ Thanks a lot for reviewing. May I ask you a re-review and hopefully give your third Tested ACK for this change? :)

  29. Crypt-iQ commented at 10:18 am on September 20, 2020: contributor
    ACK 6f9bc3f
  30. in src/netbase.cpp:225 in 6f9bc3f67b outdated
    221@@ -204,7 +222,7 @@ bool LookupHost(const std::string& name, CNetAddr& addr, bool fAllowLookup)
    222  * @returns Whether or not the service string successfully resolved to any
    223  *          resulting services.
    224  */
    225-bool Lookup(const std::string& name, std::vector<CService>& vAddr, int portDefault, bool fAllowLookup, unsigned int nMaxSolutions)
    226+bool Lookup(const std::string& name, std::vector<CService>& vAddr, int portDefault, bool fAllowLookup, unsigned int nMaxSolutions, std::function<Optional<std::vector<CNetAddr>>(const std::string&, bool)> mockable_dns_lookup_function)
    


    D4nte commented at 11:04 am on September 20, 2020:
    camelCase is used for the other arguments, is it fine to have an odd snake case argument?

    jonatack commented at 6:10 am on September 21, 2020:
    Yes, for new code it’s preferred. See doc/developer-notes.md, “Coding Style (C++), Symbol naming conventions”.
  31. D4nte commented at 5:07 am on September 21, 2020: none

    6f9bc3f67bd177cc7a5dc4f93d0c20c2d23307ec

    No issue found when fuzzing with ./configure --enable-fuzz --with-sanitizers=address,fuzzer CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm for 6 hours.

    I tried to run the fuzz on mac os with ./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm

     0 ./src/test/fuzz/process_message
     1/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment
     20x000000000001: note: pointer points here
     3<memory cannot be printed>
     4SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44 in
     5AddressSanitizer:DEADLYSIGNAL
     6=================================================================
     7==4275==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fff6c3d7712 bp 0x7ffeef323bd0 sp 0x7ffeef323ba0 T0)
     8==4275==The signal is caused by a READ memory access.
     9==4275==Hint: address points to the zero page.
    10    [#0](/bitcoin-bitcoin/0/) 0x7fff6c3d7712 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a (libc++.1.dylib:x86_64+0x38712)
    11    [#1](/bitcoin-bitcoin/1/) 0x101cc2d89 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) string:2730
    12    [#2](/bitcoin-bitcoin/2/) 0x101c3c557 in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
    13    [#3](/bitcoin-bitcoin/3/) 0x100ac6bee in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
    14    [#4](/bitcoin-bitcoin/4/) 0x100ac63c0 in InitLogging(ArgsManager const&) init.cpp:869
    15    [#5](/bitcoin-bitcoin/5/) 0x101da7053 in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
    16    [#6](/bitcoin-bitcoin/6/) 0x101da941f in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
    17    [#7](/bitcoin-bitcoin/7/) 0x1008dd4e4 in initialize() process_message.cpp:48
    18    [#8](/bitcoin-bitcoin/8/) 0x101e2f77c in LLVMFuzzerInitialize fuzz.cpp:52
    19    [#9](/bitcoin-bitcoin/9/) 0x102072f87 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
    20    [#10](/bitcoin-bitcoin/10/) 0x1020a08e2 in main FuzzerMain.cpp:19
    21    [#11](/bitcoin-bitcoin/11/) 0x7fff6f0d4cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
    22
    23==4275==Register values:
    24rax = 0x0000000000000007  rbx = 0x0000000000000000  rcx = 0x0000000000000007  rdx = 0x0000000103eb3081
    25rdi = 0x0000000000000001  rsi = 0x0000000000000000  rbp = 0x00007ffeef323bd0  rsp = 0x00007ffeef323ba0
    26 r8 = 0x0000000103eb3000   r9 = 0x0000008047835f00  r10 = 0x00007fff6f218a46  r11 = 0x0000000000000206
    27r12 = 0x0000000103eb3081  r13 = 0x0000000000000000  r14 = 0x0000000000000007  r15 = 0x0000000000000001
    28AddressSanitizer can not provide additional info.
    29SUMMARY: AddressSanitizer: SEGV (libc++.1.dylib:x86_64+0x38712) in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a
    30==4275==ABORTING
    31[1]    4275 abort      ./src/test/fuzz/process_message
    
  32. jonatack commented at 6:07 am on September 21, 2020: member
    Hi @D4nte, try adding --enable-c++17 (and if that still fails, perhaps add make distclean). Edit: oh, for macOS. Not my wheelhouse, but have a look at doc/fuzzing.md if the first ideas don’t help.
  33. D4nte commented at 6:37 am on September 21, 2020: none

    Hi @D4nte, try adding --enable-c++17 (and if that still fails, perhaps add make distclean). Edit: oh, for macOS. Not my wheelhouse, but have a look at doc/fuzzing.md if the first ideas don’t help.

    Thanks @jonatack, I’ll investigate and try again tomorrow. I am not yet that familiar with fuzzing.

  34. Crypt-iQ commented at 9:06 am on September 21, 2020: contributor
    @D4nte for macOS builds, specifying the address and undefined sanitizers won’t work. This is tracked here https://github.com/bitcoin/bitcoin/issues/19789
  35. D4nte commented at 4:46 am on September 22, 2020: none

    @D4nte for macOS builds, specifying the address and undefined sanitizers won’t work. This is tracked here #19789

    Thanks. Ran with ./configure --enable-fuzz --with-sanitizers=fuzzer,undefined CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm for 6 hours no issue encountered.

  36. Crypt-iQ commented at 11:57 am on September 22, 2020: contributor

    @D4nte for macOS builds, specifying the address and undefined sanitizers won’t work. This is tracked here #19789

    Thanks. Ran with ./configure --enable-fuzz --with-sanitizers=fuzzer,undefined CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm for 6 hours no issue encountered.

    Keep your seeds! We should PR them to bitcoin-core/qa-assets after this is merged.

  37. D4nte commented at 4:50 am on September 23, 2020: none

    @D4nte for macOS builds, specifying the address and undefined sanitizers won’t work. This is tracked here #19789

    Thanks. Ran with ./configure --enable-fuzz --with-sanitizers=fuzzer,undefined CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm for 6 hours no issue encountered.

    Keep your seeds! We should PR them to bitcoin-core/qa-assets after this is merged.

    Ok thanks, I did not delete anything. I’ll prepare a branch/fork to PR them.

  38. practicalswift force-pushed on Sep 28, 2020
  39. practicalswift force-pushed on Sep 29, 2020
  40. practicalswift commented at 2:27 pm on September 29, 2020: contributor
    Updated: Added more assertions to the fuzzer to test relevant postconditions. Simplified the code that allows for mocking.
  41. DrahtBot added the label Needs rebase on Dec 15, 2020
  42. practicalswift force-pushed on Dec 16, 2020
  43. DrahtBot removed the label Needs rebase on Dec 16, 2020
  44. DrahtBot added the label Needs rebase on Dec 25, 2020
  45. practicalswift force-pushed on Dec 27, 2020
  46. DrahtBot removed the label Needs rebase on Dec 27, 2020
  47. in src/netbase.cpp:74 in daea0188c6 outdated
    69+    // IPv4,v6 addresses to vIP while respecting nMaxSolutions.
    70+    struct addrinfo *aiTrav = aiRes;
    71+    std::vector<CNetAddr> resolved_addresses;
    72+    while (aiTrav != nullptr)
    73+    {
    74+        CNetAddr resolved;
    


    Crypt-iQ commented at 10:02 pm on January 17, 2021:
    this is unused

    practicalswift commented at 9:54 pm on January 18, 2021:
    Thanks! Fixed!
  48. in src/test/fuzz/netbase_dns_lookup.cpp:6 in 63e1d7be1e outdated
    0@@ -0,0 +1,77 @@
    1+// Copyright (c) 2020 The Bitcoin Core developers
    2+// Distributed under the MIT software license, see the accompanying
    3+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    4+
    5+#include <netbase.h>
    6+#include <optional.h>
    


    Crypt-iQ commented at 9:48 pm on January 18, 2021:
    AFAICT optional.h isn’t used

    practicalswift commented at 9:54 pm on January 18, 2021:
    Thanks! Fixed!
  49. in src/test/fuzz/netbase_dns_lookup.cpp:1 in 63e1d7be1e outdated
    0@@ -0,0 +1,77 @@
    1+// Copyright (c) 2020 The Bitcoin Core developers
    


    Crypt-iQ commented at 9:50 pm on January 18, 2021:
    The year has updated 🎇

    practicalswift commented at 9:54 pm on January 18, 2021:
    Thanks! Fixed!
  50. Crypt-iQ commented at 9:50 pm on January 18, 2021: contributor
    Two more comments, running this patch.
  51. practicalswift force-pushed on Jan 18, 2021
  52. practicalswift force-pushed on Jan 18, 2021
  53. practicalswift commented at 9:59 pm on January 18, 2021: contributor
    @Crypt-iQ Thanks for reviewing. I believe all feedback should be addressed now: mind re-reviewing one last time (hopefully)? :)
  54. Crypt-iQ commented at 1:32 am on January 23, 2021: contributor

    I get the following complaint when running --with-sanitizers=integer because CService expects uint16_t

    0netbase.cpp:240:37: runtime error: implicit conversion from type 'int' of value 210314121 (32-bit, signed) to type 'uint16_t' (aka 'unsigned short') changed the value to 9097 (16-bit, unsigned)
    

    https://github.com/bitcoin/bitcoin/blob/1a22951f545b549b8b9a9266dc61a740a69d4dbe/src/netbase.cpp#L240

  55. Crypt-iQ commented at 3:39 pm on January 23, 2021: contributor
    Tested ACK 0f2cfb379d8a04efc15d14ee581d3d1ad8ccfd97
  56. practicalswift force-pushed on Jan 24, 2021
  57. DrahtBot added the label Needs rebase on Feb 11, 2021
  58. practicalswift force-pushed on Feb 11, 2021
  59. practicalswift force-pushed on Feb 11, 2021
  60. DrahtBot removed the label Needs rebase on Feb 11, 2021
  61. DrahtBot added the label Needs rebase on Feb 15, 2021
  62. practicalswift force-pushed on Feb 15, 2021
  63. DrahtBot removed the label Needs rebase on Feb 15, 2021
  64. in src/netbase.cpp:64 in 522b3b4bc2 outdated
    63+
    64+    struct addrinfo *aiRes = nullptr;
    65+    int nErr = getaddrinfo(name.c_str(), nullptr, &aiHint, &aiRes);
    66+    if (nErr)
    67+    {
    68+        return {};
    


    vasild commented at 1:31 pm on February 19, 2021:

    nit: opening { on the same line as if. Or, given that nErr is not used elsewhere maybe:

    0    if (getaddrinfo(name.c_str(), nullptr, &aiHint, &aiRes) != 0) {
    1        return {};
    
  65. in src/netbase.cpp:83 in 522b3b4bc2 outdated
    78+        {
    79+            assert(aiTrav->ai_addrlen >= sizeof(sockaddr_in));
    80+            resolved_addresses.emplace_back(((struct sockaddr_in*)(aiTrav->ai_addr))->sin_addr);
    81+        }
    82+        if (aiTrav->ai_family == AF_INET6)
    83+        {
    


    vasild commented at 1:39 pm on February 19, 2021:
    nits: this is not move-only change, so maybe worth fixing the style, opening { on the same line.
  66. in src/netbase.cpp:162 in 522b3b4bc2 outdated
    183+    for (const CNetAddr& resolved : dns_lookup_function(name, fAllowLookup))
    184     {
    185-        CNetAddr resolved;
    186-        if (aiTrav->ai_family == AF_INET)
    187+        if (nMaxSolutions == 0 || vIP.size() < nMaxSolutions)
    188         {
    


    vasild commented at 1:46 pm on February 19, 2021:
    nit: opening { on the same line
  67. in src/netbase.cpp:159 in 522b3b4bc2 outdated
    178-
    179-    // Traverse the linked list starting with aiTrav, add all non-internal
    180-    // IPv4,v6 addresses to vIP while respecting nMaxSolutions.
    181-    struct addrinfo *aiTrav = aiRes;
    182-    while (aiTrav != nullptr && (nMaxSolutions == 0 || vIP.size() < nMaxSolutions))
    183+    for (const CNetAddr& resolved : dns_lookup_function(name, fAllowLookup))
    


    vasild commented at 1:49 pm on February 19, 2021:

    We can quit from this loop earlier if we got enough addresses:

    0    for (const CNetAddr& resolved : dns_lookup_function(name, fAllowLookup)) { 
    1        if (nMaxSolutions > 0 && vIP.size() >= nMaxSolutions) { 
    2            break;
    3        } 
    4        /* Never allow resolving to an internal address. Consider any such result invalid */
    5        if (!resolved.IsInternal()) {
    6            vIP.push_back(resolved);
    7        }   
    8    } 
    

    practicalswift commented at 3:58 pm on February 22, 2021:
    Now updated as suggested.
  68. in src/test/fuzz/netbase_dns_lookup.cpp:69 in 522b3b4bc2 outdated
    63+        }
    64+    }
    65+    {
    66+        CService resolved_service = LookupNumeric(name, default_port, fuzzed_dns_lookup_function);
    67+        assert(!resolved_service.IsInternal());
    68+    }
    


    vasild commented at 2:44 pm on February 19, 2021:
    nit: here and elsewhere in the test, those extra { and } are not necessary.

    practicalswift commented at 4:00 pm on February 22, 2021:
    They are intentional scope delimiters :)
  69. vasild approved
  70. vasild commented at 3:08 pm on February 19, 2021: member

    ACK 522b3b4bc276378f93f1b6109be9c66a7f8bfd38

    Some nits below.

    Ran this for some time and submitted the generated seeds at https://github.com/bitcoin-core/qa-assets/pull/50.

  71. in src/netbase.h:83 in 522b3b4bc2 outdated
    60-bool Lookup(const std::string& name, CService& addr, int portDefault, bool fAllowLookup);
    61-bool Lookup(const std::string& name, std::vector<CService>& vAddr, int portDefault, bool fAllowLookup, unsigned int nMaxSolutions);
    62-CService LookupNumeric(const std::string& name, int portDefault = 0);
    63-bool LookupSubNet(const std::string& strSubnet, CSubNet& subnet);
    64+
    65+using DNSLookupFn = std::function<std::vector<CNetAddr>(const std::string&, bool)>;
    


    vasild commented at 3:28 pm on February 19, 2021:

    This is good as it is. Maybe it would be a bit more convenient to have a single global variable of this type (resolver function) which defaults to WrappedGetAddrInfo() and is changed in the tests. Something like:

    0DNSLookupFn g_dns_lookup = WrappedGetAddrInfo;
    1
    2bool LookupHost(..., DNSLookupFn dns_lookup_function = g_dns_lookup);
    3
    4// in the tests
    5g_dns_lookup = fuzzed_dns_lookup_function;
    

    vasild commented at 11:59 am on February 22, 2021:

    Actually, there is more to this: using such a global would allow testing methods that call Lookup*() themselves. In such cases it would not be possible for the test to override the dns resolver by calling Lookup*() with a non-default argument.

    For example CConnman::ConnectNode() calls Lookup() without passing a non-default resolver to it (of course). If we use g_dns_lookup, then a test could modify its value to point to a fuzzed resolver and then call CConnman::ConnectNode().


    practicalswift commented at 4:00 pm on February 22, 2021:
    Good points. Now using g_dns_lookup as suggested.
  72. vasild approved
  73. practicalswift force-pushed on Feb 22, 2021
  74. vasild approved
  75. vasild commented at 5:10 pm on February 22, 2021: member
    ACK 24e3ce3d51ea0ff7f19519f08db18e8df693473d
  76. DrahtBot added the label Needs rebase on Mar 3, 2021
  77. practicalswift force-pushed on Mar 3, 2021
  78. DrahtBot removed the label Needs rebase on Mar 3, 2021
  79. jonatack commented at 9:51 pm on March 6, 2021: member

    Concept ACK, will review.

    I get the following complaint when running --with-sanitizers=integer because CService expects uint16_t

    0netbase.cpp:240:37: runtime error: implicit conversion from type 'int' of value 210314121 (32-bit, signed) to type 'uint16_t' (aka 'unsigned short') changed the value to 9097 (16-bit, unsigned)
    

    https://github.com/bitcoin/bitcoin/blob/1a22951f545b549b8b9a9266dc61a740a69d4dbe/src/netbase.cpp#L240 @Crypt-iQ Good find. This should be fixed with #21328.

  80. in src/netbase.cpp:70 in 820843247a outdated
    65+    if (nErr) {
    66+        return {};
    67+    }
    68+
    69+    // Traverse the linked list starting with aiTrav, add all non-internal
    70+    // IPv4,v6 addresses to vIP while respecting nMaxSolutions.
    


    Crypt-iQ commented at 0:36 am on March 7, 2021:
    This comment could be clarified - vIP + nMaxSolutions aren’t used here

    practicalswift commented at 11:19 pm on March 8, 2021:
    Done!
  81. in src/test/fuzz/netbase_dns_lookup.cpp:22 in 173272bf92 outdated
    16+
    17+std::vector<CNetAddr> fuzzed_dns_lookup_function(const std::string& name, bool allow_lookup)
    18+{
    19+    std::vector<CNetAddr> resolved_addresses;
    20+    while (fuzzed_data_provider_ptr->ConsumeBool()) {
    21+        resolved_addresses.push_back(ConsumeNetAddr(*fuzzed_data_provider_ptr));
    


    Crypt-iQ commented at 0:42 am on March 7, 2021:
    After reading the getaddrinfo manpage, I realized ConsumeNetAddr can return CNetAddr with m_scope_id set for non link-local addresses, which is different from getaddrinfo behaviour. I’m not an expert here but I think there could be sanity-checking for that in ConsumeNetAddr.

    practicalswift commented at 10:41 pm on March 8, 2021:
    I don’t think that is likely to matter in practice (don’t hesitate to prove me wrong!). Leaving unchanged for now.
  82. Crypt-iQ commented at 0:43 am on March 7, 2021: contributor
    Some comments, will test again
  83. in src/netbase.cpp:65 in 173272bf92 outdated
    60+    // hostname lookups.
    61+    aiHint.ai_flags = allow_lookup ? AI_ADDRCONFIG : AI_NUMERICHOST;
    62+
    63+    struct addrinfo* aiRes = nullptr;
    64+    int nErr = getaddrinfo(name.c_str(), nullptr, &aiHint, &aiRes);
    65+    if (nErr) {
    


    jonatack commented at 6:07 pm on March 7, 2021:

    proposed comment (and maybe simplification, as we’re not using nErr for anything, otherwise if we keep nErr it can be const)

    0-    int nErr = getaddrinfo(name.c_str(), nullptr, &aiHint, &aiRes);
    1-    if (nErr) {
    2+    // getaddrinfo(3) returns 0 if it succeeds or a non-zero error code.
    3+    if (getaddrinfo(name.c_str(), nullptr, &aiHint, &aiRes)) {
    

    practicalswift commented at 11:05 pm on March 8, 2021:
    Made n_err const as suggested. Now using the more explicit if (n_err != 0) which I find easier to read.
  84. in src/netbase.cpp:80 in 173272bf92 outdated
    75+            assert(aiTrav->ai_addrlen >= sizeof(sockaddr_in));
    76+            resolved_addresses.emplace_back(((struct sockaddr_in*)(aiTrav->ai_addr))->sin_addr);
    77+        }
    78+        if (aiTrav->ai_family == AF_INET6) {
    79+            assert(aiTrav->ai_addrlen >= sizeof(sockaddr_in6));
    80+            struct sockaddr_in6* s6 = (struct sockaddr_in6*)aiTrav->ai_addr;
    


    jonatack commented at 6:14 pm on March 7, 2021:
    here and line 76, can we use named casts?

    practicalswift commented at 11:18 pm on March 8, 2021:
    Sure. Now using reinterpret_cast.
  85. in src/netbase.cpp:90 in 173272bf92 outdated
    85+    freeaddrinfo(aiRes);
    86+
    87+    return resolved_addresses;
    88+}
    89+
    90+DNSLookupFn g_dns_lookup = WrappedGetAddrInfo;
    


    jonatack commented at 6:25 pm on March 7, 2021:
    0DNSLookupFn g_dns_lookup{WrappedGetAddrInfo};
    
  86. in src/netbase.cpp:63 in 173272bf92 outdated
    58+    // If we don't allow lookups, then use the AI_NUMERICHOST flag for
    59+    // getaddrinfo to only decode numerical network addresses and suppress
    60+    // hostname lookups.
    61+    aiHint.ai_flags = allow_lookup ? AI_ADDRCONFIG : AI_NUMERICHOST;
    62+
    63+    struct addrinfo* aiRes = nullptr;
    


    jonatack commented at 6:28 pm on March 7, 2021:
    0    struct addrinfo* aiRes{nullptr};
    
  87. in src/netbase.cpp:71 in 173272bf92 outdated
    66+        return {};
    67+    }
    68+
    69+    // Traverse the linked list starting with aiTrav, add all non-internal
    70+    // IPv4,v6 addresses to vIP while respecting nMaxSolutions.
    71+    struct addrinfo* aiTrav = aiRes;
    


    jonatack commented at 6:28 pm on March 7, 2021:
    0    struct addrinfo* aiTrav{aiRes};
    
  88. in src/netbase.cpp:158 in 173272bf92 outdated
    193-            assert(aiTrav->ai_addrlen >= sizeof(sockaddr_in6));
    194-            struct sockaddr_in6* s6 = (struct sockaddr_in6*) aiTrav->ai_addr;
    195-            resolved = CNetAddr(s6->sin6_addr, s6->sin6_scope_id);
    196+    for (const CNetAddr& resolved : dns_lookup_function(name, fAllowLookup)) {
    197+        if (nMaxSolutions > 0 && vIP.size() >= nMaxSolutions) {
    198+            break;
    


    jonatack commented at 6:34 pm on March 7, 2021:
    Can this if (nMaxSolutions > 0 && vIP.size() >= nMaxSolutions) check be done before the loop?

    practicalswift commented at 11:03 pm on March 8, 2021:

    Not that I can see: vIP is added to within the loop.

    Please provide a diff explaining your suggestion if it still stands :)


    jonatack commented at 11:16 am on March 15, 2021:
    You’re right, thanks.
  89. in src/test/fuzz/netbase_dns_lookup.cpp:6 in 173272bf92 outdated
    0@@ -0,0 +1,76 @@
    1+// Copyright (c) 2021 The Bitcoin Core developers
    2+// Distributed under the MIT software license, see the accompanying
    3+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    4+
    5+#include <netbase.h>
    


    jonatack commented at 6:37 pm on March 7, 2021:

    suggestion, as this uses CNetAddr defined in netaddress.h

    0+#include <netaddress.h>
    

    practicalswift commented at 10:40 pm on March 8, 2021:
    Thanks! Addressed.
  90. jonatack commented at 6:42 pm on March 7, 2021: member

    Tested approach ACK, a few comments below.

     0$ FUZZ=netbase_dns_lookup src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/
     1INFO: Seed: 3479425365
     2INFO: Loaded 1 modules   (643152 inline 8-bit counters): 643152 [0x555d231dc188, 0x555d232791d8), 
     3INFO: Loaded 1 PC tables (643152 PCs): 643152 [0x555d232791d8,0x555d23c496d8), 
     4INFO:   129774 files found in ../qa-assets/fuzz_seed_corpus/
     5INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
     6INFO: seed corpus: files: 129774 min: 1b max: 3986616b total: 1875939121b rss: 231Mb
     7[#8192](/bitcoin-bitcoin/8192/)	pulse  cov: 970 ft: 1825 corp: 59/244b exec/s: 4096 rss: 337Mb
     8[#16384](/bitcoin-bitcoin/16384/)	pulse  cov: 1763 ft: 3612 corp: 98/809b exec/s: 4096 rss: 436Mb
     9[#32768](/bitcoin-bitcoin/32768/)	pulse  cov: 2090 ft: 6274 corp: 158/2458b exec/s: 4681 rss: 590Mb
    10[#65536](/bitcoin-bitcoin/65536/)	pulse  cov: 2121 ft: 7423 corp: 192/5608b exec/s: 5041 rss: 593Mb
    11[#129775](/bitcoin-bitcoin/129775/)	INITED cov: 2235 ft: 10320 corp: 297/984Kb exec/s: 3415 rss: 1271Mb
    12[#129813](/bitcoin-bitcoin/129813/)	REDUCE cov: 2235 ft: 10320 corp: 297/984Kb lim: 286065 exec/s: 3416 rss: 1271Mb L: 62/286047 MS: 3 ChangeBit-ChangeBit-EraseBytes-
    13.../...
    14[#113401432](/bitcoin-bitcoin/113401432/)	REDUCE cov: 2669 ft: 12318 corp: 458/295Kb lim: 1048576 exec/s: 1595 rss: 1271Mb L: 942/16122 MS: 4 CMP-ChangeBit-ChangeBit-EraseBytes- DE: "\x9e\xff\xff\xff"-
    15[#113449634](/bitcoin-bitcoin/113449634/)	REDUCE cov: 2669 ft: 12318 corp: 458/295Kb lim: 1048576 exec/s: 1595 rss: 1271Mb L: 2682/16122 MS: 2 ShuffleBytes-EraseBytes-
    16[#113480169](/bitcoin-bitcoin/113480169/)	REDUCE cov: 2669 ft: 12318 corp: 458/295Kb lim: 1048576 exec/s: 1595 rss: 1271Mb L: 7438/16122 MS: 5 ChangeBit-EraseBytes-PersAutoDict-InsertRepeatedBytes-InsertRepeatedBytes- DE: "\x00\x00bP\x01_\x86\xac"-
    
  91. practicalswift force-pushed on Mar 8, 2021
  92. practicalswift force-pushed on Mar 8, 2021
  93. net: Make DNS lookup code testable c6b4bfb4b3
  94. tests: Add fuzzing harness for Lookup(...)/LookupHost(...)/LookupNumeric(...)/LookupSubNet(...) e528075189
  95. practicalswift force-pushed on Mar 8, 2021
  96. practicalswift commented at 11:20 pm on March 8, 2021: contributor

    @jonatack @Crypt-iQ Thanks for reviewing. Feedback addressed.

    This PR should now hopefully be ready for final review :)

  97. Crypt-iQ commented at 11:39 pm on March 14, 2021: contributor
    cr ACK e5280751890b02abb558b37eb0e0401493f148b4
  98. vasild approved
  99. vasild commented at 10:58 am on March 15, 2021: member
    ACK e5280751890b02abb558b37eb0e0401493f148b4
  100. MarcoFalke merged this on Mar 15, 2021
  101. MarcoFalke closed this on Mar 15, 2021

  102. in src/test/fuzz/netbase_dns_lookup.cpp:16 in e528075189
    11+#include <cstdint>
    12+#include <string>
    13+#include <vector>
    14+
    15+namespace {
    16+FuzzedDataProvider* fuzzed_data_provider_ptr = nullptr;
    


    MarcoFalke commented at 11:10 am on March 15, 2021:
    style-nit: Any reason for this namespace and pointer? Looks like a lambda should be able to do the same in less code and smaller scope

    practicalswift commented at 3:31 pm on March 15, 2021:

    Using a lambda here is a good idea: less code, smaller scope and overall beautiful :)

    TBH I don’t remember why I didn’t do it that way nine months ago when this code was written: a lambda is strictly better AFAICT :)

    Address in PR #21443.

  103. jonatack commented at 11:18 am on March 15, 2021: member

    ACK e528075

    Thanks for the update and s/aiTrav/ai_trav/ was a nice touchup.

  104. sidhujag referenced this in commit 9013667d4c on Mar 15, 2021
  105. MarcoFalke referenced this in commit 9a5e097435 on Mar 15, 2021
  106. practicalswift deleted the branch on Apr 10, 2021
  107. Fabcien referenced this in commit 83a625be64 on Feb 21, 2022
  108. DrahtBot locked this on Aug 16, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 19:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me