Prevent valgrind false positive in rest_blockhash_by_height #18785

pull ryanofsky wants to merge 1 commits into bitcoin:master from ryanofsky:pr/grind changing 1 files +1 −1
  1. ryanofsky commented at 7:20 pm on April 27, 2020: member

    A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2 optimizations makes valgrind misleadingly imply C++ code is reading an uninitialized blockheight value in rest_blockhash_by_height just because that’s what clang optimized code is doing. The C++ code looks like:

    0int32_t blockheight;
    1if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
    

    while the optimized code looks like:

    00x00000000000f97ab <+123>:   callq  0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
    10x00000000000f97b0 <+128>:   mov    0xc(%rsp),%ebx
    20x00000000000f97b4 <+132>:   test   %ebx,%ebx
    30x00000000000f97b6 <+134>:   js     0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
    40x00000000000f97bc <+140>:   xor    $0x1,%al
    50x00000000000f97be <+142>:   jne    0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
    

    During the rest_interface.py test:

    https://github.com/bitcoin/bitcoin/blob/eef90c14ed0f559e3f6e187341009270b84f45cb/test/functional/interface_rest.py#L266

    when height_str is empty, ParseInt32 returns false and blockheight value is never assigned. The optimized code reads the uninitialized blockheight value in 0xc(%rsp) before the checking the ParseInt32 return value in %al, which is harmless, but triggers the following error from valgrind:

     0==30660== Thread 13 b-httpworker.2:
     1==30660== Conditional jump or move depends on uninitialised value(s)
     2==30660==    at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
     3==30660==    by 0x2041B9: operator() (rest.cpp:670)
     4==30660==    by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
     5==30660==    by 0x3EC994: operator() (std_function.h:706)
     6==30660==    by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
     7==30660==    by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
     8==30660==    by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
     9==30660==    by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
    10==30660==    by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
    11==30660==    by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
    12==30660==    by 0x3EDAAA: operator() (thread:243)
    13==30660==    by 0x3EDAAA: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
    14==30660==    by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
    15==30660==    by 0x54876DA: start_thread (pthread_create.c:463)
    16==30660==    by 0x6DC888E: clone (clone.S:95)
    17==30660==  Uninitialised value was created by a stack allocation
    18==30660==    at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
    19==30660==
    20{
    21   <insert_a_suppression_name_here>
    22   Memcheck:Cond
    23   fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
    24   fun:operator()
    25   fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
    26   fun:operator()
    27   fun:_ZN12HTTPWorkItemclEv
    28   fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
    29   fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
    30   fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
    31   fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
    32   fun:_M_invoke<0, 1, 2>
    33   fun:operator()
    34   fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
    35   obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
    36   fun:start_thread
    37   fun:clone
    38}
    

    This is a known bad interaction between clang and valgrind. The clang optimized code is correct but valgrind has no way of knowing that accessing the uninitialized value isn’t a problem. Issue has been reported previously:

    This commit just sets blockheight to -1 as a workaround.

    This change was originally made in 41d5d651594c6c939add7a58b7e30c97dccdf24a from #18740 to fix the travis error there (https://travis-ci.org/github/bitcoin/bitcoin/jobs/678453061#L7157) but MarcoFalke suggested #18740 (review) moving to a new PR, since apparently the error’s been seen on travis previously

  2. Prevent valgrind false positive in rest_blockhash_by_height
    A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2
    optimizations makes valgrind misleadingly imply C++ code is reading an
    uninitialized blockheight value in rest_blockhash_by_height just because that's
    what clang optimized code is doing. The C++ code looks like:
    
        int32_t blockheight;
        if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
    
    while the optimized code looks like:
    
        0x00000000000f97ab <+123>:   callq  0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
        0x00000000000f97b0 <+128>:   mov    0xc(%rsp),%ebx
        0x00000000000f97b4 <+132>:   test   %ebx,%ebx
        0x00000000000f97b6 <+134>:   js     0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
        0x00000000000f97bc <+140>:   xor    $0x1,%al
        0x00000000000f97be <+142>:   jne    0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
    
    During the rest_interface.py test:
    
       self.test_rest_request("/blockhashbyheight/", ret_type=RetType.OBJ, status=400)
    
    when height_str is empty, ParseInt32 returns false and blockheight value is
    never assigned. The optimized code reads the uninitialized blockheight value
    in 0xc(%rsp) before the checking the ParseInt32 return value in %al, which is
    harmless, but triggers the following error from valgrind:
    
    ==30660== Thread 13 b-httpworker.2:
    ==30660== Conditional jump or move depends on uninitialised value(s)
    ==30660==    at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
    ==30660==    by 0x2041B9: operator() (rest.cpp:670)
    ==30660==    by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
    ==30660==    by 0x3EC994: operator() (std_function.h:706)
    ==30660==    by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
    ==30660==    by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
    ==30660==    by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
    ==30660==    by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
    ==30660==    by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
    ==30660==    by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
    ==30660==    by 0x3EDAAA: operator() (thread:243)
    ==30660==    by 0x3EDAAA: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
    ==30660==    by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
    ==30660==    by 0x54876DA: start_thread (pthread_create.c:463)
    ==30660==    by 0x6DC888E: clone (clone.S:95)
    ==30660==  Uninitialised value was created by a stack allocation
    ==30660==    at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
    ==30660==
    {
       <insert_a_suppression_name_here>
       Memcheck:Cond
       fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
       fun:operator()
       fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
       fun:operator()
       fun:_ZN12HTTPWorkItemclEv
       fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
       fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
       fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
       fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
       fun:_M_invoke<0, 1, 2>
       fun:operator()
       fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
       obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
       fun:start_thread
       fun:clone
    }
    
    This is a known bad interaction between clang and valgrind. The clang optimized
    code is correct but valgrind has no way of knowing that accessing the
    uninitialized value isn't a problem. Issue has been reported previously:
    
        https://bugs.llvm.org/show_bug.cgi?id=32604#c4
        https://github.com/Z3Prover/z3/issues/972
    
    This commit just sets blockheight to 0 as a workaround.
    fcb7261625
  3. in src/rest.cpp:610 in 86433fd89d outdated
    606@@ -607,7 +607,7 @@ static bool rest_blockhash_by_height(HTTPRequest* req,
    607     std::string height_str;
    608     const RetFormat rf = ParseDataFormat(height_str, str_uri_part);
    609 
    610-    int32_t blockheight;
    611+    int32_t blockheight = 0;
    


    MarcoFalke commented at 7:30 pm on April 27, 2020:
    0    int32_t blockheight = 0; // Initialization only needed as workaround for -O2 compiler optimization and valgrind interaction, see [#18785](/bitcoin-bitcoin/18785/)
    

    This will prevent valgrind from detecting real issues, so maybe add a comment with motivation?


    ryanofsky commented at 8:02 pm on April 27, 2020:

    re: https://github.com/bitcoin/bitcoin/pull/18785/files#r416090195

    This will prevent valgrind from detecting real issues, so maybe add a comment with motivation?

    Thanks, added comment. I’m not sure if it is too likely to be helpful to someone reading the code, but I guess it’s better to have an explanation for something you wouldn’t expect to be necessary.

    I also changed init value from 0 to -1 just to make it clear that if height is not parsed it will always trigger the error, regardless of the bool return value, and there’s no strange case where this is supposed to return information about block 0

  4. MarcoFalke approved
  5. MarcoFalke commented at 7:31 pm on April 27, 2020: member
    ACK. Nice catch. I stared at this code for at least half an hour, not knowing that the code is right to begin with.
  6. ryanofsky force-pushed on Apr 27, 2020
  7. ryanofsky commented at 8:04 pm on April 27, 2020: member
    Updated 86433fd89d4c52064b8c3b59cba0fcd18fef15ca -> fcb72616253ed22e364bc312992d77efc1c4a3c1 (pr/grind.1 -> pr/grind.2, compare) with explanation / less confusing init value
  8. MarcoFalke commented at 8:14 pm on April 27, 2020: member
    ACK fcb72616253ed22e364bc312992d77efc1c4a3c1
  9. DrahtBot added the label RPC/REST/ZMQ on Apr 27, 2020
  10. DrahtBot commented at 11:05 pm on April 27, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #18740 (Remove g_rpc_node global by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  11. practicalswift commented at 6:29 am on April 28, 2020: contributor

    ACK fcb72616253ed22e364bc312992d77efc1c4a3c1

    Note that MemorySanitizer is not fooled by this type of optimization.

    MemorySanitizer can be trivially be enabled in Travis by merging #18288. Review welcome :)

    Valgrind has well documented optimization issues. From the Valgrind documentation:

    If you are planning to use Memcheck: On rare occasions, compiler optimisations (at -O2 and above, and sometimes -O1) have been observed to generate code which fools Memcheck into wrongly reporting uninitialised value errors, or missing uninitialised value errors. We have looked in detail into fixing this, and unfortunately the result is that doing so would give a further significant slowdown in what is already a slow tool. So the best solution is to turn off optimisation altogether. Since this often makes things unmanageably slow, a reasonable compromise is to use -O. This gets you the majority of the benefits of higher optimisation levels whilst keeping relatively small the chances of false positives or false negatives from Memcheck. Also, you should compile your code with -Wall because it can identify some or all of the problems that Valgrind can miss at the higher optimisation levels. (Using -Wall is also a good idea in general.)

  12. MarcoFalke merged this on Apr 29, 2020
  13. MarcoFalke closed this on Apr 29, 2020

  14. sidhujag referenced this in commit 7e91f6db54 on Apr 29, 2020
  15. fanquake referenced this in commit 708e3c7e85 on May 5, 2020
  16. MarcoFalke referenced this in commit 52ce396b2a on May 5, 2020
  17. sidhujag referenced this in commit 07718ed7f0 on May 5, 2020
  18. PastaPastaPasta referenced this in commit 9f2d812b7d on Jun 27, 2021
  19. PastaPastaPasta referenced this in commit 43eeca6d47 on Jun 28, 2021
  20. PastaPastaPasta referenced this in commit 278ec083a7 on Jun 29, 2021
  21. PastaPastaPasta referenced this in commit e9e3cadc79 on Jul 1, 2021
  22. PastaPastaPasta referenced this in commit 22a576b4bd on Jul 1, 2021
  23. PastaPastaPasta referenced this in commit d54f195f3d on Jul 14, 2021
  24. DrahtBot locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-01 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me