ASAN use-after-free in `m_reconnections` #33615

I ran into this while shutting my node down during IBD. I think it's benign.

What happens:

• CConnman is started here.
  • This initializes semOutbound here.
• While CConnman is active, m_reconnections is added to in DisconnectNodes here.
• CConnman is stopped here
  • The call to Stop will call StopNodes which will reset semOutbound here.
• ~CConnman is invoked a few lines below here.
  • In addition to calling Stop a second time, it also destructs a possibly non-empty m_reconnections (a list of ReconnectionInfo which contains a pointer to the memory that semOutbound also pointed to) by calling clear(). To destroy each element of m_reconnections, it looks like it accesses the grant field but that memory was already freed.

I haven't been able to reproduce this reliably, it appears timing-dependent. I think this can be fixed by calling WITH_LOCK(m_reconnections_mutex, m_reconnections.clear()) before this line.

ASAN output:

=================================================================
==22244==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020018ba9d0 at pc 0x000100837e58 bp 0x00016f826280 sp 0x00016f826278
WRITE of size 8 at 0x6020018ba9d0 thread T0
==22244==WARNING: Can't write to symbolizer at fd 5
    [#0](/bitcoin-bitcoin/0/) 0x100837e54 in void std::__1::__destroy_at[abi:ne180100]<CConnman::ReconnectionInfo, 0>(CConnman::ReconnectionInfo*) construct_at.h:67
    [#1](/bitcoin-bitcoin/1/) 0x100837754 in std::__1::__list_imp<CConnman::ReconnectionInfo, std::__1::allocator<CConnman::ReconnectionInfo>>::clear() list:635
    [#2](/bitcoin-bitcoin/2/) 0x1008202b8 in CConnman::~CConnman() net.cpp:3526
    [#3](/bitcoin-bitcoin/3/) 0x10075802c in Shutdown(node::NodeContext&) init.cpp:328
    [#4](/bitcoin-bitcoin/4/) 0x1005decc0 in main bitcoind.cpp:292
    [#5](/bitcoin-bitcoin/5/) 0x1937bab48  (<unknown module>)

0x6020018ba9d0 is located 0 bytes inside of 8-byte region [0x6020018ba9d0,0x6020018ba9d8)
freed by thread T0 here:
    [#0](/bitcoin-bitcoin/0/) 0x1034ed2e4 in _ZdlPv+0x6c (libclang_rt.asan_osx_dynamic.dylib:arm64+0x612e4)
    [#1](/bitcoin-bitcoin/1/) 0x10081fe9c in CConnman::StopNodes() net.cpp:3511
    [#2](/bitcoin-bitcoin/2/) 0x100757ebc in Shutdown(node::NodeContext&) init.cpp:315
    [#3](/bitcoin-bitcoin/3/) 0x1005decc0 in main bitcoind.cpp:292
    [#4](/bitcoin-bitcoin/4/) 0x1937bab48  (<unknown module>)

previously allocated by thread T0 here:
    [#0](/bitcoin-bitcoin/0/) 0x1034ececc in _Znwm+0x6c (libclang_rt.asan_osx_dynamic.dylib:arm64+0x60ecc)
    [#1](/bitcoin-bitcoin/1/) 0x10081c77c in CConnman::Start(CScheduler&, CConnman::Options const&) net.cpp:3361
    [#2](/bitcoin-bitcoin/2/) 0x10077d364 in AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) init.cpp:2183
    [#3](/bitcoin-bitcoin/3/) 0x1005debd0 in main bitcoind.cpp:288
    [#4](/bitcoin-bitcoin/4/) 0x1937bab48  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free construct_at.h:67 in void std::__1::__destroy_at[abi:ne180100]<CConnman::ReconnectionInfo, 0>(CConnman::ReconnectionInfo*)
Shadow bytes around the buggy address:
  0x6020018ba700: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x6020018ba780: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x6020018ba800: fa fa fd fd fa fa fd fa fa fa fa fa fa fa fa fa
  0x6020018ba880: fa fa 00 00 fa fa fa fa fa fa fa fa fa fa 00 00
  0x6020018ba900: fa fa fd fa fa fa fd fd fa fa fd fa fa fa fd fd
=>0x6020018ba980: fa fa fd fd fa fa fd fd fa fa[fd]fa fa fa fd fa
  0x6020018baa00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x6020018baa80: fa fa fd fa fa fa fd fd fa fa fa fa fa fa fa fa
  0x6020018bab00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x6020018bab80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x6020018bac00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==22244==ABORTING

@maflcko please assign this issue to me let me fix the bug

@big14way I was planning on fixing this issue myself, but if you want to fix it, feel free. I think it'd be valuable if you were able to reproduce the use-after-free to make sure that the issue is fixed.