net: only monitor select exceptions while connecting #36716

pull Sjors wants to merge 6 commits into bitcoin:master from Sjors:2026/10/windows-select-exceptions changing 12 files +274 −18
  1. Sjors commented at 4:13 AM on October 31, 2026: member

    Alternative approach to #36688, citing the relevant parts of its description:

    There is currently nothing preventing us to receive TCP urgent data, which would never be returned by recv(). Linux's poll() is indifferent to this, but the select() socket monitoring may keep reporting it as pending on some platforms.

    This would not prevent progress nor lead to a crash anymore after #36665, but it's still undesirable.

    Fix this by only adding a socket to the select() exception set while waiting for a non-blocking connect() to complete. Windows needs that to detect a failed connection. Established connections no longer monitor the exception set on any platform. This approach was suggested by theuni in #36716 (comment) and replaces the earlier version of this PR, which ignored the exception set outside Windows and disconnected on it on Windows.

    We could instead have set SO_OOBINLINE on our sockets, so recv() returns any urgent data sent to us (and likely leads us to disconnect the peer). RFC 6093 doesn't forbid this, but the current approach seems more clear and direct to me.

    The PR makes one additional change for clarity: if errorSet is reported for a socket we did not request to read from, disconnect directly instead of first attempting a send that would fail (emits a different log message). if (recvSet || errorSet) is left as is. Since select() no longer reports errors for established connections, this only applies to poll().

    Test coverage is introduced in p2p_socket_errors.py.

    Finally the same early error check is applied to HTTP, with urgent data coverage in interface_http.py.

  2. DrahtBot added the label P2P on Oct 31, 2026
  3. DrahtBot commented at 4:13 AM on October 31, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK midnightmagic
    Approach ACK theuni
    Stale ACK darosior

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36916 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36916.svg"></sub> (test: avoid BSD ephemeral-port collisions in HTTP clients by willcl-ark)
    • #36434 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36434.svg"></sub> (http: support unix sockets by pinheadmz)
    • #36147 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36147.svg"></sub> (ci: add native freeBSD ci job by willcl-ark)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • test/functional/test_framework/netutil.py: the node sees the urgent indication, and reads that stop at the urgent mark → the node sees the urgent indication, and its reads stop at the urgent mark [the original phrasing makes “reads” sound like a noun rather than an action]

    <sup>2026-11-19 10:00:30</sup>

  4. pinheadmz commented at 11:38 AM on October 31, 2026: member

    Mine as well apply the same logic to HTTP, probably right here:

    https://github.com/bitcoin/bitcoin/blob/5669fda5ed85ac739c1678c03bd7c25193d099c6/src/httpserver.cpp#L916-L918

    I wrote functional test for the first PR, could be adapted to ensuring HTTP disconnects on urgent data:

    https://gist.github.com/pinheadmz/f5bb77865f870865e44c54d5751b569a

  5. Sjors commented at 7:42 AM on November 1, 2026: member

    My preference is to fix HTTP separately as a followup, but I'm open to doing it here.

  6. in src/util/sock.cpp:222 in 8754e5c1c4 outdated
     215 | @@ -216,7 +216,10 @@ bool Sock::WaitMany(std::chrono::milliseconds timeout, EventsPerSock& events_per
     216 |          const auto& s = sock->m_socket;
     217 |          if (events.requested & RecvEvent) {
     218 |              FD_SET(s, &recv);
     219 | +#ifdef WIN32
     220 | +            // Windows reports a failed non-blocking connect() in the exception set.
     221 |              FD_SET(s, &err);
     222 | +#endif
    


    vasild commented at 4:05 PM on November 2, 2026:

    Some lines below is this code:

             if (FD_ISSET(s, &err)) {
                 events.occurred |= ErrorEvent;
             }       
    

    which handles the outcome of select(2). Is it true that with this PR that would be no-op on non-Windows? Because we only FD_SET(s, &err) on Windows. If yes, then consider also putting it inside #ifdef WIN32 to make that more clear.


    Sjors commented at 11:44 AM on November 4, 2026:

    Added a commit for this.

  7. in src/net.cpp:2205 in 8754e5c1c4 outdated
    2200 | @@ -2201,6 +2201,18 @@ void CConnman::SocketHandlerConnected(const std::vector<CNode*>& nodes,
    2201 |              }
    2202 |          }
    2203 |  
    2204 | +#ifdef WIN32
    2205 | +        if (errorSet) {
    


    vasild commented at 4:09 PM on November 2, 2026:

    Should the below if (recvSet || errorSet) be revised as well? If we only set errorSet on Windows and we already handle it here, then the below can be just if (recvSet).

    Or, given that we only set errorSet on Windows, this #ifdef WIN32 here is not needed.


    Sjors commented at 1:53 PM on November 4, 2026:

    Should the below if (recvSet || errorSet) be revised as well?

    I added a commit to check errorSet separately and earlier, for poll().

    I spent some time trying to understand if that's a behavior change, but AFAIK it just emits a different log message. I added a test to demonstrate that.

  8. in src/netbase.cpp:574 in 8754e5c1c4
     569 | +        // transport treats TCP as a uniform byte stream; interpreting the
     570 | +        // cleartext URG flag as an interrupt would violate that design. P2P
     571 | +        // should never support such interrupts.
     572 | +        //
     573 | +        // RFC 6093 §6 ¶1 requires SO_OOBINLINE for applications that employ
     574 | +        // urgent indications. P2P does not, so leave SO_OOBINLINE unset.
    


    darosior commented at 11:55 PM on November 2, 2026:

    This does not follow, and regardless that seems like a confusing comment to add here.

    My reading of RFC 6093 is that it 1) discourages applications from sending urgent data 2) advises applications to set SO_OOBINLINE to fix the broken out of band mechanism regardless of their use for urgent data.

    But regardless, i think that comment just adds more confusion than it clears up, and it would be better to drop it.


    Sjors commented at 8:23 AM on November 4, 2026:
    1. discourages applications from sending urgent data

    The RFC says:

    Even though applications SHOULD NOT employ the urgent mechanism

    This isn't about a misbehaving sender, but rather about application design (both sides).

    This does not follow

    I'm not sure what exactly "this" refers to. It's true that the RFC does not forbid applications from setting SO_OOBINLINE if they don't "employ urgent indications".

    I do think it's good to document that we don't "employ urgent indications". The absence of the setting implies it, but an actual code comment is more clear.

    In any case, I'll drop the comment commit.

  9. darosior commented at 11:55 PM on November 2, 2026: member

    crACK 1fde57f1deda683129ccead9c30cfad0e235914e because it fixes the issue in P2P, but i prefer @theuni's suggestion to just always fail on error/exception during a wait (And i guess start listening for POLLPRI on Linux to consistently disconnect on URG data rather than ignoring it on one platform and disconnecting on another).

    I don't understand the point of the second commit.

  10. Sjors marked this as a draft on Nov 4, 2026
  11. test: cover P2P TCP urgent data handling
    Check that urgent indications do not prevent ordinary P2P input from
    being processed or count as received message bytes.
    
    Co-authored-by: Matthew Zipkin <pinheadmz@pm.me>
    2ec146efa0
  12. Sjors force-pushed on Nov 4, 2026
  13. Sjors commented at 3:57 PM on November 4, 2026: member

    I dropped the comment-only commit.

    Added two commits based on @vasild's feedback.

    Also added and expanded @pinheadmz's tests.

    Finally, I added the http fix at the end. It should be simple to review since it's the same approach.

    I also tested on Windows native.

  14. Sjors marked this as ready for review on Nov 4, 2026
  15. midnightmagic commented at 12:48 AM on November 6, 2026: contributor

    We could instead have set SO_OOBINLINE on our sockets, so recv() returns any urgent data sent to us (and likely leads us to disconnect the peer). RFC 6093 doesn't forbid this, but the current approach seems more clear and direct to me.

    I believe it's possible to construct an oob urgent ptr for data which will never be sent and which will then never be received but will still trigger the exception set. Setting SO_OOBINLINE thus can't ultimately clear the select() exception because no data is actually inline to read. On the BSDs I examined (and therefore likely OS/X) this will still generate an exception so the current select() call will return immediately again. As noted elsewhere, this doesn't technically prevent eventual node progress.

    fwiw, I strongly suggest that oob data be considered an error condition based on this. There is nowhere for UNIX-y OSes which an oob byte will ever be generated by a valid p2p peer, and thus it'll only ever be generated in this context — therefore disconnection is a logical and completely fair result. I personally think at this point it's worth a ban.

    (Uh, just to be clear, at the very least I think we should delete the exceptfds from the select() call entirely, so strong ACK for this approach.)

  16. Sjors commented at 8:43 AM on November 6, 2026: member

    I strongly suggest that oob data be considered an error condition based on this

    I agree that if there was a DISCONNECT_ON_OOB flag, we should set it. But we can't directly detect if such data was present. All we can know is that errorSet was triggered, which could be for unrelated OS reasons. That could cause us to disconnect a well behaved peer.

    That's why this PR simply ignores the exception set (outside Windows), instead of trying to handle it. It might be possible with poll(), but that would be better for a followup.

  17. theuni commented at 9:27 PM on November 6, 2026: member

    @midnightmagic agreed.

    Imo we should just always disconnect when an exception is set, or when oob/prio are found. I don't see any need for platform-specific behavior here. I'd rather go with: https://github.com/theuni/bitcoin/commits/net-oob-error/

    That branch unifies the select/poll behaviors, and causes a disconnect (on all platforms) if any of those conditions are true.

    It differs from what's here in that it turns POLLHUP into a read event as it should be, rather than an error. That simplifies things significantly, as it allows draining to happen naturally, and it means that the exception set is truly only comprised of exceptions.

    If receiving is off (meaning that we have data to send) and we receive a SIGHUP, we'll disconnect on the next send.

    It fixes up the tests to show that OOB data always causes a disconnect.

  18. DrahtBot added the label Needs rebase on Nov 7, 2026
  19. Sjors force-pushed on Nov 9, 2026
  20. Sjors commented at 12:07 PM on November 9, 2026: member

    Rebased after #36744. @theuni wrote:

    Imo we should just always disconnect when an exception is set

    So you're not worried about the operating system triggering this set for reasons that have nothing to do with OOB data?

    That branch unifies the select/poll behaviors, and causes a disconnect (on all platforms) if any of those conditions are true.

    Since that's a completely different approach from this PR, it seems better to open it as a separate (and alternative) PR.

  21. DrahtBot removed the label Needs rebase on Nov 9, 2026
  22. theuni commented at 6:15 PM on November 16, 2026: member

    Ok, I retract the above. After experimenting for the last few days, nothing really works here to align the two behaviors without logical hacks.

    I really dislike the platform-specific hacks here, but I agree that they're a necessary evil as long as we're supporting select and poll. I'll drop the suggestion to use my branch and review what's here for correctness.

  23. theuni commented at 10:28 PM on November 16, 2026: member

    Sorry to throw yet another wrench in this, but when trying to tease apart the uses and platform differences for the select error set, it occurred to me that all of the hacks have to do with the fact that we have a single obscure use-case for that set, and in all other cases it's harmful. Rather than trying to isolate the platforms that require that use-case, why not isolate the use-case instead?

    Please see here: https://github.com/theuni/bitcoin/commit/f8fbc168bbe0995d498a1ee594515869a78d4361 for a change that splits out the connection failure case. That allows us to completely ignore the select error set in all other cases where it'd be considered harmful. Imo that's much cleaner because it requires a single opt-in where needed, rather than opt-outs sprinkled all around.

    With that change, I don't think the early if (errorSet && !recvSet) disconnects are needed at all, but could (and probably should) be added for belt-and-suspenders. @Sjors @darosior mind taking a look?

  24. net: only use the select error set while connecting
    Windows uses the select() error set to report a failed non-blocking
    connect(). Add ConnErrorEventFlag, which ConnectToSocket() passes to
    Sock::Wait() to opt in to the error set while connecting. This is safe
    on all platforms, as any error condition is worth waking for there.
    
    All other uses of select() now leave the error set empty. With
    select(), sockets waited on without the flag no longer wake up for or
    report exceptional conditions, such as TCP urgent data on an
    established connection. No behavior change with poll().
    
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    6919268d71
  25. test: move send-stall wait into the framework 9a61d7f17a
  26. test: cover P2P reset while sending
    Reset a slow reader with block output stalled and more input queued
    behind receive backpressure.
    7fa2daf07f
  27. Sjors renamed this:
    net: stop monitoring select exceptions outside Windows
    net: only monitor select exceptions while connecting
    on Nov 18, 2026
  28. Sjors force-pushed on Nov 18, 2026
  29. Sjors commented at 5:36 PM on November 18, 2026: member

    @theuni to put it into my own words, IIUC:

    There's two phases to a connection, phase 1 establishes it ConnectToSocket and phase 2 maintains it after (in GenerateWaitSockets, etc). Your proposal makes it so that in phase 2 nothing watches errorSet, while in phase 1 it's now watched on all platforms. A follow-up could narrow phase 1 to just Windows as belt-and-suspenders.

    This makes sense, and I adopted it.

    On ancient versions of Windows it was unsafe to ignore errorSet, and it had to be drained to prevent memory exhaustion. See https://mailarchive.ietf.org/arch/msg/tcpm/55ecR94wP1NkrL5cuPj0PWtTIXc/

    The following checks for this: https://gist.github.com/Sjors/9582259267a038e06c7f0730356416de

    I tested on a physical Windows 11 Home 26H2 machine. Connections reset after 17 urgent bytes. I also tested Windows Server 2022 on a Github CI runner: https://github.com/Sjors/bitcoin/actions/runs/44390882699/job/135107875034

    The latter should roughly match Windows 10 in this regard, which is the oldest version we support. So this is no longer a concern in theory, but it doesn't hurt to actually try if you can find such a machine.

    BSD is probably also worth testing, cc @vasild.

  30. DrahtBot added the label CI failed on Nov 18, 2026
  31. in src/net.cpp:2211 in 67bdc2d3b5 outdated
    2206 | +        // whichever events we requested.
    2207 | +        // If we requested POLLIN, then recv() below can read buffered data
    2208 | +        // before it detects the disconnect, unless a failed send disconnects
    2209 | +        // first. If receiving is paused, we did not request POLLIN, so
    2210 | +        // disconnect now: sending on a broken connection would fail anyway.
    2211 | +        if (errorSet && !recvSet) {
    


    theuni commented at 10:36 PM on November 18, 2026:

    I experimented with removing select() entirely on top of this changeset, and ended up needing a little change here. Rather than recvSet, what we really want to test (as your comment states) is recvReq, where

    bool recvReq = it->second.requested & Sock::RecvEvent;
    

    I don't think it matters here, but in order to support poll in a cross-platform way, the exact condition we want to test is "did we see an error when we didn't request a receive event", which is slightly different than "did we see an error when we didn't SEE a receive event" as we have here.

    This matters for platforms (Windows) which raise only POLLERR | POLLHUP for disconnections, even when POLLIN was requested.

    Again.. it shouldn't matter here. Just pointing it out for the future.


    Sjors commented at 10:00 AM on November 19, 2026:

    The concept of recvRequested makes a lot of sense, so I added this and used it for the early disconnect.

    67bdc2d3b5c62dd4d8cf0eb757b78fcf0d5b56b3 -> 8b8a2f49267daa6a3f9d85d9cb511bd7c286e52a

    Also applied to http in 05084e7879140aaf7c9b77e372ebc828f98e3153.

  32. in src/net.cpp:2221 in 67bdc2d3b5 outdated
    2230 | @@ -2218,7 +2231,7 @@ void CConnman::SocketHandlerConnected(const std::vector<CNode*>& nodes,
    2231 |              }
    2232 |          }
    2233 |  
    2234 | -        if (recvSet || errorSet)
    


    theuni commented at 10:57 PM on November 18, 2026:

    According to the poll Linux manpage, we may receive a lone POLLHUP (without POLLIN or POLLERR) when the socket has been drained and the next recv would close it. I think if we skip the || errorSet here, we could end up waking for the error (POLLHUP) event, and never doing anything with it. Then we're back to the tight loop.

    Since select is a no-op here now, I don't see the harm in leaving this as if (recvSet || errorSet) ?

    Here's the output of the Linux manpage example. It's a file descriptor rather than a socket, it's possible that the behavior is different.

     $ ./poll myfifo
    Opened "myfifo" on fd 3
    About to poll()
    Ready: 1
      fd=3; events: POLLIN POLLHUP
        read 4 bytes: foo
    
    About to poll()
    Ready: 1
      fd=3; events: POLLHUP
        closing fd 3
    All file descriptors closed; bye
    

    Sjors commented at 10:00 AM on November 19, 2026:

    I think if we skip the || errorSet here

    The poll code in WaitMany sets errorSet if it gets POLLHUP. The if (errorSet && !recvSet) check above disconnects, so we don't get here.

    I still dropped this change: 67bdc2d3b5c62dd4d8cf0eb757b78fcf0d5b56b3 -> 8b8a2f49267daa6a3f9d85d9cb511bd7c286e52a

    Given that SocketHandlerConnected() only handles established connections:

    • with select(), errorSet is never set, so recvSet || errorSet is just recvSet
    • with poll(), a lone POLLHUP or POLLERR gives errorSet && !recvSet, which the early disconnect above already handles
  33. theuni commented at 10:59 PM on November 18, 2026: member

    Approach ACK, thanks for the update.

    See my comment about keeping the (recvSet || errorSet) check as-is.

    Other than that, looks good to me.

  34. DrahtBot removed the label CI failed on Nov 19, 2026
  35. net: disconnect on poll errors before sending
    Disconnect directly on POLLERR/POLLHUP when no read was requested,
    skipping the send attempt that would otherwise fail and disconnect
    the peer. This also changes the debug log message.
    
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    8b8a2f4926
  36. http: handle socket errors separately from reads
    When poll reports an error or hangup and no read was requested,
    disconnect before sending instead of reading. Cover TCP urgent data
    in the functional test.
    
    Co-authored-by: Matthew Zipkin <pinheadmz@pm.me>
    05084e7879
  37. Sjors force-pushed on Nov 19, 2026
  38. DrahtBot added the label CI failed on Nov 19, 2026
  39. in src/util/sock.h:185 in 6919268d71
     182 | +     *
     183 | +     * Do not use on established connections: the `select()` exception set
     184 | +     * also reports TCP urgent data, which `recv()` does not return, so
     185 | +     * `Wait()` could keep returning with nothing to read.
     186 | +     */
     187 | +    static constexpr EventFlag ConnErrorEventFlag = 0b10000;
    


    darosior commented at 6:43 PM on November 19, 2026:

    Why the jump from (1 << 2) to (1 << 4)?


    theuni commented at 10:18 PM on November 19, 2026:

    Just keeping the low half reserved for Events and the high for Flags. I can't imagine it matters in any real way... 1 << 3 is fine if you prefer it.

    After all, this is just a hack to keep the diff small as opposed to adding the more straightforward WaitForNewConnection() function.

  40. in test/functional/p2p_socket_errors.py:51 in 05084e7879
      46 | +        node = self.nodes[0]
      47 | +        with socket.create_connection(("127.0.0.1", p2p_port(0)), timeout=10) as peer:
      48 | +            self.wait_until(lambda: len(node.getpeerinfo()) == 1)
      49 | +            send_probe(peer)
      50 | +            # Processing the ordinary version message demonstrates progress
      51 | +            # despite the urgent indication. The urgent byte is excluded.
    


    darosior commented at 10:10 PM on November 19, 2026:

    Yes, it demonstrates progress. But the bug you are trying to fix in this PR is not a lack of progress on urgent data.

  41. darosior commented at 10:18 PM on November 19, 2026: member

    This PR has changed multiple times, and considerably grown, since it was opened. And it's also losing momentum, whereas it's currently the main PR for an important fix. So i think it's worth taking a step back and asking what it's trying to achieve.

    I think the initial goal was to get a minimal fix in, because a proper cleanup of this logic like #36761 would be riskier to backport. But now this PR adds as much code as switching to poll entirely (and removes 10 times less).

    It's not exactly trivial code either: we introduce new flags to Sock, leak the connection stage down to the socket wait logic to only register the exception set at this point, have some callers (but not all!) now disconnect on errors.. That's a bunch of behaviour changes not strictly necessary to fix the bug, that i would be hesitant to backport (i.e. ship with much less testing --especially if we deem the bug important enough to cut minor releases as soon as the fix is ready).

    On top of that, this PR adds a whole bunch of functional tests that seem pretty brittle to me, since they rely on kernel settings that may differ between machines, or break when we update CI (that's the reason why i did not push similar tests i had performed locally to #36688). It's also not clear to me how useful they are, as they wouldn't even catch the bug this PR set out to fix in the first place (following my review comment below i tested this by cherry-picking your tests on master and running them on a Windows VM).

    Why not go back to the simple approach of simply always disconnecting when a wait returns an ErrorEvent? It's short, easy to reason about and safer to backport.

    Sure, it preserves the behaviour difference between platforms using select() and those using poll(). That can be cleaned up later, in a more involved change we don't intend to backport, since this is a pre-existing issue.

    I'm thinking about something like this https://github.com/darosior/bitcoin/commit/97666aa7a6230641af7e00e372ec1dbd88436c92.

  42. midnightmagic commented at 12:23 AM on November 20, 2026: contributor

    I strongly suggest that oob data be considered an error condition based on this

    I agree that if there was a DISCONNECT_ON_OOB flag, we should set it. But we can't directly detect if such data was present. All we can know is that errorSet was triggered, which could be for unrelated OS reasons. That could cause us to disconnect a well behaved peer.

    That's why this PR simply ignores the exception set (outside Windows), instead of trying to handle it. It might be possible with poll(), but that would be better for a followup.

    Minor context: On many OSes (mostly BSD-derived) I believe there is only a single trigger for the exceptfd set in a select() call for a normal TCP socket and that's OOB data. The reason to consider OOB data now almost certainly hostile incoming data in all cases is because the codebase's current (?) behaviour makes it so the only reasonable interpretation is a hostile action — i.e. there is no well-behaving peer which will ever generate OOB data and send it to us.

    If the codebase ignores it in a way where someone sending it means nothing — yes absolutely that will solve the problem in the least and most minimally-invasive fashion, and people like myself will just run our own detection patch independently. I agree with you in that if we're immune to it in the first place, detecting it above that is just special-case pointlessness. My only concern really is making sure the .. uh error condition isn't possible anymore. So, yeah you're right.

    Elsewise if we retain the ability to detect OOB data, a noisy log entry and a disconnect would be welcome to folks like myself but certainly not necessary. I really don't have a particular dog in this race either way, and I'll just be grateful when the error condition gets fixed.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-21 11:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me