See individual commits for more description. In particular, 75ead75890b9b5ca5565a31d5fe9ad31dc9a96fa.
This (along with #8707) fixes the current maxupload test failure.
Concept ACK
Needs rebase
Concept ACK
5037 | @@ -5038,7 +5038,7 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv, 5038 | 5039 | // Be shy and don't send version until we hear 5040 | if (pfrom->fInbound) 5041 | - pfrom->PushVersion(); 5042 | + connman.PushVersion(pfrom, GetAdjustedTime());
Any idea why we use a different time for inbound vs outbound peers? If anything I'd think we'd announce our real time to inbound peers and our adjusted time to outbound ones, but we do the opposite.
@TheBlueMatt I'm not sure I see the reasoning for either. I'd expect us to use our best guess of time everywhere.
388 | @@ -389,6 +389,9 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char *pszDest, bool fCo 389 | 390 | // Add node 391 | CNode* pnode = new CNode(GetNewNodeId(), nLocalServices, GetBestHeight(), hSocket, addrConnect, CalculateKeyedNetGroup(addrConnect), pszDest ? pszDest : "", false); 392 | + 393 | + PushVersion(pnode, GetTime());
Can we go ahead and move the PushVersion logic into InitializeNode? I think thats the right place since it will move into net_processing.cpp with the rest of the "node-message-processing" stuff.
Got halfway through a review and realized I just dont know how I feel about this...CNode moving towards a "I speak to the CConnman and serialize messages for the wire" seems like a sane design (and seems to be what you're going for here), but it means there is just no good place to put serialization-version logic that isnt a complete layer violation.
On Oct 6, 2016 1:06 PM, "Matt Corallo" notifications@github.com wrote:
@TheBlueMatt commented on this pull request.
Got halfway through a review and realized I just dont know how I feel about this...CNode moving towards a "I speak to the CConnman and serialize messages for the wire" seems like a sane design (and seems to be what you're going for here), but it means there is just no good place to put serialization-version logic that isnt a complete layer violation.
Yea, I agree. I started creating a CSendMessage that would hold the serialized result, which could go directly to CConnman, but decided for the more straightforward change here. I'm happy to make that change, either as a part of this pr or after.
That has the benefit of testing the serialized result for p2p without actually requiring the send.
In src/main.cpp:
@@ -5038,7 +5038,7 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv, // Be shy and don't send version until we hear if (pfrom->fInbound) - pfrom->PushVersion(); + connman.PushVersion(pfrom, GetAdjustedTime());
Any idea why we use a different time for inbound vs outbound peers? If anything I'd think we'd announce our real time to inbound peers and our adjusted time to outbound ones, but we do the opposite.
Unsure of the reason there.
In src/net.cpp:
@@ -389,6 +389,9 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char pszDest, bool fCo // Add node CNode pnode = new CNode(GetNewNodeId(), nLocalServices, GetBestHeight(), hSocket, addrConnect, CalculateKeyedNetGroup(addrConnect), pszDest ? pszDest : "", false); + + PushVersion(pnode, GetTime());
Can we go ahead and move the PushVersion logic into InitializeNode? I think thats the right place since it will move into net_processing.cpp with the rest of the "node-message-processing" stuff.
I had planned on doing exactly that as a next step, but it looked kinda out of place by itself. With your changes, it makes perfect sense. Happy to change it to whatever makes it easier for you to move.
Glad to see we're in sync :)
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Yea, seems like we're mostly on the same page here... Probably reasonable to give the serialization to net_processing, though I am concerned with how to merge type with P2P encryption, given that it also changes the message hashing, but only after messages are exchanged (I believe? Maybe @jonasschnelli wants to comment). Even more unsure of how to handle that.
As for this PR, I'm fine with it as it is now, noting that we want to tweak things further as we split code, though I'd prefer we go ahead and move the version message handling to Initialize so we don't end up moving the same block three times.
On October 6, 2016 3:25:54 PM EDT, Cory Fields notifications@github.com wrote:
On Oct 6, 2016 1:06 PM, "Matt Corallo" notifications@github.com wrote:
@TheBlueMatt commented on this pull request.
Got halfway through a review and realized I just dont know how I feel about this...CNode moving towards a "I speak to the CConnman and serialize messages for the wire" seems like a sane design (and seems to be what you're going for here), but it means there is just no good place to put serialization-version logic that isnt a complete layer violation.
Yea, I agree. I started creating a CSendMessage that would hold the serialized result, which could go directly to CConnman, but decided for the more straightforward change here. I'm happy to make that change, either as a part of this pr or after.
That has the benefit of testing the serialized result for p2p without actually requiring the send.
In src/main.cpp:
@@ -5038,7 +5038,7 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv, // Be shy and don't send version until we hear if (pfrom->fInbound) - pfrom->PushVersion(); + connman.PushVersion(pfrom, GetAdjustedTime());
Any idea why we use a different time for inbound vs outbound peers? If anything I'd think we'd announce our real time to inbound peers and our adjusted time to outbound ones, but we do the opposite.
Unsure of the reason there.
In src/net.cpp:
@@ -389,6 +389,9 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char pszDest, bool fCo // Add node CNode pnode = new CNode(GetNewNodeId(), nLocalServices, GetBestHeight(), hSocket, addrConnect, CalculateKeyedNetGroup(addrConnect), pszDest ? pszDest : "", false); + + PushVersion(pnode, GetTime());
Can we go ahead and move the PushVersion logic into InitializeNode? I think thats the right place since it will move into net_processing.cpp with the rest of the "node-message-processing" stuff.
I had planned on doing exactly that as a next step, but it looked kinda out of place by itself. With your changes, it makes perfect sense. Happy to change it to whatever makes it easier for you to move.
Glad to see we're in sync :)
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #8708 (comment)
Needs rebase
rebased. @TheBlueMatt I added some commits to handle the version push in InitializeNode. I left the serialization happening in CConnman for now. Agreed that it's a layer violation and have plans for separating it, but I'd prefer not to change the scope here too much unless you hugely object.
2567 | {
2568 | - ENTER_CRITICAL_SECTION(cs_vSend);
2569 | - assert(ssSend.size() == 0);
2570 | - ssSend << CMessageHeader(Params().MessageStart(), pszCommand, 0);
2571 | - LogPrint("net", "sending: %s ", SanitizeString(pszCommand));
2572 | + return {SER_NETWORK, nVersion ? nVersion | flags : pnode->GetSendVersion() | flags, CMessageHeader(Params().MessageStart(), sCommand.c_str(), 0) };
GetSendVersion needs a lock (or make it atomic with release/aquire).
2581 | + // Set the size 2582 | + unsigned int nSize = strm.size() - CMessageHeader::HEADER_SIZE; 2583 | + WriteLE32((uint8_t*)&strm[CMessageHeader::MESSAGE_SIZE_OFFSET], nSize); 2584 | + // Set the checksum 2585 | + uint256 hash = Hash(strm.begin() + CMessageHeader::HEADER_SIZE, strm.end()); 2586 | + assert(strm.size () >= CMessageHeader::CHECKSUM_OFFSET + CMessageHeader::CHECKSUM_SIZE);
Can you move this up to cover the MESSAGE_SIZE_OFFSET write two lines up?
ok
2588 | 2589 | - LogPrint("net", "(aborted)\n"); 2590 | } 2591 | 2592 | -void CNode::EndMessage(const char* pszCommand) UNLOCK_FUNCTION(cs_vSend) 2593 | +void CConnman::PushMessage(CNode* pnode, CDataStream& strm, const std::string& sCommand)
strm can be const, I believe? Would be nice to go ahead and do that since I'd like to use this to create a cmpctblock fully-encoded message once and then just call this on each node that wants it.
ok
Er, this isn't intended to be used that way. This is internal and should be private.
The next set of changes will introduce a CSendMessage which is fully serialized. PushMessage will take that instead. That fixes the current layer violation here, and your issue at the same time I think?
OK, sounds good.
413 | @@ -411,6 +414,24 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char *pszDest, bool fCo 414 | return NULL; 415 | } 416 | 417 | +void CConnman::PushVersion(CNode* pnode, int64_t nTime)
nit: I'd kinda prefer we not move PushVersion twice.
369 | + CAddress addr = pnode->addr; 370 | + 371 | + CAddress addrYou = (addr.IsRoutable() && !IsProxy(addr) ? addr : CAddress(CService(), addr.nServices)); 372 | + CAddress addrMe = CAddress(CService(), nLocalNodeServices); 373 | + 374 | + connman.PushMessageWithVersion(pnode, INIT_PROTO_VERSION, NetMsgType::VERSION, PROTOCOL_VERSION, (uint64_t)nLocalNodeServices, GetTime(), addrYou, addrMe,
This is a slight change in behavior - we are now pushing our non-adjusted time to all peers, instead of just outbound peers. Not actually sure if it matters, but taking note here.
Whoops, good catch. That was not intentional. Will fix.
361 | - state.name = pnode->addrName; 362 | - state.address = pnode->addr; 363 | +void PushNodeVersion(CNode *pnode, CConnman& connman) 364 | +{ 365 | + ServiceFlags nLocalNodeServices = pnode->GetLocalServices(); 366 | + uint64_t nonce = pnode->GetLocalNonce();
I think half the functions here need locks or to become atomics to be correct. (unless I'm missing a required lock to call this function, in which case please AssertLockHeld to make it obvious).
May be easiest to just make half the ints in CNode atomic.
This value doesn't change. I have a branch with additional commits that turns a bunch of stuff into const to make it clear that it's threadsafe. I could add those here if you'd like?
I'm ok with this happening in a separate PR, as AFAICT there are already dragons here, but should definitely happen before 0.14.
A few nits, and some things that look like missing locks to me - I think we should do a pass over CNode and just convert a bunch of the various ints to std::atomic to fix most of the issues (ie they aren't issues on at least x86 right now).
758 | + // only one version message is allowed per session. We can therefore 759 | + // treat this value as const and even atomic as long as it's only used 760 | + // once the handshake is complete. Any attempt to set this twice is an 761 | + // error. 762 | + assert(nSendVersion == 0); 763 | + *const_cast<int*>(&nSendVersion) = nVersionIn;
This violates C++ spec - nSendVersion must exist non-const somewhere for const_cast to be permitted (I believe).
Ok, I'll just change this back to non-const. The const was only to prove that it's not racy, since it can only be set once.
Updated for @TheBlueMatt's nits. Since the changes were very small, I went ahead and squashed.
PRs are coming up to address the const issues, as well as fixing the serialization layer violation.
2579 | - 2580 | - LEAVE_CRITICAL_SECTION(cs_vSend); 2581 | + // Set the size 2582 | + unsigned int nSize = strm.size() - CMessageHeader::HEADER_SIZE; 2583 | + WriteLE32((uint8_t*)&strm[CMessageHeader::MESSAGE_SIZE_OFFSET], nSize); 2584 | + assert(strm.size () >= CMessageHeader::CHECKSUM_OFFSET + CMessageHeader::CHECKSUM_SIZE);
This assert needs to move up a line.
2620 | - // Set the checksum 2621 | - uint256 hash = Hash(ssSend.begin() + CMessageHeader::HEADER_SIZE, ssSend.end()); 2622 | - assert(ssSend.size () >= CMessageHeader::CHECKSUM_OFFSET + CMessageHeader::CHECKSUM_SIZE); 2623 | - memcpy((char*)&ssSend[CMessageHeader::CHECKSUM_OFFSET], hash.begin(), CMessageHeader::CHECKSUM_SIZE); 2624 | + unsigned int nSize = strm.size() - CMessageHeader::HEADER_SIZE; 2625 | + LogPrint("net", "%s (%d bytes) peer=%d\n", sCommand.c_str(), nSize, pnode->id);
nit: any reason you changed the text here? It used to read "sending: %s (%d bytes) peer=%d\n", SanitizeString(pszCommand, nSize, id) (split across two lines).
No, they just ended up combined since the actual data push is all done in one place now.
Is there a reason to prefer having them split? How about adding back the "sending" for easy grepping, and the sanitizestring for extra paranoia, but leaving it as a combined message?
1136 | @@ -1156,10 +1137,6 @@ void CConnman::ThreadSocketHandler() 1137 | { 1138 | TRY_LOCK(pnode->cs_vSend, lockSend); 1139 | if (lockSend) { 1140 | - if (pnode->nOptimisticBytesWritten) {
I cant see anything wrong with this, but it seems obviously equivalent...what was the reason for nOptimisticBytesSent previously?
It was a hack. In fact, this PR was originally created to fix this.
If the node is in charge of pushing data, Connman has no way of knowing if optimistic writes have succeeded. The hackish solution was to cache the periodically check for optimistic written bytes (stored by CNode) and add them to the node's total, but that fails miserably if the optimistic writes succeed most of the time.
Now that CConnman is in charge of the writes, it knows immediately how much was written, and doesn't have to try to catch up with it later.
Aside from the few comments, utACK 2df814b2ecfcfe67f5a7419b662df1ec1efd9d2d.
Oh totally, didn't mean to imply the sending message should be two prints, just that you changed the text spuriously.
On November 1, 2016 5:08:13 PM EDT, Cory Fields notifications@github.com wrote:
theuni commented on this pull request.
- // Set the checksum
- uint256 hash = Hash(ssSend.begin() + CMessageHeader::HEADER_SIZE, ssSend.end());
- assert(ssSend.size () >= CMessageHeader::CHECKSUM_OFFSET + CMessageHeader::CHECKSUM_SIZE);
- memcpy((char*)&ssSend[CMessageHeader::CHECKSUM_OFFSET], hash.begin(), CMessageHeader::CHECKSUM_SIZE);
- unsigned int nSize = strm.size() - CMessageHeader::HEADER_SIZE;
- LogPrint("net", "%s (%d bytes) peer=%d\n", sCommand.c_str(), nSize, pnode->id);
No, they just ended up combined since the actual data push is all done in one place now.
Is there a reason to prefer having them split? How about adding back the "sending" for easy grepping, and the sanitizestring for extra paranoia, but leaving it as a combined message?
You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #8708
839 | - void EndMessage(const char* pszCommand) UNLOCK_FUNCTION(cs_vSend); 840 | - 841 | - void PushVersion(); 842 | - 843 | - 844 | - void PushMessage(const char* pszCommand)
Good to get rid of this repetition.
2593 | +void CConnman::PushMessage(CNode* pnode, CDataStream& strm, const std::string& sCommand) 2594 | { 2595 | - // The -*messagestest options are intentionally not documented in the help message, 2596 | - // since they are only used during development to debug the networking code and are 2597 | - // not intended for end-users. 2598 | - if (mapArgs.count("-dropmessagestest") && GetRand(GetArg("-dropmessagestest", 2)) == 0)
135 | @@ -136,6 +136,33 @@ class CConnman 136 | 137 | bool ForNode(NodeId id, std::function<bool(CNode* pnode)> func); 138 | 139 | + template <typename... Args>
These are not used?
PushMessageWithVersionAndFlag could be private I suppose. It's only used by the others.
PushMessageWithFlag is the same as before.
PushMessageWithVersion is used explicitly before the handshake is complete, before we know what upgraded version to use.
Note that as next steps, an intermediary and fully serialized "CSendMessage" (or so) class will be added, which CConnman accepts rather than bare arguments, and forwards to the network. This makes CConnman mostly oblivious as to what it's pushing (other than the type of message), which further improves encapsulation.
Also, The sendversion logic will move to CNodeState once the locking situation there is improved.
I believe the removal of CNode::Fuzz maybe remove the need for CDataStream::insert or CDataStream::erase.
2686 | @@ -2686,6 +2687,52 @@ void CNode::EndMessage(const char* pszCommand) UNLOCK_FUNCTION(cs_vSend) 2687 | LEAVE_CRITICAL_SECTION(cs_vSend); 2688 | } 2689 | 2690 | +CDataStream CConnman::BeginMessage(CNode* pnode, int nVersion, int flags, const std::string& sCommand) 2691 | +{ 2692 | + return {SER_NETWORK, nVersion ? nVersion | flags : pnode->GetSendVersion() | flags, CMessageHeader(Params().MessageStart(), sCommand.c_str(), 0) };
That's pretty hard to read. Can you factor the version logic out, or at least combine as (nVersion ? nVersion : pnode->GetSendVersion()) | flags?
Will do.
2717 | + LOCK(pnode->cs_vSend); 2718 | + if(pnode->hSocket == INVALID_SOCKET) { 2719 | + return; 2720 | + } 2721 | + bool optimisticSend(pnode->vSendMsg.empty()); 2722 | + pnode->vSendMsg.emplace_back(strm.begin(), strm.end());
Not for this PR, but we should try to make a CSerializeData move-constructable from a CDataStream, and make PushMessage take a CDataStream&& perhaps. Or is this going away anyway?
Yea, going away. Instead we'll have a new (very simple) stream type that serializes args for the network, which CConnman will use directly. As it'll construct in-place and be movable, it should eliminate at least 2 copies, I think.
354 | @@ -355,10 +355,10 @@ void UpdatePreferredDownload(CNode* node, CNodeState* state) 355 | } 356 | 357 | void InitializeNode(NodeId nodeid, const CNode *pnode) { 358 | + CAddress addr = pnode->addr;
Why copy pnode->addr twice?
utACK
Also add a variadic CDataStream ctor for ease-of-use.
The changes here are dense and subtle, but hopefully all is more explicit
than before.
- CConnman is now in charge of sending data rather than the nodes themselves.
This is necessary because many decisions need to be made with all nodes in
mind, and a model that requires the nodes calling up to their manager quickly
turns to spaghetti.
- The per-node-serializer (ssSend) has been replaced with a (quasi-)const
send-version. Since the send version for serialization can only change once
per connection, we now explicitly tag messages with INIT_PROTO_VERSION if
they are sent before the handshake. With this done, there's no need to lock
for access to nSendVersion.
Also, a new stream is used for each message, so there's no need to lock
during the serialization process.
- This takes care of accounting for optimistic sends, so the
nOptimisticBytesWritten hack can be removed.
- -dropmessagestest and -fuzzmessagestest have not been preserved, as I suspect
they haven't been used in years.
Drop all of the old stuff.
This is now handled properly in realtime.
Rebased after #9050. The net.cpp diff before/after the rebase:
git diff 19d3daef936905743ef108aee1ccfe46fd77b283..c7b8effb96726ba5367a7b1c9c7374c4d2cc10e2 -- net.cpp
can be seen here: https://gist.github.com/theuni/cc4eb87f0196ddadb637d1a41c5c5b2e.
Looks like it needs rebased again, sorry :/.
The rebase is trivial: https://github.com/sipa/bitcoin/commits/pr_8708
Was merged via c8c572f8f1ea0a98ee3e7b3343c766ad52848bef (from sipa's rebased branch https://github.com/sipa/bitcoin/commits/pr_8708)
Thanks!
17 | @@ -18,6 +18,7 @@ 18 | #include "init.h" 19 | #include "merkleblock.h" 20 | #include "net.h" 21 | +#include "netbase.h"
what does it use in netbase.h?
@rebroad An easy way to find out is to remove the line and compile. The compiler will tell you what is used but missing.