blocktxns are not requested when a cmpctblock received if in flight from another peer #9242

1. 
   rebroad commented at 3:12 am on November 30, 2016: contributor

   There are cases when it could certainly be faster to acquire the block by requesting blocktxns following a received cmpctblock even though a full-block is in flight, and some where it would not be.

   This could be determined by seeing how many requests are needed in the blocktxns and seeing how much of the full-block has already been downloaded. In cases where the full-block has only just started to be received (it will know how big the incoming full-block is once it has started to arrive, and also can calculate how long it will take to finish downloading), it could disconnect that connection (to abort the download) and request the blocktxns, for example.

2. rebroad renamed this:
   blocktxns are not requested when cmpctblock received if a full block is in flight
   blocktxns are often not requested when a cmpctblock received if in flight from another peer
   on Dec 4, 2016
3. 
   rebroad commented at 2:57 am on December 4, 2016: contributor

   E.g.:-

    02016-12-02 19:48:57.973416 recv new best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=73
    12016-12-02 19:48:57.974570 send sendcmpct (no announce) ver=1 peer=23
    22016-12-02 19:48:57.975228 send sendcmpct (announce) ver=1 peer=73
    32016-12-02 19:48:57.975901 send getdata cmpctblock 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=73
    42016-12-02 19:48:58.930527 Incoming cmpctblock (14576 of 17132 bytes) chk=8a684f74 from peer=3
    52016-12-02 19:48:58.932063 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=23
    62016-12-02 19:48:58.957902 Incoming cmpctblock (14576 of 17132 bytes) chk=4fda2093 from peer=622
    72016-12-02 19:48:59.032385 recv cmpctblock 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) size=17132 peer=622
    82016-12-02 19:48:59.069218 recv cmpctblock 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) size=17132 peer=3
    92016-12-02 19:48:59.393061 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=33
   102016-12-02 19:48:59.980811 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=1628
   112016-12-02 19:49:00.063698 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=58
   122016-12-02 19:49:00.125696 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=2141
   132016-12-02 19:49:00.183047 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=268
   142016-12-02 19:49:00.201492 recv best header 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) peer=1114
   152016-12-02 19:49:00.256338 Incoming cmpctblock (14576 of 17132 bytes) chk=084e622b from peer=73
   162016-12-02 19:49:00.348837 recv cmpctblock 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) size=17132 peer=73
   172016-12-02 19:49:00.453634 send getblocktxn 000000000000000002c98773446aac1c780435c172eda7bcc8044060654a9ca8 (441603) indexes=2812 size=60 peer=73
   

   i.e. it is not until the cmpctblock that was requested arrives that the blocktxns are finally requested.

   Block propagation time is therefore delayed unnecessarily as the blocktxns could be requested from the peer that first provides the cmpctblock.

4. rebroad renamed this:
   blocktxns are often not requested when a cmpctblock received if in flight from another peer
   blocktxns are not requested when a cmpctblock received if in flight from another peer
   on Dec 4, 2016
5. rebroad referenced this in commit 438f580fc9 on Dec 11, 2016
6. rebroad referenced this in commit a6370a6262 on Dec 11, 2016
7. rebroad referenced this in commit ee65677de3 on Dec 11, 2016
8. rebroad referenced this in commit 888cfa6e42 on Dec 11, 2016
9. rebroad referenced this in commit 536f70f236 on Dec 12, 2016
10. 
    rebroad commented at 6:18 am on December 12, 2016: contributor

    I have something working, which seems (to me at least) better behavior than the current behaviour:-

     02016-12-12 06:08:08.294900 recv new best header 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) age=40s peer=44
     12016-12-12 06:08:08.294952 send getdata cmpctblock 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) peer=44
     22016-12-12 06:08:08.295022 send sendcmpct (no announce) ver=1 peer=52
     32016-12-12 06:08:08.295131 send sendcmpct (announce) ver=1 peer=44
     42016-12-12 06:08:08.343020 recv best cmpctblock 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) age=40s size=5660 peer=52
     52016-12-12 06:08:08.343120 MarkBlockAsReceived: first=44, peer=52
     62016-12-12 06:08:08.345246 send getblocktxn 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) indexes=2 size=59 peer=52
     72016-12-12 06:08:08.454836 recv best header 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) age=40s peer=51
     82016-12-12 06:08:08.627894 Incoming cmpctblock (4248 of 5660 bytes) chk=33157dc1 from peer=44
     92016-12-12 06:08:08.707884 recv best cmpctblock 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) age=40s size=5660 peer=56
    102016-12-12 06:08:08.707987 MarkBlockAsReceived: first=52, peer=56
    112016-12-12 06:08:08.710195 send getblocktxn 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) indexes=2 size=59 peer=56
    122016-12-12 06:08:08.767387 recv blocktxn 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 (443086) indexes=2 size=3514 expected-peer=56 peer=52
    132016-12-12 06:08:08.779325 made block 000000000000000001eaac17c38bb5734549af9fefda55eb6baee5e5f974ef96 size=437170 txs: mempool=895 req=2
    

11. rebroad referenced this in commit 8f664b7011 on Dec 12, 2016
12. rebroad referenced this in commit 64ee10c87e on Dec 12, 2016
13. 
    rebroad commented at 7:23 am on December 12, 2016: contributor

    Also speeds up block propagation in instances where only the full block would have been requested:-

     02016-12-12 07:18:43.335164 recv new best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=32s peer=4
     12016-12-12 07:18:43.335368 send getdata block 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) peer=4
     22016-12-12 07:18:43.982027 Incoming cmpctblock (7176 of 11460 bytes) chk=2cebf26b from peer=5
     32016-12-12 07:18:44.004116 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=15
     42016-12-12 07:18:44.174490 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=1
     52016-12-12 07:18:44.185140 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=11
     62016-12-12 07:18:44.187417 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=13
     72016-12-12 07:18:44.192600 recv best cmpctblock 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s size=11460 peer=5
     82016-12-12 07:18:44.192702 MarkBlockAsReceived: first=4, peer=5
     92016-12-12 07:18:44.198876 send getblocktxn 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) indexes=4 size=63 peer=5
    102016-12-12 07:18:44.540087 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=3
    112016-12-12 07:18:44.641163 recv best header 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=33s peer=2
    122016-12-12 07:18:44.758474 recv blocktxn 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) indexes=4 size=1596 peer=5
    132016-12-12 07:18:44.784560 made block 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 size=998001 txs: mempool=1869 req=4
    142016-12-12 07:18:57.060002 UpdateTip: new best=000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) ver=0x20000000 age=46s tx=1874
    152016-12-12 07:18:57.060115  warning='2 of last 100 blocks have unexpected version'
    162016-12-12 07:18:57.500227 Incoming block (174156 of 998001 bytes) chk=64d1c13a from peer=4
    172016-12-12 07:19:26.547916 recv block 000000000000000003087cfe7be23ba7c120a77e6ab73c3d54504e170e121b03 (443090) age=75s size=998001 peer=4
    

    A further improvement could be to disconnect the node sending the full-block once we have validated the block constructed from blocktxns to save bandwidth. Out of scope for #9325 though.

14. rebroad referenced this in commit 6820abbe1f on Dec 13, 2016
15. rebroad referenced this in commit cbf1a2684e on Dec 13, 2016
16. rebroad referenced this in commit 79a420d360 on Dec 13, 2016
17. rebroad referenced this in commit 12a4b6bffc on Dec 13, 2016
18. rebroad referenced this in commit 3ceb0f1d4a on Dec 14, 2016
19. rebroad referenced this in commit e582c0f596 on Dec 16, 2016
20. rebroad referenced this in commit 21cd2736dc on Dec 18, 2016
21. rebroad referenced this in commit 9e476fdf4d on Dec 20, 2016
22. rebroad referenced this in commit e54f58ebed on Dec 23, 2016
23. rebroad referenced this in commit efb1047f66 on Dec 23, 2016
24. rebroad referenced this in commit 859819b9de on Dec 23, 2016
25. rebroad referenced this in commit e0f7575e6c on Dec 24, 2016
26. rebroad referenced this in commit 3c95c299bb on Dec 26, 2016
27. rebroad referenced this in commit ab96a4b661 on Dec 26, 2016
28. rebroad referenced this in commit 5fa73e1e5b on Dec 26, 2016
29. rebroad referenced this in commit b49c8ae737 on Dec 26, 2016
30. rebroad referenced this in commit de372b0c5a on Dec 27, 2016
31. rebroad referenced this in commit 424529e081 on Dec 27, 2016
32. rebroad referenced this in commit c34a8314a2 on Dec 27, 2016
33. fanquake closed this on Oct 7, 2017

    ---

34. MarcoFalke locked this on Sep 8, 2021

Contributors
rebroad