[Qt] Add option to pause/resume block downloads

jonasschnelli commented at 5:35 pm on January 10, 2017: contributor

This, almost UI only change, will add a Pause/Resume button to the modal overlay to pause/resume block downloads during IBD.

This is an effective way to pause/resume IBD during a time when the computers resources are required somewhere else.

jonasschnelli added the label GUI on Jan 10, 2017

jonasschnelli force-pushed on Jan 10, 2017

paveljanik commented at 5:37 pm on January 10, 2017: contributor

Isn’t network toggle button usable in this case? If it is not, let’s fix it instead…

jonasschnelli commented at 6:05 pm on January 10, 2017: contributor

I extracted this from my SPV branch. Especially there, it is very useful to pause IBD and continue with SPV during a time where you don’t want to use all available resources on verification.

But also without SPV, I think this can be useful (pause IBD and not loose broadcast capabilities, fetch headers but not the blocks)…

luke-jr commented at 6:06 pm on January 10, 2017: member

IMO it would be too confusing to be worth it without “SPV” mode, but probably should go in after the latter is.

MarcoFalke commented at 6:33 pm on January 10, 2017: member

Isn’t network toggle button usable in this case? If it is not, let’s fix it instead…

Imo those are different features, but I agree that the GUI should not “diverge” in regard to presenting features. The toggle network functionality should be removed from the network icon and a proper button should be put beside the new “Pause” button?

luke-jr commented at 6:44 pm on January 10, 2017: member

The “Pause” button won’t be visible in normal cases.

jonasschnelli commented at 7:16 pm on January 10, 2017: contributor

Isn’t network toggle button usable in this case? If it is not, let’s fix it instead…

Imo those are different features, but I agree that the GUI should not “diverge” in regard to presenting features. The toggle network functionality should be removed from the network icon and a proper button should be put beside the new “Pause” button?

Yes. These are internally two completely different features. Expose to the users, these have similar effects.

The “Pause” button won’t be visible in normal cases.

Yes. The modal overlay is currently only accessible during IBD. Though, we could extend it to support a state where the chain is in-sync and show it when someone click on the network statusbar icon.

in src/net_processing.h: in abf0941e09 outdated

13@@ -14,6 +14,10 @@ void RegisterNodeSignals(CNodeSignals& nodeSignals);
14 /** Unregister a network node */
15 void UnregisterNodeSignals(CNodeSignals& nodeSignals);
16 
17+/** if disabled, blocks will not be requested automatically, usefull for low-resources-available mode */

kallewoof commented at 10:10 pm on February 28, 2017:

Typo: useful~l~ (one L)

in src/qt/clientmodel.h: in abf0941e09 outdated

80@@ -81,6 +81,10 @@ class ClientModel : public QObject
81     QString formatClientStartupTime() const;
82     QString dataDir() const;
83 
84+    // get/set state about autorequesting-blocks during IBD
85+    bool isAutorequestBlocks() const;

kallewoof commented at 10:11 pm on February 28, 2017:

Nit: isAutorequestingBlocks (add ing). The setAutorequestBlocks below is fine as is, though.

kallewoof commented at 10:13 pm on February 28, 2017: member

utACK abf0941

laanwj added this to the milestone 0.15.0 on Mar 14, 2017

laanwj commented at 8:57 am on March 14, 2017: member

Added 0.15 milestone

jonasschnelli commented at 2:22 pm on March 17, 2017: contributor

Rebased.

jonasschnelli force-pushed on Mar 17, 2017

in src/net_processing.h:28 in e2e86ec771 outdated

22@@ -23,6 +23,10 @@ void RegisterNodeSignals(CNodeSignals& nodeSignals);
23 /** Unregister a network node */
24 void UnregisterNodeSignals(CNodeSignals& nodeSignals);
25 
26+/** if disabled, blocks will not be requested automatically, usefull for low-resources-available mode */
27+static const bool DEFAULT_AUTOMATIC_BLOCK_REQUESTS = true;
28+extern std::atomic<bool> fAutoRequestBlocks;

laanwj commented at 10:25 am on April 19, 2017:

Instead of exporting this variable (whose definition is an implementation detail) from net_processing.h I’d prefer a setter/getter, e.g. SetAutoRequestBlocks(bool) GetAutoRequestBlocks()

jonasschnelli commented at 10:26 am on April 19, 2017:

Agree. Will change…

in src/qt/clientmodel.cpp:237 in e2e86ec771 outdated

233@@ -233,6 +234,22 @@ QString ClientModel::dataDir() const
234     return GUIUtil::boostPathToQString(GetDataDir());
235 }
236 
237+bool ClientModel::isAutorequestBlocks() const

laanwj commented at 10:26 am on April 19, 2017:

isAutorequestBlocks is a bit of a strange name; isAutoRequestingBlocks maybe?

laanwj commented at 10:40 am on April 19, 2017: member

When re-enabling AutoRequestBlocks, what ‘kicks off’ the block requesting process again? Setting the flag will make it request blocks the next time FindNextBlocksToDownload is called; is that good enough? I suppose it is, because SendMessages is called periodically (every 100 ms?).

jonasschnelli force-pushed on Apr 20, 2017

jonasschnelli commented at 8:33 am on April 20, 2017: contributor

Overhauled the PR and addresses @laanwj points.

Also, I added the info “Blocks requested from peers” (blocks in flight). This may be important because pause will not result in disconnecting peers. Already requested blocks will be downloaded (and verified) in the “pause” state.

If blocks are in flight and the pause has been triggered, there is now a special info label Wait to finish current downloads.

paveljanik commented at 9:10 am on April 20, 2017: contributor

Well, I have to change my previous opinion. I think this can be useful.

Concept ACK

Will test soon.

paveljanik commented at 10:17 am on April 25, 2017: contributor

Needs rebase.

ryanofsky commented at 6:35 pm on May 1, 2017: member

ACK e952b67d294043d9758b37bf2be57a982b41c051

Code looks good. On the UI feature, two minor comments:

Pausing the download throws off “Estimated time left till synced.” It would be better pausing didn’t affect estimated time.
Pause/Resume button really sticks out where it’s currently placed. Maybe it would make more sense next to the progress bar or near the hide button.

sipa commented at 10:54 pm on May 1, 2017: member

Needs rebase.

jonasschnelli force-pushed on May 11, 2017

jonasschnelli commented at 1:26 pm on May 11, 2017: contributor

Rebased.

paveljanik commented at 4:56 pm on May 15, 2017: contributor

Testing this again.

ACK https://github.com/bitcoin/bitcoin/pull/9502/commits/fc84323d8ee447f1461a10b7f3b29d113f9f4a43

I think this could be even more usable if it can be called once fully in sync with the network. But I can’t display the overlay then…

laanwj commented at 6:54 am on May 24, 2017: member

I think this could be even more usable if it can be called once fully in sync with the network. But I can’t display the overlay then…

Hm I vaguely remember I added that functionality once, you should be able to bring up the overlay by the secret trick of clicking on the sync icon.

paveljanik commented at 6:57 am on May 24, 2017: contributor

@laanwj But the sync icon (you mean the triangle with an exclamation mark inside?) is not displayed when you are “in sync” with the network.

jonasschnelli commented at 6:57 am on May 24, 2017: contributor

I think this could be even more usable if it can be called once fully in sync with the network. But I can’t display the overlay then…

There are four ways how the modal-overlay can be opened: -> auto-opens when in IBD/sync -> Click on the warning icons next to the balance -> Click on the progress bar during IBD/sync -> Click on the sync icon in the status bar

Though I agree with you, there is no option how to open it once you are in sync… which could be useful, but independent to this PR.

laanwj commented at 7:03 am on May 24, 2017: member

@paveljanik Oh, though it also worked with the checkmark that takes its place.

jonasschnelli commented at 7:05 am on May 24, 2017: contributor

@paveljanik Oh, though it also worked with the checkmark that takes its place.

Yes. We should probably allow that.

sdaftuar commented at 6:43 pm on June 1, 2017: member

Concept ACK. (I haven’t reviewed the qt code at all, but I did look over the net_processing changes.)

What is the intended behavior if the user turns off block download after being synced or nearly synced? I believe this PR only disables the parallel fetch logic, and not the direct fetch – so block download could conceivably be disabled by the user, but blocks could still be requested as they are announced. (Maybe this is hard to accomplish, if the modal overlay is not accessible when bitcoind is close to synced, but my recollection is that the modal overlay becomes visible whenever the headers chain is out of sync, which can happen even after leaving IBD, does that sound right?)

Perhaps the ability to disable block download should disappear if we’re close to caught up, eg if CanDirectFetch() is true?

Additionally, perhaps if we’ve disabled block download, we should also disable transaction download.

jonasschnelli force-pushed on Jun 29, 2017

jonasschnelli commented at 3:18 pm on June 29, 2017: contributor

Rebased and addressed the CanDirectFetch() issue @sdaftuar mentioned.

The current implementation avoids exposing net_processing.cpp’s CanDirectFetch() for the reasons a) it does always use chainActive (lock) and b) to avoid another core-layer dependency.

I’m not doing to tackle the transaction download in the PR (main scope is disabling block download).

achow101 commented at 9:48 pm on July 12, 2017: member

ACK

I have tested this on a node that was a few weeks behind and I was able to pause the block download multiple times while it was syncing. I noticed that even when paused, UpdateTip is still happening for hundreds of blocks after what the GUI said still needed to be downloaded. I assume this is because it still processes all of the blocks that have been downloaded and the downloading happens faster than the processing. The UpdateTip lines do stop after a while which I assume is because all downloaded blocks have been processed.

sipa commented at 10:37 pm on July 12, 2017: member

I assume this is because it still processes all of the blocks that have been downloaded and the downloading happens faster than the processing.

Yes, that’s expected. We download multiple blocks in parallel, ahead of the next one to connect. I expect that this PR also keeps processing the ones that were in flight at the time blocks downloads are disabled.

in src/qt/forms/modaloverlay.ui:357 in f1b057f69e outdated

353@@ -334,6 +354,33 @@ QLabel { color: rgb(40,40,40);  }</string>
354             <number>10</number>
355            </property>
356            <item>
357+            <widget class="QPushButton" name="pauseResumeVerification">

promag commented at 8:27 pm on July 17, 2017:

Nit, rename toggleDownloadButton.

jonasschnelli commented at 10:18 am on August 18, 2017:

I prefer to keep it because it’s not pure download, it’s the whole verification (and maybe we further extend what it does in the background later).

in src/qt/forms/modaloverlay.ui:356 in f1b057f69e outdated

353@@ -334,6 +354,33 @@ QLabel { color: rgb(40,40,40);  }</string>
354             <number>10</number>
355            </property>
356            <item>

promag commented at 8:28 pm on July 17, 2017:

Nit, can’t comment there, but remove leftMargin above to be consistent.

jonasschnelli commented at 9:33 pm on November 16, 2017:

Why remove?

in src/qt/modaloverlay.cpp:210 in f1b057f69e outdated

203+
204+void ModalOverlay::updatePauseState(bool pauseActive)
205+{
206+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks": "Blocks requested from peers"));
207+    ui->pauseResumeVerification->setText((pauseActive ? "Resume downloading blocks ": "Pause downloading blocks"));
208+    ui->infoLabel->setText((pauseActive && getAmountOfBlocksInFlight() > 0 ? "Wait to finish current downloads...": ""));

promag commented at 8:36 pm on July 17, 2017:

I don’t know if you plan to show other messages for this label, but if not then toggle visibility like above?

promag commented at 8:49 pm on July 17, 2017:

Also, when paused, nBlocksInFlight doesn’t always decrease to zero.

jonasschnelli commented at 5:49 pm on September 7, 2017:

I think this is fine how it is, as long as nBlocksInFlight > 0 we need to expect network traffic and therefor should still remind the user with the "Wait to finish current...".

in src/qt/modaloverlay.cpp:208 in f1b057f69e outdated

201+    updatePauseState(pauseActive);
202+}
203+
204+void ModalOverlay::updatePauseState(bool pauseActive)
205+{
206+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks": "Blocks requested from peers"));

promag commented at 8:52 pm on July 17, 2017:

Don’t see the need to change label, maybe a generic Downloading: %d blocks?

jonasschnelli commented at 5:48 pm on September 7, 2017:

I think the currents PR label switch makes it more clear what happens in the background (pause syncing will not lead to immediate network download halt).

laanwj added this to the milestone 0.16.0 on Jul 18, 2017

laanwj removed this from the milestone 0.15.0 on Jul 18, 2017

laanwj commented at 3:20 pm on July 18, 2017: member

This has missed the 0.15 feature freeze, moving to 0.16.

jonasschnelli force-pushed on Sep 7, 2017

jonasschnelli commented at 6:02 pm on September 7, 2017: contributor

Rebased.

jonasschnelli force-pushed on Sep 7, 2017

jonasschnelli commented at 7:48 pm on September 7, 2017: contributor

rebased and slightly overhauled the blocks-in-flight information flow.

jonasschnelli force-pushed on Oct 5, 2017

jonasschnelli commented at 4:00 am on October 5, 2017: contributor

Rebased. @paveljanik @achow101 @ryanofsky @kallewoof: care to re-ack?

in src/net_processing.cpp:3783 in 1375063565 outdated

3318+
3319+bool isAutoRequestingBlocks() {
3320+    return fAutoRequestBlocks;
3321+}
3322+
3323+unsigned int getAmountOfBlocksInFlight() {

kallewoof commented at 4:05 am on October 5, 2017:

I think they should all start with capital or non-capital (SetAuto vs isAuto/getAmount) for consistency.

meshcollider commented at 9:28 am on October 5, 2017:

Also nit, the braces should be on new lines

kallewoof approved

kallewoof commented at 4:08 am on October 5, 2017: member

utACK 1375063565a257057707379588b02a8aa44b7bcc (my QT skills are weak though)

in src/net_processing.h:70 in 1375063565 outdated

63@@ -61,4 +64,10 @@ bool GetNodeStateStats(NodeId nodeid, CNodeStateStats &stats);
64 /** Increase a node's misbehavior score. */
65 void Misbehaving(NodeId nodeid, int howmuch);
66 
67+void SetAutoRequestBlocks(bool);
68+bool isAutoRequestingBlocks();
69+
70+/** retruns the amount of blocks in flight (in total) */

paveljanik commented at 7:45 am on October 5, 2017:

typo retruns

in src/qt/modaloverlay.cpp:142 in 1375063565 outdated

137+    // show already requested blocks (in total)
138+    ui->numberBlocksRequested->setText(QString::number(getAmountOfBlocksInFlight()));
139+    eventuallyShowHeaderSyncing(count);
140+    updatePauseState(verificationPauseActive);
141+
142+    // disable pause button when we we can fetch directly

paveljanik commented at 7:47 am on October 5, 2017:

we we

in src/qt/modaloverlay.cpp:144 in 1375063565 outdated

139+    eventuallyShowHeaderSyncing(count);
140+    updatePauseState(verificationPauseActive);
141+
142+    // disable pause button when we we can fetch directly
143+    // avoid using the core-layer's existing CanFetchDirectly()
144+    bool canFetchDirecly = (blockDate.toTime_t() > GetAdjustedTime() - Params().GetConsensus().nPowTargetSpacing * 20);

meshcollider commented at 9:16 am on October 5, 2017:

Typo in variable name canFetchDirecly, should be Directly

in src/qt/modaloverlay.cpp:208 in 1375063565 outdated

203+    updatePauseState(pauseActive);
204+}
205+
206+void ModalOverlay::updatePauseState(bool pauseActive)
207+{
208+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks": "Blocks requested from peers"));

meshcollider commented at 9:18 am on October 5, 2017:

nit, space missing before : (same on next two lines)

in src/qt/modaloverlay.cpp:209 in 1375063565 outdated

204+}
205+
206+void ModalOverlay::updatePauseState(bool pauseActive)
207+{
208+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks": "Blocks requested from peers"));
209+    ui->pauseResumeVerification->setText((pauseActive ? "Resume downloading blocks ": "Pause downloading blocks"));

meshcollider commented at 9:20 am on October 5, 2017:

nit, space at the end of the first string instead of before :

meshcollider commented at 9:26 am on October 5, 2017: contributor

utACK https://github.com/bitcoin/bitcoin/pull/9502/commits/1375063565a257057707379588b02a8aa44b7bcc

Just a couple of formatting nits inline

jonasschnelli force-pushed on Nov 16, 2017

jonasschnelli commented at 9:43 pm on November 16, 2017: contributor

Rebased and addressed nits (code style).

in src/net_processing.cpp:3783 in f820e86450 outdated

3661+bool isAutoRequestingBlocks()
3662+{
3663+    return fAutoRequestBlocks;
3664+}
3665+
3666+unsigned int getAmountOfBlocksInFlight() {

kallewoof commented at 2:39 am on November 17, 2017:

Brace starts on same line for this function but on different lines for above two functions. I think separate line is the preferred style these days, but I could be wrong?

kallewoof commented at 2:40 am on November 17, 2017: member

utACK f820e8645071e66b33fd97bcaaab8516ca429f77

jonasschnelli force-pushed on Nov 30, 2017

laanwj removed this from the milestone 0.16.0 on Jan 11, 2018

laanwj added this to the milestone 0.17.0 on Jan 11, 2018

jonasschnelli force-pushed on Mar 19, 2018

jonasschnelli commented at 11:51 am on March 20, 2018: contributor

@cfields: since this is touching net code, can you do a final review?

jonasschnelli assigned theuni on Mar 20, 2018

fanquake commented at 3:19 pm on March 21, 2018: member

Had a quick test of this (b0a9b8d6dae2a9efcdbc3607b235c37e1f7db31d) on top of master (9b8b1079ddab64ac955766536c38d23dc57bc499). macOS 10.13.3 and Qt 5.10.1.

This is what the overlay model looks like now with this PR (b0a9b8d6dae2a9efcdbc3607b235c37e1f7db31d): 147511

One thing I noticed is after hitting “Pause”, it can take a while to stop downloading blocks. i.e In the images below, we hit pause with 127 blocks “requested from peers” and 145022 total blocks left. However we don’t appear to stop downloading blocks until we’ve reached 144601 total blocks left to download, > 400 blocks later. This could be confusing, and might appear worse on slower connections?

I’ll comment inline about simplifying the new strings.

in src/qt/modaloverlay.cpp:208 in b0a9b8d6da outdated

203+    updatePauseState(pauseActive);
204+}
205+
206+void ModalOverlay::updatePauseState(bool pauseActive)
207+{
208+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks" : "Blocks requested from peers"));

fanquake commented at 3:23 pm on March 21, 2018:

I’d suggest changing this string to just “Blocks in download queue” or similar. Then we don’t have to worry about switching strings between paused and downloading.

in src/qt/modaloverlay.cpp:210 in b0a9b8d6da outdated

205+
206+void ModalOverlay::updatePauseState(bool pauseActive)
207+{
208+    ui->labelNumberBlocksRequested->setText((pauseActive ? "Finish downloading blocks" : "Blocks requested from peers"));
209+    ui->pauseResumeVerification->setText((pauseActive ? "Resume downloading blocks " : "Pause downloading blocks"));
210+    ui->infoLabel->setText((pauseActive && getAmountOfBlocksInFlight() > 0 ? "Wait to finish current downloads..." : ""));

fanquake commented at 3:25 pm on March 21, 2018:

Maybe use “Downloading blocks remaining in queue..” or similar here?

jonasschnelli commented at 7:11 pm on April 10, 2018: contributor

One thing I noticed is after hitting “Pause”, it can take a while to stop downloading blocks. i.e In the images below, we hit pause with 127 blocks “requested from peers” and 145022 total blocks left. However we don’t appear to stop downloading blocks until we’ve reached 144601 total blocks left to download, > 400 blocks later. This could be confusing, and might appear worse on slower connections?

Yes. That could be confusing,.. though I guess there is an explanation. You may still have blocks on your disk that hasn’t been verified due to a missing the next blocks after your tip. This means it is possible to jump a couple of block after downloading a single block.

Also,… we don’t want to ignore the request because the peer is sending it anyway (once requested) and we don’t want to disconnect peers (that feature is already available).

It would be possible to hide that information from the user, though I think it makes more sense to display it so its clear that the “pause” means not “disconnect”.

jonasschnelli force-pushed on Apr 11, 2018

jonasschnelli commented at 8:00 am on April 11, 2018: contributor

Rebased. Thanks for re-reviewing.

jonasschnelli force-pushed on Apr 18, 2018

jonasschnelli commented at 6:16 pm on April 22, 2018: contributor

Rebased.

MarcoFalke added the label Needs rebase on Jun 6, 2018

DrahtBot removed the label Needs rebase on Jul 14, 2018

DrahtBot commented at 12:55 pm on July 22, 2018: member

DrahtBot closed this on Jul 22, 2018

DrahtBot reopened this on Jul 22, 2018

in src/qt/bitcoingui.cpp:463 in 20795517da outdated

492@@ -493,6 +493,10 @@ void BitcoinGUI::setClientModel(ClientModel *_clientModel)
493         connect(_clientModel, SIGNAL(networkActiveChanged(bool)), this, SLOT(setNetworkActive(bool)));
494 
495         modalOverlay->setKnownBestHeight(_clientModel->getHeaderTipHeight(), QDateTime::fromTime_t(_clientModel->getHeaderTipTime()));
496+        modalOverlay->setPauseResumeState(!_clientModel->isAutoRequestingBlocks());
497+        connect(modalOverlay, SIGNAL(requestVerificationPauseOrResume()), _clientModel, SLOT(toggleAutoRequestBlocks()));
498+        connect(_clientModel, SIGNAL(verificationProgressPauseStateHasChanged(bool)), modalOverlay, SLOT(setPauseResumeState(bool)));

MarcoFalke commented at 7:20 pm on July 22, 2018:

could use the new connect syntax, so it wouldn’t have to be changed again in the future?

laanwj removed this from the milestone 0.17.0 on Jul 23, 2018

laanwj added this to the milestone 0.18.0 on Jul 23, 2018

DrahtBot added the label Needs rebase on Aug 21, 2018

DrahtBot commented at 10:36 am on August 21, 2018: member

Add fAutoRequestBlocks to disabled/enable the verification progress 3a3f2b0b79

[Qt] update header-syncing progress when not processing a block 62ce306fdb

[Qt] Add option to pause/resume block downloads 0e155d170e

jonasschnelli commented at 7:37 pm on October 17, 2018: contributor

Rebased.

jonasschnelli force-pushed on Oct 17, 2018

Rename autorequest blocks methods c03274017e

Add setter/getter for fAutoRequestBlocks 5189ca497d

Show blocks-in-flight 95686e988c

Show special info when autodownload is disabled but blocks are in flight 53027a91c8

Disable "pause downloading blocks" when we can fetch blocks directly 718ceefc32

jonasschnelli force-pushed on Oct 17, 2018

in src/qt/modaloverlay.cpp:144 in 718ceefc32

139+    eventuallyShowHeaderSyncing(count);
140+    updatePauseState(verificationPauseActive);
141+
142+    // disable pause button when we can fetch directly
143+    // avoid using the core-layer's existing CanFetchDirectly()
144+    bool canFetchDirectly = (blockDate.toTime_t() > GetAdjustedTime() - Params().GetConsensus().nPowTargetSpacing * 20);

practicalswift commented at 6:33 am on October 19, 2018:

nit: Redundant parentheses :-)

laanwj removed this from the milestone 0.18.0 on Feb 6, 2019

fanquake added the label Up for grabs on Jun 17, 2019

fanquake commented at 9:34 am on June 17, 2019: member

Discussion has dragged on and died out here. Going to label “Up for grabs” and close for now.

fanquake closed this on Jun 17, 2019

laanwj removed the label Needs rebase on Oct 24, 2019

MarkLTZ referenced this in commit 6293e78f97 on Nov 17, 2019

MarcoFalke locked this on Dec 16, 2021

fanquake removed the label Up for grabs on Aug 4, 2022

[Qt] Add option to pause/resume block downloads #9502