Wallet rescan progress indicator not intuitive #9135

Issue

Wallet rescan progress indicator in debug.log is not intuitive. For example, 0.953423 doesn't mean 95% done. This isn't a big issue for me, but I want to report it because it may save the next person some time (unnecessary wait).

Steps to reproduce:

1. Import privkey with rescan
2. Tail the log

Expected behavior

I would expect that Progress=0.980699 means I'm 98% done.

Actual behavior

On testnet (obviously) I'm 0.98 complete (is that 98% complete?) rescanning block 576517, whereas currently height=1032846 or so. 2016-11-11 20:50:02 Still rescanning. At block 576517. Progress=0.980699 It's not done yet (2+ hours now), but I hope progress won't end up being greater than 1, because that would be even more confusing.

What version of bitcoin-core are you using?

0.12.1 with addrindex patch (source)

Machine specs:

• OS: Ubuntu 16.04
• CPU: ARMv8
• RAM: 1GB
• Disk Type (HD/SDD): SSD

Any extra information that might be useful in the debugging process.

Here's couple of sections from my debug.log. At block 53K it's already 0.17 done (wonderful) so instead of killing the daemon and retrying with no rescan, I decide to wait. Encouraged by the fast progress I keep waiting... and waiting. If I knew 0.17 doesn't really mean "17% done" I would have saved myself few hours of wait.

2016-11-11 19:24:49 Still rescanning. At block 53404. Progress=0.172387
2016-11-11 19:25:49 Still rescanning. At block 67174. Progress=0.276670
2016-11-11 19:26:49 Still rescanning. At block 79253. Progress=0.382439
...
2016-11-11 19:38:49 Still rescanning. At block 205610. Progress=0.813022
2016-11-11 19:39:49 Still rescanning. At block 205669. Progress=0.813941
...
2016-11-11 21:00:05 Still rescanning. At block 600560. Progress=0.983855
2016-11-11 21:01:05 Still rescanning. At block 600772. Progress=0.983972
...
2016-11-11 21:22:11 Still rescanning. At block 608068. Progress=0.986720
2016-11-11 21:23:11 Still rescanning. At block 608287. Progress=0.986824
...

2016-11-11 21:55:11 Still rescanning. At block 632290. Progress=0.989739
2016-11-11 21:56:11 Still rescanning. At block 632447. Progress=0.989824
2016-11-11 21:57:11 Still rescanning. At block 632818. Progress=0.989920

:) @jonasschnelli actually, the progress shows as less than it really is, so it shouldn't be causing unneccessary wait. i.e. if the last block downloaded is 0.989920 progress, then the rescan will finish at 0.989920, i.e. you won't need to wait for it to get to 1.000000.

The risk of the number being misleading is that you'll go away to make a sandwich thinking it is going to take longer than it does need, and will come back later than need be.

Very easy to fix, we just need to tweak the number to factor in that the 0.989920 (or whatever the progress of the last downloaded block is) is actually 1.000000, and that the number of the first block to rescan is 0.000000.

No, the progress indicator does not show less and there is no download involved. I's about rescan. The testnet blockchain has over 1 million blocks and 53,000 (the beginning of my log sample above) isn't 17% of that and 632,818 (the end of my sample) isn't 98.9% of that. (Assuming 0.xy means xy %.)

And no, it does cause extra wait because if it took me 3 minutes to get to 38%, it should take me less than 9 minutes to complete rescan. If I can tell online rescan is expected to take hours, I can stop bitcoind, rescan, start bitcoind again and likely save 50% of the total time required for online rescan. That doesn't have practical downside compared to online scan because during online rescan bitcoind can't do anything anyway.

In any case, either a clarification (docs) of what the progress bar indicates or a change in the code would make it easier for users to make informed decisions.

If this is confusing possibly the % should just be removed, and only "N of M blocks" shown.

Getting progress indicators to estimate necessary time in not-entirely-linear processes is very hard, and probably not worth the hassle in this case. Just ask Microsoft - since when do % indicators of installers, or file copying, give any realistic estimate of the time involved? In this case the early blocks are almost empty while later blocks are much more involved to scan through.

@laanwj it might be quite a simple fix to make the % more accurate. If the time to scan a block is related to its size, then the % calculation just needs to take into account the size of data so far known, and the size so far scanned to come up with a more accurate percentage. It seems it is not currently doing that for it to be considering up to block 53404 as 17% of the entire block chain.

The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Closing due to lack of interest. Pull requests with improvements are always welcome.