Slow Sync & 2 GUI bugs #12514

<!-- Describe the issue -->

Slow Initial Sync - Every time I open the client (0.15.1 - 64 Bit) it is slow to sync and the network shows 50% of the time it does nothing. Also I suspect my 16-26 peers are not sending data, and suggest peer limit variable so it can be increased at will.

Possibly related to #11560

<!--- What behavior did you expect? -->

Fast Initial Sync - constant data transfer (download) and enough peers so that there is always peers sending data

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

Download core client from GH and install it, open it.

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

AMD Phenom X1100T BLACK 6 core processor, 16 GB ram, 1 x 2TB HDD, 1 x 2TB HSSD (Hybrid SSD)

<!-- Any extra information that might be useful in the debugging process. -->

• LONG breaks in network transfer
• Too few peers (IMO, which is why need peer variable)
• Network window looses data every time the time slider is changed... It should adjust and show the current data in the new time frame. And make it track with the debug window closed.
• Port is forwarded (and seen from outside - connections in and out)
• nice -n -20 bitcoin-qt does not help
• 6000 DBCache not help
• Added bunch of nodes/seednodes/connects
• Still have to make sure the userdir is on the HSSD (I confuse the drive manufacturers sometimes)

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

Config: server=1 daemon=1

Debug.log - unable to fit under maximum for every pastebin website in sight :-/

Thanks for your help :)

And more data on the 30 minute timeframe. I will change it now, loose that data and watch it for an hour... Just a shot of how many users I usually have- I suspect most or all of them are stopping sending data. Suggest adding a "peerlimit" variable so peer limit can be adjusted.

UPDATE:

I started a compile of a git clone of the source, and it just completed... Turns out I was not using the HSSD so I moved it to the HSSD, and ran it, and it resync'd the already downloaded part lightning fast, and then I looked at debug window, and peers just kept on climbing higher and higher... I got over 150 peers!

I looked at the version number and it is still the same, maybe a patch was applied or this is a snapshot and version number not yet updated.

I closed the client, ran the make install, and now I think I should be good...

EDIT: Turns out I've screwed my previous resync, this is the second time I got over 50%. There has to be a better way, can it not just look and see the files are there and continue from last time?

Update: I ran sudo nice -n -10 sudo -u MYUSERNAME bitcoin-qt -data-dir=/path/to/my/hssd/.bitcoin/ & with a -reindex-chainstate on the end

Now the niceness is changing properly, and the average % per hour increased a bi - but not only that, it is holding solid now!

I think a combination of things fixed it, the recompile with git clone, moving to HSSD, upping dbcache to 6K, and fixing the niceness to work.

Unfortunately I still have to start back from 0% for the millionth timebut the network stream looks constant :)

Here we see the network stream is now constant :D

Here we see over 200 peers :D Only concern here is the lack of inbound connections that were there just before I restarted bitcoin-qt

It looks like right now (correct if wrong) it is analyzing the data on disk and only downloading the headers? Maybe that is why after 7% it only downloaded 36 MB? I suspect once it runs out of data on disk I will see the network traffic increase again...

PLS HODL FOR CONFIRM

It appears that... maybe it is fixed... But there seems to be a bug in updating this little modal-like box here, it shows I only 11% but looking at the progress bar at bottom I am clearly at least 50%

Also there is a bug in the network performance tab slider (deletes data on changing slider/close, doesnt record data when debug window closed).

I'm a computer programmer and I can understand why a regular person would be frustrated with this... I am frustrated with this...

Fail:

So the peers/connections drop off and data transfer to 0.

I am closing the wallet, but it has taken so far 1 hour to close ("shutting down, please no close")

I will re-open it again without the reindex option, and hopefully I retain my %age

Help would be appreciated...

Crap.... It finished reindexing but data/peers is now 0

I closed the wallet and have been waiting for this "do not shut down" window to close for 3-4 hours now... I don't want to close it and have to start back from 0 all over again...

Waiting it out I guess...

FINALLY after 4 or 5 hours the message disappeared and I re-opened the GUI:

So to everyone: WAIT for the window to close. If you close early and it screws it, run with -reindex-chainstate on the end. WAIT for it to reindex the data on the disk (it takes as long as it did to download/process) and then close/reopen (without reindex option) and WAIT for it to continue loading the blockchain from it's last state.

Now after several days of frustration I am damn near close to the full chain :) Patience and persistence seems to be key.

Update: Now 50 weeks to go... Anyone having this issue should follow this thread. I will close it with summary if I reach a full sync. Few hours to go still but very confident :D

Confirmed Working I am not sure whether it was the update made in relation to #11560 or the changes that I made, or both... But I have solved my problem! Summary of changes for those experiencing the same issue is below.

Proof I am now nearly done downloading the blockchain:

I had to stop the client and wait for the window to close many times because the peers either disappeared or stopped sending data. Wait for window to close. If you have no choice and have to close it or you crash and it fails to start, run with the reindex-chainstate and let it run. Then run it without reindex option.

STEPS TO FIX IF YOU EXPERIENCE SAME ISSUE:

1. Move your /home/user/.bitcoin folder to your best/fastest hard drive. -- SSD (Solid State) is obviously best choice -- HSSD (Hybrid Solid State) is second choice -- HDD (Regular HDD - IDE/SATA - NON SSD) is LAST choice

2. Open Bitcoin-QT with the following command: -- sudo nice -n -20 sudo -u MYUSERNAME bitcoin-qt -data-dir=/path/to/my/hssd/.bitcoin/ &

3. Wait for download - monitor peer count / network transfer in the debug window (help -> debug)

4. If the peer count drops to 0 and/or the network transfer drops, close Bitcoin but WAIT for the window to disappear - I waited 6 hours for it to disappear. It is basically saving it's state, and if you quit it looses track of where it was.

IF YOU CRASHED OR CLOSED THE WINDOW that says not to close it, you can reindex the data on disk by performing the following:

1. Run Bitcoin with reindex: -- sudo nice -n -20 sudo -u MYUSERNAME bitcoin-qt -data-dir=/path/to/my/hssd/.bitcoin/ -reindex-chainstate &

2. Once completed, it may resume downloading Blockchain, OR it may stop all-together (0 peers, 0 data)

3. If it completes and does NOT resume downloading, close Bitcoin-QT again, WAIT AGAIN for window to close

4. Run Bitcoin without reindex: -- sudo nice -n -20 sudo -u MYUSERNAME bitcoin-qt -data-dir=/path/to/my/hssd/.bitcoin/ &

I hope anyone else with this issue has the same success as me, and that this information is helpful. Message me or post here if you need help with this issue.

Ticket left open as GUI bug not resolved: -- Debug window does not record network data while closed -- Network graph looses data when the slider to change scope of graph is adjusted -- Closing debug window looses network graph -- During reindex, the progress bar does not show anything/shows wrong value

After several days of hard work I have a 100% sync'd node :D

Thanks to MyS5Droid for the help, and the developers for their hard work in this project!

Thanks for your detailed report!

Could you close this issue, and report a new short concise bug report which you focus on one bug? It makes easier for everyone to get a good overview what the issue is.