bitcoind shouldn't fail to progress with synchronization: endless [leveldb] Generated table ... logs

GregTonoski commented at 3:00 pm on February 16, 2025: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

bitcoind doesn’t progress with synchronization despite approx. 4 months lag, doesn’t consume much CPU or RAM, uses disk IO, produces large amount of logs like e.g. [leveldb] Generated table [#3982690](/bitcoin-bitcoin/3982690/)@4: 31588 keys, 2162389 bytes for hours (there was a test for more than 8 hours).

Reproduced 3 times out of 3 tests (one with a pre-built version 27).

Expected behaviour

bitcoind finishes synchronization in an hour or so.

Steps to reproduce

Preconditions:

Bitcoin 28.1
Pruned node (550MB)

Steps:

Launch bitcoind bitcoind -datadir=/mnt/sd/Bitcoin_data -daemon=0 -debug=all -blocksonly=1 -debuglogfile=/tmp/debug.log -prune=550 -dbcache=2048

Relevant log output

2025-02-16T13:04:33Z [leveldb] Generated table [#3987352](/bitcoin-bitcoin/3987352/)@3: 31812 keys, 2161649 bytes

Also: error code: -28 error message: Starting network threads…

after the command: bitcoin-cli stop.

logs.tar.gz

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

28.1

Operating system and version

Arch Linux

Machine specifications

Intel x64, 4GB RAM, 32GB SD (exfat), 1Gbit Ethernet, 100/10 Mbit/s FTTH.

GregTonoski renamed this:
~~bitcoind shouldn't fail to progress with synchronization: endless [leveldb] Generated table... logs~~
bitcoind shouldn't fail to progress with synchronization: endless [leveldb] Generated table ... logs
on Feb 16, 2025

l0rinc commented at 4:04 pm on February 16, 2025: contributor

bitcoind finishes synchronization in an hour or so.

That alone would require a 1.5 Gbps download speed - and even then, we’re not there yet (especially with such a resource-constrained setup). But you could give AssumeUTXO a try to see if it helps - this will jump the process ahead to 95% complete and validate the blocks afterward.

bitcoind doesn’t progress with synchronization

The attached iostats and debug.log indicate that the bottleneck is the extremely slow write speed of the SD card (mmcblk0 is often below 1 MB/s), which is causing LevelDB to become the main limiting factor, dramatically slowing down the synchronization process. The logs show frequent LevelDB compaction activity in the chainstate database - meaning that during block verification, bitcoind is repeatedly compacting the database, severely hindering progress. Additionally, exFAT is quite problematic, moving the -datadir to an SSD should drastically improve performance.

Produces a large amount of logs

The -debug=all flag generates an excessive amount of logs and isn’t typically needed for normal operation.

GregTonoski commented at 4:41 pm on February 16, 2025: none

That alone would require a 1.5 Gbps download speed - and even then, we’re not there yet (especially with such a resource-constrained setup).

Disagree with the assumption of the 1.5 Gbps download speed requirement. There isn’t initial block download scenario here. There are 4 months of blocks data that remain to be downloaded here.

I would guess there is a livelock or deadlock or similar issue.

maflcko added the label Resource usage on Feb 17, 2025

maflcko commented at 7:11 am on February 17, 2025: member

Does it happen with a smaller (default) dbcache? Does it happen on a different filesystem than exfat? Does it happen on a different filesystem hardware, maybe something that is faster? (Can you benchmark the hardware?)

maflcko added the label Questions and Help on Feb 17, 2025

GregTonoski commented at 8:10 am on February 17, 2025: none

Does it happen with a smaller (default) dbcache?

Yes, it does happen with default settings, including default value 450 of dbcache.

sipa commented at 2:54 pm on February 18, 2025: member

would guess there is a livelock or deadlock or similar issue.

~~Is there progress at all? As in, is the number of validation blocks going not going up at all? That would be a bug.~~ EDIT: just saw you included logs. It’s just very slow, so clearly no deadlock or livelock.

If it’s just (very) slow, that may be due to low I/O speed of your disk + pruning + low dbcache. #28280 may speed things up somewhat (which will be in 29.0, not in any release yet).

GregTonoski commented at 6:53 pm on February 18, 2025: none

just saw you included logs. It’s just very slow, so clearly no deadlock or livelock

I would suggest investigating the number of “generated table …” logs. There shouldn’t be so many of them especially in comparison to a correct execution.

There are:

17176 logs out of which: — 15117 logs (or 88%) of the type: [leveldb] out of which: —– 6470 logs of the type: [leveldb] Generated table ....

maflcko commented at 7:14 pm on February 18, 2025: member

How fast is the hardware? What is the speed for contiguous reads/writes that do not go through the filesystem cache? What is the speed for random read/writes that also do not go through the filesystem cache?

GregTonoski commented at 7:35 pm on February 18, 2025: none

How fast is the hardware? What is the speed for contiguous reads/writes that do not go through the filesystem cache? What is the speed for random read/writes that also do not go through the filesystem cache?

Tell me what they are expected to be normally and I will measure the actual values, please.

GregTonoski commented at 9:00 pm on February 18, 2025: none

If it’s just (very) slow, that may be due to low I/O speed of your disk

Disagree. The alleged root cause is not supported by the data: %iowait = 30.51% kB_read = 21 624 950 kB_wrtn = 10 959 841 Only 31 blocks (~120 MB) downloaded during the time (c.a. 90 minutes), e.g. 2025-02-16T11:51:13Z UpdateTip: new best=000000000000000000020d77c16bf3836faf1ef3af28f5f31fbdd8955557d6f0 height=869931 version=0x3fffe000 log2_work=95.263443 tx=1114900731 date='2024-11-12T03:25:56Z' progress=0.952213 cache=2.0MiB(14666txo)

GregTonoski commented at 9:05 pm on February 18, 2025: none

20250218-debug.log

Does it happen on a different filesystem than exfat? Does it happen on a different filesystem hardware, maybe something that is faster? (Can you benchmark the hardware?)

Yes, the same occurs in the same environment except NTFS instead of exFAT and HDD instead of SD.

bitcoind shouldn’t fail to progress with synchronization: endless [leveldb] Generated table … logs #31882