Feature Request - ZIP the blockchain

ranseier86 commented at 7:01 pm on April 29, 2019: none

Instead of prune there could be a ZIP mode. Zipping blk00010.dat with default Windows 10 build in ZIP shrinks it from 130 MB to 97 MB. Zipping blk00010.dat with 7zip default settings shrinks it from 130 MB to 77 MB. I love to have the whole blockchain stored but there is i huge potential to compress it like Java does with JARs and Microsoft does it with their Office Dokuments for example.

MarcoFalke added the label Brainstorming on Apr 29, 2019

MarcoFalke added the label Block storage on Apr 29, 2019

MarcoFalke commented at 7:11 pm on April 29, 2019: member

Unless I am mistaken, that would increase the CPU load on nodes when they are asked for blocks other than the tip (which is kept in memory, I believe).

Also, I am not sure about the code complexity, since handling archives is easy to get wrong (e.g. CVE-2019-9195)

kristapsk commented at 7:22 pm on April 29, 2019: contributor

I don’t see increased CPU load on nodes as a problem if it’s optional feature. For some users CPU load could be the biggest issue, for others disk space. Also, probably sending compressed blocks P2P could be improvement for some full node operators, as for many people bandwidth is the biggest issue currently.

tryphe commented at 7:48 pm on May 5, 2019: contributor

LevelDB compaction has support for Snappy compression by default, which would probably be much easier to support than adding something else in. But there’s a catch. LevelDB compaction only works in key-value mode, and the block-store uses a raw 4K-block (disk blocks) mode with no keys, which is a contiguous mode. So, you’d need to convert the block store to key-values to use Snappy compression, which would add a new overhead of a key index, and would change performance in other ways.

Something that could technically be done would be to add a “change block-store type” function that rebuilds the block storage as a key-value OR contiguous data. Then the option of enabling or disabling compression would be possible. But it would also add some code complexity that I’m not sure is desirable.

sipa commented at 7:51 pm on May 5, 2019: member

@tryphe The blockchain isn’t stored in LevelDB at all, and there is no point in doing so.

tryphe commented at 7:53 pm on May 5, 2019: contributor

@sipa, thanks!

~~I’m surprised you can actually compress any of it since it’s bitmasked anyways.~~

I guess that’s only the chainstate data with the bitmask. Sorry for the confusion. I guess I figured someone would’ve put data flagged by AV heuristics into the blocks somehow by now.

tryphe commented at 8:55 pm on May 5, 2019: contributor

@ranseier86 I wonder if the better solution would be to just use something like ZFS or btrfs. You’d get compression at disk level without changing anything.

russoj88 commented at 5:19 am on May 7, 2019: none

@tryphe I’m using ZFS on a FreeNAS system for two BTC nodes in jails. One is lz4 compression. The other is gzip-9.

Here’s some numbers for you:

0root@btc:~ # du -A -h -d 0 .bitcoin/blocks .bitcoin/chainstate
1231G	.bitcoin/blocks
23.0G	.bitcoin/chainstate
3root@btc:~ # du -h -d 0 .bitcoin/blocks .bitcoin/chainstate
4208G	.bitcoin/blocks
53.0G	.bitcoin/chainstate

0root@btc-gzip9:~ # du -A -h -d 0 .bitcoin/blocks .bitcoin/chainstate
1231G	.bitcoin/blocks
23.0G	.bitcoin/chainstate
3root@btc-gzip9:~ # du -h -d 0 .bitcoin/blocks .bitcoin/chainstate
4190G	.bitcoin/blocks
52.8G	.bitcoin/chainstate

I have space to test other compression algorithms if that’d be helpful.

MarcoFalke commented at 4:03 pm on April 26, 2020: member

In any case this is a p2p network change, so needs discussion on the mailing list, not here.

MarcoFalke closed this on Apr 26, 2020

MarcoFalke commented at 4:05 pm on April 26, 2020: member

To be clear:

Storing compressed blocks locally is not a p2p change. Though, this feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues. Pull requests with improvements are always welcome.
Serving compressed block is a network-wide p2p change and needs to be discussed on the mailing list, not here.

MarcoFalke locked this on Feb 15, 2022

Feature Request - ZIP the blockchain #15923