Feature Request - ZIP the blockchain #15923

issue ranseier86 openend this issue on April 29, 2019
  1. ranseier86 commented at 7:01 pm on April 29, 2019: none
    Instead of prune there could be a ZIP mode. Zipping blk00010.dat with default Windows 10 build in ZIP shrinks it from 130 MB to 97 MB. Zipping blk00010.dat with 7zip default settings shrinks it from 130 MB to 77 MB. I love to have the whole blockchain stored but there is i huge potential to compress it like Java does with JARs and Microsoft does it with their Office Dokuments for example.
  2. MarcoFalke added the label Brainstorming on Apr 29, 2019
  3. MarcoFalke added the label Block storage on Apr 29, 2019
  4. MarcoFalke commented at 7:11 pm on April 29, 2019: member

    Unless I am mistaken, that would increase the CPU load on nodes when they are asked for blocks other than the tip (which is kept in memory, I believe).

    Also, I am not sure about the code complexity, since handling archives is easy to get wrong (e.g. CVE-2019-9195)

  5. kristapsk commented at 7:22 pm on April 29, 2019: contributor
    I don’t see increased CPU load on nodes as a problem if it’s optional feature. For some users CPU load could be the biggest issue, for others disk space. Also, probably sending compressed blocks P2P could be improvement for some full node operators, as for many people bandwidth is the biggest issue currently.
  6. tryphe commented at 7:48 pm on May 5, 2019: contributor

    LevelDB compaction has support for Snappy compression by default, which would probably be much easier to support than adding something else in. But there’s a catch. LevelDB compaction only works in key-value mode, and the block-store uses a raw 4K-block (disk blocks) mode with no keys, which is a contiguous mode. So, you’d need to convert the block store to key-values to use Snappy compression, which would add a new overhead of a key index, and would change performance in other ways.

    Something that could technically be done would be to add a “change block-store type” function that rebuilds the block storage as a key-value OR contiguous data. Then the option of enabling or disabling compression would be possible. But it would also add some code complexity that I’m not sure is desirable.

  7. sipa commented at 7:51 pm on May 5, 2019: member
    @tryphe The blockchain isn’t stored in LevelDB at all, and there is no point in doing so.
  8. tryphe commented at 7:53 pm on May 5, 2019: contributor

    @sipa, thanks!

    I’m surprised you can actually compress any of it since it’s bitmasked anyways.

    I guess that’s only the chainstate data with the bitmask. Sorry for the confusion. I guess I figured someone would’ve put data flagged by AV heuristics into the blocks somehow by now.

  9. tryphe commented at 8:55 pm on May 5, 2019: contributor
    @ranseier86 I wonder if the better solution would be to just use something like ZFS or btrfs. You’d get compression at disk level without changing anything.
  10. russoj88 commented at 5:19 am on May 7, 2019: none

    @tryphe I’m using ZFS on a FreeNAS system for two BTC nodes in jails. One is lz4 compression. The other is gzip-9.

    Here’s some numbers for you:

    0root@btc:~ # du -A -h -d 0 .bitcoin/blocks .bitcoin/chainstate
    1231G	.bitcoin/blocks
    23.0G	.bitcoin/chainstate
    3root@btc:~ # du -h -d 0 .bitcoin/blocks .bitcoin/chainstate
    4208G	.bitcoin/blocks
    53.0G	.bitcoin/chainstate
    
    0root@btc-gzip9:~ # du -A -h -d 0 .bitcoin/blocks .bitcoin/chainstate
    1231G	.bitcoin/blocks
    23.0G	.bitcoin/chainstate
    3root@btc-gzip9:~ # du -h -d 0 .bitcoin/blocks .bitcoin/chainstate
    4190G	.bitcoin/blocks
    52.8G	.bitcoin/chainstate
    

    I have space to test other compression algorithms if that’d be helpful.

  11. MarcoFalke commented at 4:03 pm on April 26, 2020: member
    In any case this is a p2p network change, so needs discussion on the mailing list, not here.
  12. MarcoFalke closed this on Apr 26, 2020

  13. MarcoFalke commented at 4:05 pm on April 26, 2020: member

    To be clear:

    • Storing compressed blocks locally is not a p2p change. Though, this feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues. Pull requests with improvements are always welcome.

    • Serving compressed block is a network-wide p2p change and needs to be discussed on the mailing list, not here.

  14. MarcoFalke locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-31 03:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me