Compressed Bitcoin Transactions

TomBriar commented at 5:01 pm on December 22, 2023: none

This is a draft PR and reference implementation for a Compressed Bitcoin Transaction, As stated in the doc the main application for this is steganography, satellite/radio broadcast, and other low bandwidth channels with a high CPU availability on decompression.

doc: Added Compressed Transaction Schema Documentation util: Added a variable length bitstream encoder script: Added the rest of the IsPayTo Script functions, And Split ExtractDestination into two functions for ease of use node: Added a vector based coin lookup primitives: Added Compressed Transaction validation: Added OutPoint Compression methods to the chainstate class rpc: Added RPC endpoints for compression and decompression of Transactions test: Added Tests for the RPC and new Primitive fuzz: Added a fuzz test for Compressed Transactions

Link to first post on the bitcoin mailing list: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021924.html

DrahtBot commented at 5:01 pm on December 22, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	1440000bytes

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29280 (Implement OP_CHECKTEMPLATEVERIFY by reardencode)
#29221 (Implement 64 bit arithmetic op codes in the Script interpreter by Christewart)
#29050 (Add OP_TXHASH and OP_CHECKTXHASHVERIFY opcodes by stevenroose)
#28930 (wallet: Add scan_utxo option to getbalances RPC by BrandonOdiwuor)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

TomBriar force-pushed on Dec 22, 2023

DrahtBot added the label CI failed on Dec 22, 2023

TomBriar force-pushed on Dec 22, 2023

TomBriar force-pushed on Dec 23, 2023

1440000bytes commented at 3:56 pm on December 24, 2023: none

Concept ACK

25-50% savings look good and compressed raw transactions can help in some cases as mentioned in the PR description.

Related Q&A: https://bitcoin.stackexchange.com/questions/98886/compress-transaction-hex-string (still wont fit in one text message but will need less text messages)

TomBriar force-pushed on Dec 26, 2023

DrahtBot removed the label CI failed on Dec 26, 2023

achow101 commented at 1:19 am on December 27, 2023: member

Is this intended to be used for storage, p2p relay, or something else? If for our storage, I would be interested in seeing benchmarks as this would have an effect on how quickly things can be pulled from disk to be relayed. If for relay, then I think there should first be a BIP that describes the protocol for using compression and how that is negotiated. And if neither of those, then I don’t think it’s within the scope of Bitcoin Core. It seems to be a pretty significant maintenance burden for something that, as implemented, is only used by specific new RPCs.

TomBriar force-pushed on Jan 2, 2024

DrahtBot added the label CI failed on Jan 2, 2024

DrahtBot removed the label CI failed on Jan 2, 2024

TomBriar force-pushed on Jan 2, 2024

TomBriar force-pushed on Jan 3, 2024

DrahtBot added the label CI failed on Jan 3, 2024

TomBriar force-pushed on Jan 3, 2024

TomBriar force-pushed on Jan 4, 2024

furszy commented at 7:18 pm on January 4, 2024: member

You could run the lint whitespace task locally. It is inside test/lint folder.

TomBriar force-pushed on Jan 4, 2024

TomBriar force-pushed on Jan 5, 2024

DrahtBot removed the label CI failed on Jan 5, 2024

TomBriar force-pushed on Jan 5, 2024

DrahtBot added the label CI failed on Jan 15, 2024

maflcko commented at 6:14 pm on February 22, 2024: member

And if neither of those, then I don’t think it’s within the scope of Bitcoin Core. It seems to be a pretty significant maintenance burden for something that, as implemented, is only used by specific new RPCs.

My understanding is that it requires the chainstate, so this is written inside of Bitcoin Core. However, I am also wondering if existing RPCs can be used to query the chainstate, or if the RPC interface could be extended, if needed. Putting this piece of code directly in the satellite broadcast software seems easier and preferable.

TomBriar force-pushed on Mar 7, 2024

doc: Added Compressed Transaction Schema b4c96a7ce8

util: Added a variable length bitstream encoder f629554a8d

script: Added the rest of the IsPayTo functions 5f020a6411

addresstype: Split ExtractDestination into two functions ebba0f0ee8

node: Added vector based coin lookup 593d58dc68

TomBriar force-pushed on Mar 9, 2024

primitives: Added Compressed Transaction e7be62ead7

validation: Added OutPoint (De)Compression fab3a65847

rpc: Added RPC endpoints for Compressed Transactions 2a50fa570a

TomBriar force-pushed on Mar 9, 2024

test: Added RPC tests for compressrawtransaction and decompressrawtransaction, For valid assets_tests run compression tests, Added unit tests for new primitives 028760ed25

fuzz: Added Fuzz tests for Compressed Transactions a850ab0012

TomBriar force-pushed on Mar 10, 2024

maflcko commented at 9:06 am on March 11, 2024: member

Closing for now due to lack of replies to questions. Please reply below, and leave a comment if this should be reopened.

maflcko closed this on Mar 11, 2024

TomBriar commented at 4:30 pm on May 14, 2024: none

Hi,

This is a reference implementation for the compression scheme described in this WIP BIP: https://github.com/bitcoin/bips/pull/1556

It may be a bit heavy to be PR’d into bitcoin core at this time, although it has a great application for both the P2P and Storage in bitcoin core. Another project that implements the same schema is underway as an external tool and the RPC commands were found to be sufficient.

Thanks- Tom.

Compressed Bitcoin Transactions #29134

Code Coverage

Reviews

Conflicts