BIP 337: Compressed Transactions #1556

pull TomBriar wants to merge 3 commits into bitcoin:master from TomBriar:bip-tombriar-compressed-transactions changing 2 files +308 −0
  1. TomBriar commented at 5:53 pm on March 5, 2024: contributor

    This document proposes a serialization scheme for compressing bitcoin transactions. The compressed bitcoin transactions can reach a serialized size of less than 50% of the original serialized transaction. One of the methods for compressing, involves reducing the transaction outpoints in a potentially lossy way, therefore it is an optional path for compression. Without compressing the outpoints, compressed transactions still reach less then 70% of the original size.

    Reference Implementation: https://github.com/bitcoin/bitcoin/pull/29134

    The reference implementation includes two new RPC endpoints to demonstrate how to use compressed transactions. The compressrawtransaction takes in a raw hex-encoded transaction and returns a compressed hex-encoded transaction, while the decompressrawtransaction accepts a compressed hex-encoded transaction and returns the original raw hex-encoded transaction.

    Thanks- Tom.

  2. apoelstra commented at 2:50 pm on March 19, 2024: contributor

    Hi Tom,

    Thanks for posting this. I’d suggest rewording the abstract and PR description to avoid talking in terms of Core internals (classes, subclasses, etc) and instead simply describe the serialization. It’s fine to describe suggested RPC endpoints since these give an idea of how the specification might be used.

  3. in bip-tombriar-compressed-transactions.mediawiki:44 in 517ba8414c outdated 
    39+|-
40+! Name !! Width !! Description
41+|-
42+| Transaction metadata || 1 bytes || Information on the structure of the transaction. See [[#transaction-metadata|Transaction Metadata]]
43+|-
44+| Version || 0-5 Bytes || An optional CompactSize containing the transactions version.
    apoelstra commented at 2:51 pm on March 19, 2024:

    grammar: transactions should be transaction or transaction's here and in the 3 lines below.

    I’d also change “optional” to “if present according to the metadata field, a CompactSize encoding the transaction version”. So it’s clear on a first read what “optional” means.

    TomBriar commented at 7:49 pm on April 1, 2024:
    Fixed, Thanks!
  4. in bip-tombriar-compressed-transactions.mediawiki:42 in 517ba8414c outdated 
    37+
38+{| class="wikitable" style="margin:auto"
39+|-
40+! Name !! Width !! Description
41+|-
42+| Transaction metadata || 1 bytes || Information on the structure of the transaction. See [[#transaction-metadata|Transaction Metadata]]
    apoelstra commented at 2:53 pm on March 19, 2024:
    grammar: you inconsistently capitalize Bits/bits and Bytes/bytes, and inconsistently use plural bytes/byte and bits/bit
    TomBriar commented at 7:49 pm on April 1, 2024:
    Fixed, Thanks!
  5. in bip-tombriar-compressed-transactions.mediawiki:84 in 517ba8414c outdated 
    79+
80+{| class="wikitable" style="margin:auto"
81+|-
82+! Name !! Width !! Description
83+|-
84+| Compressed Signature || 1 Bit || Signature compression flag. For P2TR: 1 for keyspend, 0 for scriptspend; For P2SH: 0 for p2sh, 1 for p2sh-wpkh.
    apoelstra commented at 2:54 pm on March 19, 2024:
    Maybe mention that for optut types other than P2TR and P2SH this bit is unused.
    TomBriar commented at 6:07 pm on March 26, 2024:
    It’s used for all script types but has special implications for P2TR and P2SH-WPKH. I reworded for clarity.
  6. in bip-tombriar-compressed-transactions.mediawiki:88 in 517ba8414c outdated 
    83+|-
84+| Compressed Signature || 1 Bit || Signature compression flag. For P2TR: 1 for keyspend, 0 for scriptspend; For P2SH: 0 for p2sh, 1 for p2sh-wpkh.
85+|-
86+| Standard Hash || 1 Bit || A flag to determine if this Input's Signature Hash Type is standard (0x00 for Taproot, 0x01 for Legacy/Segwit).
87+|-
88+| Standard Squence || 2 Bits || A CompactSize flag for the inputs sequence. Encode literal values as follows: 1 = 0x00000000, 2 = 0xFFFFFFFE, 3 = 0xFFFFFFFF.
    apoelstra commented at 2:54 pm on March 19, 2024:
    inputs should be input's or input. Also you inconsistently capitalize Input in this section.
    TomBriar commented at 7:49 pm on April 1, 2024:
    Fixed, Thanks!
  7. in bip-tombriar-compressed-transactions.mediawiki:136 in 517ba8414c outdated 
    131+|-
132+| Sequence || 0-5 Bytes || An Optional VarInt containing the sequence if it was non-standard.
133+|-
134+| Txid Blockheight || 1-5 Bytes || A VarInt Either containing 0 if this an uncompressed input, or it contains the offset from Minimum Blockheight for this Txid.
135+|-
136+| Txid/Signature Data || 65+ Bytes || Txid/Signatures are determined to be uncompressed either by the output script of the previous transaction, or if the Txid Blockheight is zero. For each Compressed Txid/Signature see [[#compressed-data|here]]. For each Uncompressed Txid/Signature see [[#uncompressed-data|here]].
    apoelstra commented at 2:57 pm on March 19, 2024:
    When linking you should use the title of the section rather than here. This helps screen readers and other accessibility tools, and also helps sight-readers know where the link is going to go.
    TomBriar commented at 7:49 pm on April 1, 2024:
    Fixed, Thanks!
  8. in bip-tombriar-compressed-transactions.mediawiki:183 in 517ba8414c outdated 
    178+
179+==== Ideal Transaction ====
180+
181+The target transaction for the most optimal compression was chosen
182+based off the most common transactions that are likely to be used
183+for purposes that requires the best compression.
    apoelstra commented at 2:59 pm on March 19, 2024:

    I’d reword this because it’s hard to tell whether you’re about to describe an “ideal” transaction or a “common” one or what. I suggest

    0The compression scheme was designed to be optimal for a "typical" transaction
1spending a few close-in-age inputs and having one or two outputs. Here are size
2values for such a transaction which demonstrate the effectiveness of the compression.
    TomBriar commented at 7:49 pm on April 1, 2024:
    Fixed, Thanks!
  9. in bip-tombriar-compressed-transactions.mediawiki:225 in 517ba8414c outdated 
    220+
221+2. Replacing 32-bit numeric values with either variable-length integers (VarInts) or compact-integers (CompactSizes).
222+
223+3. Using compressed signatures and public key recovery upon decompression.
224+
225+4. Replacing the 36-byte txid/vout pair with a block height and index.
    apoelstra commented at 3:00 pm on March 19, 2024:
    Both the “Motivation” and “Rationale” sections should go up top, after the abstract :). Nobody is going to read this far without a motivation.
    TomBriar commented at 7:50 pm on April 1, 2024:
    Fixed, Thanks!
  10. in bip-tombriar-compressed-transactions.mediawiki:213 in 517ba8414c outdated 
    208+|-
209+| Output Amount || No Restrictions || (-1) - 7 Bytes For Each Output
210+|}
211+
212+=== Motivation ===
213+Bitcoin transactions currently contain a large amount of white space and padding. Specific fields are often one of a minimal number of possibilities. Leveraging these facts, we can create an encoding for 90% of Bitcoin transactions that are roughly 25-50% smaller.
    apoelstra commented at 3:01 pm on March 19, 2024:
    I’d say “typical Bitcoin transactions” rather than just “Bitcoin transactions”. Technically, no part of a transaction is “whitespace” or meaningless. It’s just that usually the version, locktime, sequence numbers, etc., are one of a few values.
    TomBriar commented at 7:50 pm on April 1, 2024:
    Fixed, Thanks!
  11. in bip-tombriar-compressed-transactions.mediawiki:268 in 517ba8414c outdated 
    263+
264+===== Compressed =====
265+<code>16b1ec7c2db1ec7d981431a20f5d9212023b510599c9d53d082f8e07faaa2d51482e078f8e398cb50d77635abd99220ad713a081c4f20b83cb3f491ed8bd032cb151a3521ed144164d9c8efefefe7d2da377ed4978fefa043a58489912f8e28e162262a608</code>
266+
267+== Acknowledgements ==
268+Thank you to Andrew Poelstra who helped invent and develop the ideas in the proposal and the code in the reference implementation!
    apoelstra commented at 3:02 pm on March 19, 2024:
    I appreciate the enthusiasm but I think this’d read better with a period instead of the exclamation point.
    TomBriar commented at 7:50 pm on April 1, 2024:
    Fixed, Thanks!
  12. TomBriar force-pushed on Apr 1, 2024
  13. TomBriar force-pushed on Apr 1, 2024
  14. TomBriar force-pushed on Apr 1, 2024
  15. TomBriar force-pushed on Apr 15, 2024
  16. murchandamus added the label New BIP on Apr 23, 2024
  17. in bip-tombriar-compressed-transactions.mediawiki:8 in b6be5615a6 outdated 
    0@@ -0,0 +1,293 @@
1+<pre>
2+  BIP: ?
3+  Layer: API/RPC
4+  Title: Compressed Transactions RPC
5+  Author: Tom Briar <tombriar11@protonmail.com>
6+  Comments-URI: https://github.com/bitcoin/bitcoin/pull/29134
7+                https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021924.html
    murchandamus commented at 1:19 pm on April 29, 2024:

    I think this is mixing up the “Comments-URI” and the “Post-History” headers.

    0  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-tombriar-compressed-transactions
    TomBriar commented at 4:34 pm on May 14, 2024:
    Fixed, Thanks!
  18. in bip-tombriar-compressed-transactions.mediawiki:10 in b6be5615a6 outdated 
     6+  Comments-URI: https://github.com/bitcoin/bitcoin/pull/29134
 7+                https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021924.html
 8+  Status: Draft
 9+  Type: Standards Track
10+  Created: 2024-02-01
11+  License: BSD-3-Clause
    murchandamus commented at 1:20 pm on April 29, 2024: 
    0  License: BSD-3-Clause
1  Post-History: https://github.com/bitcoin/bitcoin/pull/29134
2                https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021924.html
    TomBriar commented at 4:34 pm on May 14, 2024:
    Fixed, Thanks!
  19. in bip-tombriar-compressed-transactions.mediawiki:22 in b6be5615a6 outdated 
    17+This document proposes a serialization scheme for compressing Bitcoin transactions. The compressed Bitcoin transactions can reach a serialized size of less than 50% of the original serialized transaction. One method for compressing involves reducing the transaction outpoints in a potentially lossy way. Therefore, it is an optional path for compression. Compressing the outpoints is necessary for compressed transactions to reach less than 70% of the original size.
18+
19+=== Motivation ===
20+Typical Bitcoin transactions usually contain a large amount of white space and padding due to specific fields that are often one of a minimal number of possibilities. We can use this fact and a few similar methods to create an encoding for 90% of Bitcoin transactions that are roughly 25-50% smaller.
21+
22+=== Rational ===
    murchandamus commented at 1:20 pm on April 29, 2024: 
    0=== Rationale ===
    TomBriar commented at 4:34 pm on May 14, 2024:
    Fixed, Thanks!
  20. murchandamus changes_requested
  21. murchandamus commented at 1:23 pm on April 29, 2024: contributor
    Light first review. Looks pretty good, could you please add the mandatory “Backwards Compatibility” section (even if it just says that there are no concerns) and take a look at the preamble?
  22. murchandamus added the label PR Author action required on May 14, 2024
  23. TomBriar force-pushed on May 14, 2024
  24. TomBriar commented at 4:34 pm on May 14, 2024: contributor
    Thank you @murchandamus, all changes were accepted!
  25. murchandamus removed the label PR Author action required on May 14, 2024
  26. murchandamus commented at 6:40 pm on May 14, 2024: contributor
    Thanks, I aim to give this a proper review soon
  27. in bip-tombriar-compressed-transactions.mediawiki:8 in 95fbf60b4a outdated 
    0@@ -0,0 +1,301 @@
1+<pre>
2+  BIP: ?
3+  Layer: API/RPC
4+  Title: Compressed Transactions RPC
5+  Author: Tom Briar <tombriar11@protonmail.com>
6+  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-tombriar-compressed-transactions
7+  Comments-URI: https://github.com/bitcoin/bitcoin/pull/29134
8+                https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021924.html
    murchandamus commented at 2:43 pm on May 20, 2024:

    This was added below, but not removed here.

    TomBriar commented at 4:33 pm on May 24, 2024:
    Fixed, Thanks!
  28. in bip-tombriar-compressed-transactions.mediawiki:25 in 95fbf60b4a outdated 
    20+This document proposes a serialization scheme for compressing Bitcoin transactions. The compressed Bitcoin transactions can reach a serialized size of less than 50% of the original serialized transaction. One method for compressing involves reducing the transaction outpoints in a potentially lossy way. Therefore, it is an optional path for compression. Compressing the outpoints is necessary for compressed transactions to reach less than 70% of the original size.
21+
22+=== Motivation ===
23+Typical Bitcoin transactions usually contain a large amount of white space and padding due to specific fields that are often one of a minimal number of possibilities. We can use this fact and a few similar methods to create an encoding for 90% of Bitcoin transactions that are roughly 25-50% smaller.
24+
25+=== Rationale ===
    murchandamus commented at 2:59 pm on May 20, 2024:
    Has there been any related work or alternate designs worth mentioning here?
    TomBriar commented at 5:34 pm on May 28, 2024:
    https://github.com/TomBriar/compressed_transactions is a full implementation in rust
  29. murchandamus commented at 3:08 pm on May 20, 2024: contributor
    Hi @Tombriar, I went over the complete document, the Bitcoin Core pull request, and the mailing list. Your document almost fulfills the formatting requirements, I left one nit and one open question that you may want to consider. The description is clear, concise, and appears to be comprehensive. Since the Bitcoin Core pull request was closed, I was wondering whether this proposal is actively deployed by other software (although that is not necessary for this to be merged).
  30. in bip-tombriar-compressed-transactions.mediawiki:4 in 95fbf60b4a outdated 
    0@@ -0,0 +1,301 @@
1+<pre>
2+  BIP: ?
3+  Layer: API/RPC
4+  Title: Compressed Transactions RPC
    murchandamus commented at 4:28 pm on May 20, 2024:

    This document seems to be focused on describing the scheme for compressing transactions and only mentions the RPC in context of the reference implementation. Perhaps it would make sense to drop the “RPC” from the title.

    0  Title: Compressed Transactions
    TomBriar commented at 4:34 pm on May 24, 2024:
    Fixed, Thanks!
  31. murchandamus commented at 3:30 pm on May 28, 2024: contributor

    Hey @TomBriar, I’m not sure what happened here, it looks like you started working on this, but maybe you did not push the changes yet?

    If the answer to my two open questions would be that alternative designs are not relevant, and the proposal is currently not implemented in any software that would be satisfactory answers to me—I just thought if those answers were different it would be nice to mention it.

    As soon as you push your changes, I can take it from there. The only things missing would be to assign a number and merge it, which I can take care of.

  32. TomBriar commented at 3:35 pm on May 28, 2024: contributor

    @murchandamus Hi, Yes there is actually a relevant implementation I am finishing the finial touches on it today and I’ll link it all together and have this ready to go by EOD. Thank you for your patience!

    Thanks- Tom.

  33. murchandamus commented at 5:22 pm on May 28, 2024: contributor
    I see, thanks for the update!
  34. TomBriar commented at 5:33 pm on May 28, 2024: contributor

    Hi,

    Just pushed changes that address all of the above reviews, As stated, there is a project that I am aware of that will enable people to use steganography to encode data into images although it is not complete and I was not given an ETA. That said I have built a tool that implements the scheme described here. The hope is that with this BIP, future projects will all use this scheme for compressed Bitcoin transactions, and potentially bitcoin core itself can use parts of it in P2P or Block Storage.

    https://github.com/TomBriar/compressed_transactions

    Thanks- Tom.

  35. murchandamus commented at 6:12 pm on May 28, 2024: contributor
    Hey, you wrote that you pushed, but I don’t see a change. Could you check whether the push went through?
  36. TomBriar force-pushed on May 28, 2024
  37. in bip-tombriar-compressed-transactions.mediawiki:23 in a4089c86f8 outdated 
    18+This document proposes a serialization scheme for compressing Bitcoin transactions. The compressed Bitcoin transactions can reach a serialized size of less than 50% of the original serialized transaction. One method for compressing involves reducing the transaction outpoints in a potentially lossy way. Therefore, it is an optional path for compression. Compressing the outpoints is necessary for compressed transactions to reach less than 70% of the original size.
19+
20+=== Motivation ===
21+Typical Bitcoin transactions usually contain a large amount of white space and padding due to specific fields that are often one of a minimal number of possibilities. We can use this fact and a few similar methods to create an encoding for 90% of Bitcoin transactions that are roughly 25-50% smaller.
22+
23+There exists a working-in-progress app that allows the use of steganography to encode data in images to be passed around via various social media groups. When used in conjunction with this compression scheme and an elligator squared encryption, would allow for a very secure and private form of broadcasting bitcoin transactions.
    murchandamus commented at 6:24 pm on May 28, 2024: 
    0There exists a working-in-progress app that allows the use of steganography to encode data in images to be passed around via various social media groups. When used in conjunction with this compression scheme and an elligator squared encryption, this would allow for a very secure and private form of broadcasting bitcoin transactions.
    TomBriar commented at 6:30 pm on May 28, 2024:
    Fixed, Thanks!
  38. murchandamus approved
  39. murchandamus commented at 6:25 pm on May 28, 2024: contributor
    Spotted a missing word here, but otherwise LGTM.
  40. murchandamus commented at 6:27 pm on May 28, 2024: contributor
    I assign this BIP 337
  41. murchandamus renamed this:
    bip-tombriar-compressed-transactions
    BIP 337: Compressed Transactions
    on May 28, 2024
  42. TomBriar force-pushed on May 28, 2024
  43. TomBriar commented at 6:29 pm on May 28, 2024: contributor

    Hi, Thanks!

    I made that change and fixed the BIP number field.

    -Tom

  44. murchandamus force-pushed on May 28, 2024
  45. bip-tombriar-compressed-transactions 996e31fa16
  46. BIP-0337: Fix Comments-URI 73cfb05b94
  47. BIP-0337: Add table entry, move to numbered file f240c40284
  48. murchandamus force-pushed on May 28, 2024
  49. murchandamus commented at 6:46 pm on May 28, 2024: contributor
    I fixed the Comments-URI header, added an entry in the README.mediawiki table, and moved your BIP to the numbered file
  50. TomBriar commented at 6:47 pm on May 28, 2024: contributor
    Awesome thank you!
  51. murchandamus merged this on May 28, 2024
  52. murchandamus closed this on May 28, 2024
  53. murchandamus commented at 6:49 pm on May 28, 2024: contributor
    Thank you, and congrats! :)
  54. murchandamus removed the label New BIP on May 28, 2024


TomBriar apoelstra murchandamus

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bips. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-04-29 11:10 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me