Allow accepting non-standard transactions on mainnet

benthecarman commented at 11:12 pm on May 4, 2023: contributor

There are many projects today that have part of their workflow “email a miner this transaction” (inscriptions, https://github.com/supertestnet/breaker-of-jpegs, etc). This is a problem and will eventually cause a mempool divergence between nodes and miners. To help mitigate this we should have an option for users to accept non standard transactions on mainnet. There is already a setting for test networks, this PR just allows it to be set for mainnet.

DrahtBot commented at 11:12 pm on May 4, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept NACK	sdaftuar, glozow
Concept ACK	TalasDigital

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#27576 (kernel: Remove args, chainparams, chainparamsbase from kernel library by TheCharlatan)
#27491 (refactor: Move chain constants to the util library by TheCharlatan)
#27302 (init: Error if ignored bitcoin.conf file is found by ryanofsky)
#25977 (refactor: Replace std::optional<bilingual_str> with util::Result by ryanofsky)
#20892 (tests: Run both descriptor and legacy tests within a single test invocation by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Allow accepting non-standard transactions on mainnet bcdae9002c

benthecarman force-pushed on May 5, 2023

DrahtBot added the label CI failed on May 5, 2023

sdaftuar commented at 0:56 am on May 5, 2023: member

Concept NACK.

If there’s a standardness rule that is interfering with someone’s use case, we should discuss it and figure out whether the rule can be relaxed to accommodate the use case, or whether the use case can be adapted to comply with the standardness rules. Generally speaking, our policy rules are designed to prevent various kinds of denial-of-service attacks, or to help make sure the network runs smoothly – so if users are trying to bypass them, there’s a good chance that there’s insufficient understanding of why rules are in place (or possibly the rules are more restrictive than they need to be).

Moreover, while running with different policies is not something that we can prevent on the network, we should be mindful that if the prevailing network policy is different than what a use case needs, that the use case will likely not be met anyway without some sort of preferential peering with other nodes running the same policy. So this approach strikes me as insufficient to even achieve the goal it is aiming for.

DrahtBot removed the label CI failed on May 5, 2023

glozow added the label TX fees and policy on May 5, 2023

maflcko commented at 6:30 am on May 5, 2023: member

Presumed related issue: #27490

TalasDigital commented at 6:32 am on May 5, 2023: none

Concept ACK.

I agree that this is seriously worth considering in light of risks related to mempool divergence and mining centralization if non-standard transaction usage grows in popularity.

benthecarman commented at 7:20 am on May 5, 2023: contributor

If there’s a standardness rule that is interfering with someone’s use case, we should discuss it and figure out whether the rule can be relaxed to accommodate the use case, or whether the use case can be adapted to comply with the standardness rules. Generally speaking, our policy rules are designed to prevent various kinds of denial-of-service attacks, or to help make sure the network runs smoothly – so if users are trying to bypass them, there’s a good chance that there’s insufficient understanding of why rules are in place (or possibly the rules are more restrictive than they need to be).

I think it is kinda ridiculous to have to come and try and lobby bitcoin core to relay their transaction. This is just a waste of everyone’s time on both sides and accomplishes nothing. If people want to accept non standard transactions, we should let them.

Bitcoin core moves extremely slow (for good reason) and application devs move much faster (also for good reason), there will always be a contention here and giving application devs tools to not have to interfere with bitcoin core’s process should be the goal. If someone builds something but then needs to convince bitcoin core to allow it to be relayed, then create a patch and get it code reviewed, then finally merged and wait for a release, it will be months to years before their transaction can be relayed. In that amount of time they will have just built a relationship with mining pools and circumvented the mempool completely, and everyone will be worse off for it. And this is happening today! Allowing a flag like this provides a solution so nodes and users can opt into supporting different use cases that haven’t yet made it through the process of becoming a standard transaction.

Moreover, while running with different policies is not something that we can prevent on the network, we should be mindful that if the prevailing network policy is different than what a use case needs, that the use case will likely not be met anyway without some sort of preferential peering with other nodes running the same policy. So this approach strikes me as insufficient to even achieve the goal it is aiming for.

Yes, this isn’t the 100% perfect solution because there is no perfect solution, but this is an extremely simple change that is a step in the right direction.

sdaftuar commented at 12:08 pm on May 5, 2023: member

If people want to accept non standard transactions, we should let them.

The problem is that miners need DoS protection too. So I don’t understand what you think we accomplish by giving everyone a knob to turn off DoS protections – under what circumstances is that safe to use? If the argument is that just some miners are getting access to high feerate transactions, and we need the network to relay those transactions more broadly so that mining is competitive, then we need a way to do so that is DoS resistant – not require miners to turn off DoS protections in order to be competitive.

benthecarman commented at 12:29 pm on May 5, 2023: contributor

The problem is that standardness is an overloaded term, it contains some DoS protection while also just disallowing other things. As well, sometimes DoS protection also ends up being the cure that’s worse than the disease creating things like transaction pinning.

If it is that dangerous, proper warnings can be added, however, DoS attacks are already happening on bitcoin that are within standardness rules (stamps).

glozow commented at 12:43 pm on May 5, 2023: member

Concept NACK. These checks exist for very good reasons; being able to bypass them on mainnet is a huge footgun. If there is a specific standardness rule that you can show (1) provides little utility to the node or network and (2) is getting in the way of a use case, we can discuss changing that rule.

There are many projects today that have part of their workflow “email a miner this transaction” (inscriptions, https://github.com/supertestnet/breaker-of-jpegs, etc). This is a problem and will eventually cause a mempool divergence between nodes and miners.

Allowing nonstandard transactions will not help with this case; you still need to find a miner accepting nonstandard transactions to peer with. You are still “emailing a miner this transaction,” just in a more roundabout way.

Let’s say, for example, an application is creating transactions with lots of gigantic, nonstandard outputs that everyone needs to store in their UTXO set. If that application caused the UTXO set to 10x in size, making it less feasible to run a lightweight full node, we would then open an issue to Bitcoin Core asking “why the hell did you help the network relay those transactions?” It isn’t that the “process is too slow” to change the standardness rule. It would just be a terrible idea.

Yes, this isn’t the 100% perfect solution because there is no perfect solution, but this is an extremely simple change that is a step in the right direction.

This “simple change” throws the baby out with the bathwater. The perfect solution is to write applications that don’t DoS the network.

The problem is that standardness is an overloaded term, it contains some DoS protection while also just disallowing other things. As well, sometimes DoS protection also ends up being the cure that’s worse than the disease creating things like transaction pinning.

Again can you please point to a specific example that you want to change? Otherwise this PR is just saying “standardness in general is unnecessary,” and that is incorrect.

jamesob commented at 2:49 pm on May 5, 2023: member

It seems inevitable to me that well-coordinated “third party” mempools will emerge which eschew all policy and use out-of-band payment as the only means of DoS resistance. Seems likely these sources will be paid attention by large mining pools. I’m sympathetic to the motivations behind this change on those grounds, as well as with the general frustration around the sharp edges of policy (e.g. pinning).

The notion of having to lobby a council, however qualified, to “accept a usecase” is also understandably unappealing at face value. I think the contributors on this project who work on mempool/p2p would argue that the existing standardness/policy helps the project serve its widely understood goal of being a value/payment system rather than a carrier for arbitrary data payloads - but of course the approach towards this goal can become controversial.

It is maybe worth clarifying that this change wouldn’t really affect pinning either way, since replacement/ancestry-limiting policy isn’t governed by the -acceptnonstdtxn flag.

I’m curious to see how the wider scope of this situation unfolds beyond this issue, because there are many good and conflicting lines of thinking. Being choosy about the nature and topology of transactions that this software broadcasts keeps many (most?) users with a functioning node, but it also drives other users to build consensus-compatible alternatives. The mix there can be hard to observe.

Coexistence of heterogeneous mempools is maybe the steady state.

instagibbs commented at 3:30 pm on May 5, 2023: member

The notion of having to lobby a council, however qualified, to “accept a usecase” is also understandably unappealing at face value.

This is why *rdinals haven’t been seriously curtailed(or even considered in this repo); taproot was designed with specific goals in mind to allow people to do whatever they want, as long as they aren’t causing systemic issues with relay/validation/miners.

On the other hand, legacy script is full of DoS disasters(that we can’t simply softfork out because literal theft), and we shouldn’t just open the gates further for people to trash the network.

Let’s keep discussing relaxations of that type that make sense, rather than re-opening up gaping DoS holes.

benthecarman commented at 4:13 pm on May 5, 2023: contributor

Okay I’ll admit defeat. I was unware of all the DoS vectors in pre-taproot non-std txs. I still believe there should be a way for users to opt into allowing any kinds of txs into their mempool that doesn’t have to do with DoS (0 value outputs, number of op_returns, taproot annex, tx size, etc).

benthecarman closed this on May 5, 2023

benthecarman deleted the branch on May 5, 2023

glozow commented at 5:49 pm on May 5, 2023: member

I agree that the network functions best when everybody hears about transactions before they are mined, but we should never assume that mempool policies are homogeneous. I also believe that policy has room for improvement and we should change rules if they are harmful and unhelpful. Miners accepting out-of-band nonstandard transactions can be a symptom of Bitcoin Core’s rules not being accommodating enough. (fwiw it can also mean that people are doing kind of spammy things).

However, I haven’t seen any proposals to change specific standardness rules! Why? I hope it doesn’t stem from this myth that Bitcoin Core cannot move fast enough to fix things that are truly broken; it is untrue. I hope that you and other application devs feel comfortable proposing and discussing changes, even if they are not always accepted.

udiWertheimer commented at 8:18 pm on May 5, 2023: none

taproot was designed with specific goals in mind to allow people to do whatever they want, as long as they aren’t causing systemic issues with relay/validation/miners.

On the other hand, legacy script is full of DoS disasters(that we can’t simply softfork out because literal theft), and we shouldn’t just open the gates further for people to trash the network.

Admittedly I wasn’t aware of that distinction between taproot and pre-taproot scripts wrt DoS issues. That’s good news! If that’s the case, would it be reasonable to consider removing the 400kb standardness rule (only for taproot transactions)?

I know for a fact that people are now pushing >400kb txs fairly routinely via out-of-band methods, and I think it would’ve been healthier to the network if they didn’t have to. I’m not sure what’s the best place to propose this but happy to look into it if there isn’t an obvious reason why this shouldn’t be done.

instagibbs commented at 8:36 pm on May 5, 2023: member

@udiWertheimer Assuming we could switch behavior ahead of time based on all inputs being taproot(it’s a little more complicated in practice because other inputs might be ok too), I think the main drawback is optimization of the bin packing problem with larger items. Smaller items are easier to pick closer to optimally. This would fall under “keeping miners simple for decentralization” bucket instead of DoS.

benthecarman commented at 9:25 pm on May 5, 2023: contributor

This would fall under “keeping miners simple for decentralization” bucket instead of DoS.

You could argue the opposite is happening though, the miners that are accommodating these transactions are making a killing right now because they have exclusive access to these large transactions.

udiWertheimer commented at 9:36 pm on May 5, 2023: none

This would fall under “keeping miners simple for decentralization” bucket instead of DoS.

You could argue the opposite is happening though, the miners that are accommodating these transactions are making a killing right now because they have exclusive access to these large transactions.

Yes and, referring back to @sdaftuar’s point, one reason to include a flag to disable standardness rules would be to make life easier for such mining pools. Otherwise they have to modify and rebuild bitcoin core, and since we know they’re doing this, perhaps it’s better to provide a flag so they can do so safely.

1ma commented at 12:54 pm on May 6, 2023: none

Let it be known for the record that those of us who’ve been playing with https://github.com/supertestnet/breaker-of-jpegs are very much in favor of enacting a new standardness rule, not removing them. @benthecarman knows this, and citing that project as an example denotes bad faith.

If anything node runners need more control for filtering whatever they may want, such as this new kind of transactions who are abusing the “Lady Gaga Video Attack Vector” by exploiting a particular sequence of meaningless Taproot script that allow appending streams of arbitrary bytes on them.

FvKJnMtWcAkpsK_

“Mempool divergence” is a non-issue, just a testament of the decentralized nature of the Bitcoin network. Any concerted efforts at homogenizing them across all nodes can only be understood as unwarranted overreach by bad actors.

BullishNode commented at 1:17 pm on May 6, 2023: none

Strong NACK. Standardness rules protect the network from ddos and spam. The fact that you have to email a miner is a good thing: it’s rate limiting.

TheBlueMatt commented at 5:52 pm on May 6, 2023: contributor

“Mempool divergence” is a non-issue, just a testament of the decentralized nature of the Bitcoin network. Any concerted efforts at homogenizing them across all nodes can only be understood as unwarranted overreach by bad actors.

This is inaccurate - there are a whole host of issues across the stack with large mempool divergence. From compact block relay losing their efficiency, to feerate estimates getting thrown off, to transaction validation caches becoming less efficient leading to block validation time increases, etc, etc.

Some mempool divergence is unavoidable - we have to draw lines in the sand somewhere to avoid DoS issues, and each of those lines is an avenue for mempool divergence to occur, but that doesn’t mean that mempool divergence is good, rather the opposite - those lines have to be drawn to limit mempool divergence where at all possible.

1ma commented at 8:18 pm on May 6, 2023: none

I did not claim mempool diversity is “good”, just a “non-issue”, i.e. simply part of how Bitcoin works.

Trying to mitigate these quirks by imposing a uniform mempool on all node runners would rob them of an important part of their sovereignty, therefore it’d be much worse than the quirks.

joostjager commented at 4:41 pm on May 25, 2023: none

I understand and appreciate the concerns about DoS protection and the importance of maintaining standardness rules for network-wide transaction relay. But isn’t there room for flexibility when it comes to how Bitcoin nodes handle non-standard transactions submitted locally via the RPC methods sendrawtransaction and submitpackage?

These RPC calls are made by the node operator themselves or applications they trust, thus the risk of DoS attacks is significantly mitigated. The operator can assume responsibility for understanding the implications and potential risks of submitting non-standard transactions.

1ma commented at 9:03 am on May 26, 2023: none

There are no risks in submitting non-standard transactions to your own node’s mempool. Its direct peers will simply discard these when your node broadcasts them, and they will eventually be purged from your mempool as any other non-confirmed transaction.

If anything, node runners want to be able to apply more filters/standardness rules (increased sovereignty), not disable them (decreased sovereignty).

joostjager commented at 12:25 pm on May 26, 2023: none

Its direct peers will simply discard these when your node broadcasts them, and they will eventually be purged from your mempool as any other non-confirmed transaction.

Unless you have peers that run a patch to accept non-standard transactions?

Also if you are mining yourself and using core’s block template, you need those txes to enter your mempool first?

1ma commented at 2:23 pm on May 26, 2023: none

Even then they won’t go further than that, unless these peers are miners. This option would be only somewhat relevant if you run a solo mining node or a pool coordinator.

joostjager commented at 2:29 pm on May 26, 2023: none

Agreed that the option isn’t the most relevant in typical scenarios. But if a restriction can be lifted safely and benefit a specific group of users, why not?

One use case for a regular user is to gain the ability to insert out of band transactions into their mempool for proper fee estimation. Miners also mine transactions received out-of-band. This would reduce mempool divergence across the network.

Created https://github.com/bitcoin/bitcoin/issues/27768

bitcoin locked this on May 25, 2024

Allow accepting non-standard transactions on mainnet #27578

Reviews

Conflicts