Enable tx replacement on testnet. #2516

Tested: wrote a program that replaces a non-final tx and checked it against a bitcoind with this patch, replacement was successful.

False negative...sorry, Ive reset this pull so it should come back with a new test

Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/cded58cadc0202fa36be5ff67e35b45c992b6da0 for binaries and test log.

This could happen for one of several reasons:

1. It chanages paths in makefile.linux-mingw or otherwise changes build scripts in a way that made them incompatible with the automated testing scripts (you can find the patches applied at test-time at http://jenkins.bluematt.me/pull-tester/files/patches/)
2. It adds/modifies tests which test network rules (thanks for doing that), which conflicts with a patch applied at test time
3. It does not build on either Linux i386 or Win32 (via MinGW cross compile)
4. The test suite fails on either Linux i386 or Win32
5. The block test-cases failed (lookup the first bNN identifier which failed in https://github.com/TheBlueMatt/test-scripts/blob/master/FullBlockTestGenerator.java)

If you believe this to be in error, please ping BlueMatt on freenode or TheBlueMatt here.

This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

This pull needs rebased to get the updated contrib/ before it will test

NACK without anti-DoS measures and a discussion on the email list about how to price network bandwidth. Note Gavin's relevant discussion about non-chain messages attached to tx's.

How about: keep a counter in the mempool, remembering the sum of the sizes of all replacements a transaction has had. When deciding whether to accept a transaction as a replacement, increase this number and then check whether its fee is enough for that size. That's certainly overkill, as it will correctly count network and processing overhead, but not blockchain or UTXO burdening (as those only happen after finalizing). In any case, it's easy to implement, seems safe to me, and would at least allow experimenting with this on testnet.

I'm quite aware of the DoS and other issues, that's why this pull is only for the testnet. Obviously we can't prove we made the feature safe without actually being able to demo failed attacks. Re-enabling on the testnet doesn't have any downsides, and has some upsides. The fixes to make it workable on the main network can come later.

@sipa @mikehearn replied on bitcoin-dev mailing list

As one relay anti-DOS mechanism, the minimum increase in fee to replace an existing mempool transaction could be set to minrelayfee.

This would mean a languishing zero fee transaction could only be replaced if it's total fee was increased to at least 0.0001 (or 0.0001 per KB). The replacement transaction could only be again replaced if all its previous ins/outs are unchanged and the total fee is increased by at least another 0.0001 more. The resending client must also ensure that besides this resend increase, the priority-required fee is also added or increased if the transaction gains a KB or drops out of free priority, and the relay node enforces this correctly.

This gives the ability to increase pending transaction fees, but makes continued replacement not get beyond relays without spending more each time (and using up unspents). This also would discourage adding new non-change payment outputs to an existing transaction instead of creating a new transaction.

Also at the relay level, replacements should be bounced if existing ins/outs are altered, so 0-conf transactions are still as trustable as before:

Guys, your enthusiasm is commendable but can we please keep the "redesign Bitcoin" discussions on the mailing list or forum please. This pull is very simple and is not the right place to discuss strategies for re-enabling on the main network - the goal is simply to enable on the testnet so we can experiment and try different things there without needing to run locally patched testnet-in-boxes all the time.

OK, this won't destroy testnet as we know it. I just wonder if a 0.9 testnet could actually be testing something instead of allowing testing of anything.

Would this impact testing of double-spend resistance or formulas, such as people who may want to test their "dice" or commerce 0-confirmations sites with testcoins first?

Sure. I think it's clear that whatever is implemented would be deployed first on testnet before the line that disables it is removed.

@mikehearn We already know we have to test DoS behavior, and fix that behavior, so we already know the code will need to be changed a few times. Under those circumstances I see no reason to put that code into the client everyone is going to be running, potentially disrupting their testing activities for no good reason. (enabling long-disabled code is essentially the same thing as adding new code)

Setting up a dozen nodes for testing isn't a big deal - I'd be happy to run one or two with the tx-replacement patch if you want. @qubez FWIW what you wrote is probably the fourth or fifth re-invention of those ideas - a good sign! Also, how did you do that graphic BTW? We could use more graphics on the wiki explaining transactions, among other things, just saying... :)

There are already lots of ways to practically DoS the testnet, for instance, by draining all the coins from the faucet or by finding the nodes and DDoSing them, or bringing up lots of dummy nodes. I'm not sure we should be trying to optimise the testnet for robustness against malicious attackers, except in the sense that it's useful to test that.

Yes I know it's possible to run private testnets. I just don't see why this is a big deal. Forward progress is important.

Seems to me that everyone who commented has proposed something similar to requiring the fees of the replacement transaction be higher than the original. Requiring an increase of txsize * MIN_RELAY_FEE for each replacement has been proposed multiple times and would require just one or two lines of code. Its important that the design accept that network bandwidth isn't free, and that rule matches the same rule elsewhere for accepting transaction to the mempool.

I'll ACK that.

I haven't commented because this is the wrong place for the discussion. Suffice it to say, that is not how the system was designed to work and that change would make it fairly useless, so I will not be doing it that way. Once again, if you want to propose a re-design that doesn't meet Satoshi's original goals, go ahead and do that - on the forum or mailing list.

huh, so I guess you are expecting replacements are free? what are your thoughts on how dos protection might eventually work?

I'll send a mail to the list with my overall plan in it.

post first next time

What I'm going to post is not anything new, you just weren't there for the original discussions (they're buried on the forum somewhere). But I will happily repeat and elaborate on the topic.

fwiw, I've been around bitcoin a lot longer than you might realize

Please, let's keep it on topic.

As long as it is about a proof-of-concept DoS protection mechanism as long as this in testnet-only, I see no problem discussing it here. Before enabling it on the main network we can have a broader discussion, but that shouldn't prevent experimenting with the technical infrastructure for transaction replacement first.

The easiest solution is certainly requiring that every replacement adds MIN_FEE. I think that's overkill, but it should work. It means you have to keep enough change around for all times you may need to replace, and forgetting this would in practice result in your non-final transaction not being replaceable. So what I suggest is requiring that every replaced version pays for its size and the size of all its previous versions together, using whatever the fee policy is at that time. You can pay the fee once in advance, or increase it along the way if needed. This is still trivial to implement, IMHO, and one only pays for replacements effectively broadcast on the network.

EDIT: Even better, you can retain this "summed seen size" of transaction versions, and use it in priority calculations when constructing blocks. This gives you an automatic prioritization for block inclusion based on the load you've caused to the network.

Taking a higher level view: There are two conflicting, equally valid goals:

1. Mimic mainnet rules as closely as possible
2. Provide an avenue for testing new features, enabling and testing them on testnet before doing so on mainnet

Existing members of the ecosystem probably value 1) more highly. Unless it's a feature they want to see added/enabled. In which case, they would value 2) more highly. Neither is the "wrong" answer.

Enabling a new feature may inhibit testing of an existing feature, thereby hiding bugs.

Maybe 'testnet' (existing feature testing) needs a sister coin 'nextnet'?

So, I would NAK this pull req on testnet, and ACK this pull req on nextnet, using that logic.

Adding some color (or perhaps another take on it) to jgarzik's position, I'd say that to whatever extent we enable things on testnet they should be of a form and character that we at least believe we'd eventually implement on mainnet. E.g. testing not pure research. So I'm okay with testnet being "nextnet" to a degree— pulling in features we've basically agreed to for mainnet but feel need more testing... I'm less okay with it being "perhapsnet".

I don't think we should stress too much about implementing a maybe net. Lots of options like vps servers exist for people to try their new concepts. Anyway according to my seed usually only thirty to fifty test net nodes are ever running.

@sipa tax replacement can only replace outputs and scriptsigs, not inputs. You already have to pick in advance your max fee and gradually increase it by replacing outputs.

@petertodd I know. I don't see how that conflicts with my proposal? You have to pick a maximum change in advance indeed, but you're not required to actually use more and more of it at every step. You can, if you want to compensate for the decreased priority, though.

Ah good. See I think the next question is do inputs really have to be locked? What's wrong with adding to the input set?

Pieter Wuille notifications@github.com wrote:

@petertodd I know. I don't see how that conflicts with my proposal? You have to pick a maximum change in advance indeed, but you're not required to actually use more and more of it at every step. You can, if you want to compensate for the decreased priority, though.

---

Reply to this email directly or view it on GitHub: #2516 (comment)

Re: nextnet. I'm agnostic. Having a public testnet is convenient but indeed not technically necessary. Having a third "researchnet" is probably overkill though, there's overhead involved and given there aren't many testnet nodes to start with I guess a researchnet/nextnet would have even fewer.

So I'd prefer to just re-use the testnet. The cost/benefit tradeoff there feels right. But as Jeff says, it's something reasonable people can disagree on, neither position is obviously the best.

With the recent -development mailing list post pointing out how the "rapidly-adjusting micropayments" use on the Contracts page in the wiki doesn't even work due to a bug that would be obvious to anyone who actually wrote an implementation I'm also going to amend my NAK to include the requirement for a toy-implementation. At least tx-replacement example should be implemented, but ideally a few use-cases. @mikehearn you could have easily done that on a single machine with a modified bitcoind and determined if a fix is or is not possible, and what changes (if any) need to be done with tx replacement without wasting everyones' time.

Of course, I too am guilty of advocating stuff without writing any code...

Closing; no clear consensus and this never really moved forward. It is a trivial patch and we can re-open in an instant, if there is consensus that we want this.

Revisiting the discussion, I'm more agnostic: if other devs think there is value in this feature on testnet, ok. Otherwise, not. I tend to prefer testnet stay as close as possible to mainnet rules, but it is a philosophical point.

Note that this pull-req has been implemented by #6871, which also includes anti-DoS protection via fees.