This is half of an idea from http://www.tik.ee.ethz.ch/file/49318d3f56c1d525aabf7fda78b23fc0/P2P2013_041.pdf
Full idea: Relay blocks as soon as the quick, transaction-independent checks are done (most critically, has-valid-proof-of-work).
To avoid any network-splitting potential, that can only be done if relaying blocks with valid proof-of-work but (say) an invalid transaction does NOT trigger a DoS-ban.
I think that should be done for the 0.9 release.
To avoid splitting the network on un-banning, we'll need a new block message...
The plan could be:
An attacker could then try to split off a minority of the network by broadcasting a valid-pow-but-invalid-txn block, but at great cost and arbitrarily little benefit.
A new variation of the 'block' message ("block_that_might_possibly_be_invalid") would work, too, but feels like the wrong solution to me right now. Then again, having "block" mean "I'm 100% sure this block and all of its transactions are okey dokey" and another message meaning something slightly different might be best.
I don't think we should relay these as the same as validated blocks. Even with the DoS stuff out of the way, SPV nodes depend on full nodes to validate the blocks they've been handed.
Luke implemented the block_that_might_possibly_be_invalid approach a while back in a long since closed pull request. It didn't yield improved propagation because of other single threading in the network handling code.
Some thought is required here. That the existing network won't forward your blocks is part of the friction against "altcoin attacks", e.g. someone promoting an uncontrolled non-consensus hardfork.
You could name the new message "block?" ;)
Why not just bump the nVersion and only relay possibly invalid to newer peers that don't DoS ban?
Both the super-majority and new message options seem overly complex to me.
I'm wondering about the alternative of only relaying block headers after PoW is validated (which doesn't even need an 'inv' or making the block available to be queried from disk), and using the 'header first seen' criterion as tie-breaker for decision which chain to consider active in case of a fork.
@sipa Creates a perverse incentive where you announce your found header fast but withhold the actual block until you find the next one so you can mine on it where other people cant. Then that potentially creates a perverse response where you mine empty blocks on headers you have seen but haven't / can't-yet validate, and then perhaps a doubly perverse counter response where you then never announce the block when you weren't the party that got the block after it...
@gmaxwell Once a successor block is mined on top, the early-relayed header becomes meaningless, as the rule would be to always aim to connect:
@petertodd 's suggestion seems to make the most sense to me.
Relaying blocks before checking their transactions would negatively impact miner incentives.
Miners have strong incentives to minimize orphan rates of their blocks. Orphan rates are a function of block propagation time. Block propagation time is heavily impacted by how many signatures in the block need to be computed upon block receipt, as opposed to being cached when the transactions were received individually.
This means that miners are currently incentivized to disperse to the network as quickly and widely as possible not just the blocks they mine, but also the transactions contained in blocks their hashpower is attempting to mine -- which not only encourages transaction propagation, but preferentially encourages the propagation of the transactions most likely to be included in upcoming blocks.
Removing the effect of ahead-of-block transaction relay on miners' expected returns would eliminate their incentive to use their relay networks to push to the network any transactions. This is an incentive that I believe currently plays an active role in the network: I've been running my node with extra logging for a while that displays how many of the transactions in a received block were already known, and found that some of the time 100% of non-coinbase transactions have already been validated, while the average is much lower (typically ~70%). I think the most likely reason for this higher cluster would be that some miners ensure that transactions they're working on are relayed to the network as broadly as possible, while others do not have expansive relay networks or ill-advisedly use their relay networks only to push blocks.
Unfortunately this mechanism does currently disincentivize miners from including in blocks non-standard transactions, since their signatures will typically be verified only once they are received in a block. However, this can be addressed without eliminating miners' incentives to broadcast transactions: one approach would be a network message for "shares", runt blocks that meet a difficulty criterion some fixed proportion of the network's actual requirement, that could be used to prove to nodes that work on a block containing certain transactions is underway and (like merkleblock) provide a subset of the contained transactions (incentives dictate that miners would typically include only the non-standard transactions), allowing nodes to verify in advance transactions they would not otherwise process or forward.
Miners currently have an an incentive to relay transactions they include in blocks but It's more the reverse way: they have an incentive to include transactions already relayed by the network and already in other miners pools. So relaying headers is optimal, as long as the transactions included are known. Mining on empty headers is also fine, as long as the miner rolls-back to whatever block was mining before if after some time (e.g. 10 seconds) the transactions are not received and verified. The maximum empty mining time is accumulated for successive blocks (if a miner doesn't have transactions for the last 2 blocks, then he waits only 10 seconds before rolling-back, not 20 seconds, to prevent creating an infinite long empty branch).
Publishing a header but withholding transactions becomes a bad strategy. The only drawback is that a block mined on top of an unverified header must em empty (apart from the coinbase or self-transactions), and this requires the subsidy to be non-zero to incentivize this mining) So the header-first strategy with immediate mining will work for, probably, 15 years. There will be plenty of time to think of a better strategy then. As an additional measure, the header can be marked with a bit in the nVersion field if it is empty, to allow miners so start mining non-empty blocks when an empty header is received.
A little more info I posted here: http://bitslog.wordpress.com/2014/02/17/5-sec-block-interval/ but not much that has not already been said.
I had the opportunity to talk with sipa about this, and it is clear to me now: as gmaxwell points out, whoever finds a block would have an incentive to only announce the header but withhold the actual block. As long as that miner is the only one who know the full block, he is also the only one who can mine additional blocks with transactions, and thus the only one who can collect transaction fees. So the proposed timeout would lead to a situation in which the full block is withheld until just before the timeout, which is not something we would want.
-> IMHO, this issue can be closed.
Whoever finds a block and only announce the header but withhold the actual block is doing a very risky bet, that he'll probably loose. He is risking 25 BTC. He would need to wait a number of seconds between 0 and 10 hoping that no miner times-out in between, only to earn as little as 1% (tx fees / subsidy) during 10 seconds window (which represents $ 2 (USD) more on the average block reward). Miners are currently including transactions which cost them about $ 100 (USD) in block propagation time for free, so they will be happy to spend $ 2 instead of $ 100.
If you want to prevent the $ 2 incentive, then set a rule that miners won't switch back to a full-block X mining, after the header-only mining of X has time-out, before another 10 seconds timeout. So now there is a penalty for sending the header and not the data.
@SergioDemianLerner That's what I thought first too. However, once you try to define a clear and concise rule about when to accept a block and when not, things turn ugly. For example, what if the block transmission hits the timeout with 20% of the nodes, but is ok with the majority? When should you re-accept a block you previously dismissed due to reaching the timeout? Whenever you try to fix one issue, a few new ones emerge. This is typically a symptom of trying to do something that does not fit the spirit of the system - on our case accepting a block without having seen it. And even if you come up with watertight rules for when a block should be accepted, it will be extremely hard to prove that the rules actually are watertight.
Think about this: a miner A that withhold the block data (but not the header) does not get any direct additional income. Only if another miner B finds a block within the 10 second window and the block has no transactions because the block transactions were never received by X, then A could take those transaction fees if he is lucky enough to mine the next block, and assuming the remaining existent transactions in the pool do not provide equal profit. But if the block is saturated the remaining transactions, on average, should have approximately the same amount of fees. So nothing is gained.
As an example, assuming the current state of Bitcoin. Let $ 40 (USD) be the average tx fees per block. Let the probability of a collision in a 10 seconds windows be 1.6%. if miner A has 20 % of the hashing rate, and 1 every two blocks has transaction fees which are 50% higher than the next block fees, then A would get an average of 40/2/5= 4 (USD) of additional income per block by not relying txs on a collision, which happens 1.6% of the times. Or an average of 6 cents per block. Or a total of $ 57 (USD) more per month over 54 MILLION USD earned. Not very significant.
And this is what Microsoft "Bitcoin and Red Balloons" discovered two years ago. That Miners withholding transactions could get some advantage in a purely theoretical setup (or in a distant future, when subsidy is near zero), but in practice no miner is withholding transactions today, and no miner will do it for the next 15 years, if Bitcoin succeeds.
This is true while the subsidy is high. In 20 years, when Bitcoin is the world most valuable currency, but there is no subsidy and only tx fees matter, miners will voluntarily turn off this feature. No big deal.
And there there are other reasons why miners won't invest in gambling this aspect of the system. Think about the current network state. Why aren't all miners using TheBlueMatt high speed relay system? Why haven't they implemented a system to relay blocks without sending all the already known transactions? Why aren't they creating blocks without transactions and saving 100 $ in every block?
Because Miners generally follow the rules. They are conservative. They don't seem to want to maximize their short-term profit to the extend to destroying Bitcoin. So miner's won't rewrite their mining software to increase their profit in 6 cents per block (the additional mining fees they could get, if they manage not to risk 10K USD by doing it).
At any time a miner that doesn't think this is good for him can stop mining on headers, but if the majority of the remaining miners still do it, he will be loosing a lot of money. A miner switching off this feature today would loose 1.6% of every block mined on average (aprox. 200 USD).
And about the spirit of Bitcoin. I don't know which this spirit is, but (IMHO) the spirit of Bitcoin Mining is this: every miner wants his block to be accepted by the network as much as possible. Sooner or later miners will exchange block headers before block transaction-data, wether using the Bitcoin network or using a private backbone. We won't be able to stop this.
@SergioDemianLerner As you did not seem to have noticed: implementing a protocol to relay blocks faster (without having to resend known transactions) is exactly what Gavin recently proposed and is currently working on. It might look more complex at first, but it has also much fewer unintended side-effects.
Because Miners generally follow the rules. They are conservative.They don't seem to want to maximize their short-term profit to the extend to destroying Bitcoin.
GHash.IO is strong evidence otherwise. Equally, it can be interpreted that we have miners who understand the system so poorly they'll do harm to it by accident.
So miner's won't rewrite their mining software to increase their profit in 6 cents per block
gmaxwell reported recently that some miners were using getblocktemplate to get a head start on new blocks on the network and configuring their miners to mine on top of them without doing any validation.
@petertodd Interesting. I had previously said (without actually knowing) that this will happen sooner or later. And it's happening very soon.. But they are doing it not for the 6 cents (it header-only transmission were implemented in the protocol), but for 100 USD because we haven't implemented the header-only transmission! If we implement it correctly, then they will still do it, but in a controlled way (e.g. for no more than 10 seconds). This is how Bitcoin freedom works: we don't dictate the rules, we try to best accommodate the network so miners can still do what they want to do, but securely.
This is how Bitcoin freedom works: we don't dictate the rules, we try to best accommodate the network so miners can still do what they want to do, but securely.
What miners want is more money; if that involves less security so what? Attacks and screwups are rare enough that they personally aren't being harmed sufficiently to disuade them.
Anyway, it's just a flaw in the protocol, fortunately one we can fix in a soft-fork. But that's a topic for elsewhere.
@petertodd can you reference the soft-fork in question once it has a pull request please?
Compact Blocks behave like this nowadays, right?
@Sjors Depends on what you mean by "this". If a peer explicitly requests it you'll forward blocks to them that meet a subset of block validity requirements, but not otherwise, and they'll only ban you if the block is invalid in that subset you were supposed to still have checked. The subset includes hash tree consistency, so you're required to have access to all the data for the block: you won't relay a block where you don't know enough to check if it was valid or not. This avoids the above discussed attacks where block payload is withheld, the fact that the behaviour has to be requested avoids the problem of sending invalid data to clients that expected us to validate for them.