getblocktemplate fails for regtest mode. #11379

Describe the issue

In the lightning tests, we run bitcoind in regtest mode. ~~~We started seeing intermittent test failures, caused by generate not including transactions we had just sendrawtransactioned; this delay is noticeable when there are multiple submitted txs.~~~ (Edit: my test harness fail). I tried to work around it by polling getblocktemplate to see when our txs are included, only to run into this bug :(

1. bitcoind -regtest -testnet=0 &
2. bitcoin-cli -regtest -testnet=0 getblocktemplate

Expected behaviour

I should get the block template.

Actual behaviour

error code: -9 error message: Bitcoin is not connected!

What version of bitcoin-core are you using?

v0.14.2.0-gfc61c83

getblocktemplate requires that you are connected to at least one other node. This is not a bug, it is intended behavior.

I suppose the purpose behind this is so people do not make blocks without knowing the state of the network.

Didn't investigated the issue, but it may be related with the gact that there's no CChainParams::MiningRequiresPeers() exists in 0.14 but not in 0.15 ? Perhaps it is a regression? @instagibbs yeah, that's the point, but shouldn't be a requirement for regtest.

@achow101 That's not very sane for regtest, is it? And the unreliable sendrawtransaction/generate behavior on regtest seems to be most simply fixed by allowing this.

Did you investigate the unreliable sendrawtransaction/generate behavior? I'm skeptical of adding yet more network-specific behavior to chainparams, and can't think of any reasons transactions in mempool would not be included in freshly-CreateNewBlock'd things. Was the generate call to a different node than the sendrawtransaction call was to?

@TheBlueMatt actually, yes, I had a bug which caused this (I wasn't actually waiting for sendrawtransaction to complete in a corner case). So I'm not actually hitting this IRL, which reduces my care factor significantly.

The getblocktemplate failure is weird, though. regtest is all about being a solo network, AFAICT, and if I were writing mining software I'd probably want to test this way.

@jtimon It's not a regression; that you have connections is explicitly checked for in getblocktemplate: https://github.com/bitcoin/bitcoin/blob/master/src/rpc/mining.cpp#L441 @rustyrussell

That's not very sane for regtest, is it?

Perhaps.

And the unreliable sendrawtransaction/generate behavior on regtest seems to be most simply fixed by allowing this.

How so? AFAIK generate and getblocktemplate use the same code to produce the block template which generate uses to then actually make the block.

Unless you mean it would allow you to workaround your issue (by repeatedly calling getblocktemplate and just checking for it there); in which case this is just a workaround, not a fix of the actual problem (transactions just sent not getting into blocks).

And I can't just generate another block until the tx is included, because we use locktimes and explicitly test set number of blocks.

You could keep invalidating blocks at a height until one is mined which satisfies your criteria :stuck_out_tongue:

The getblocktemplate failure is weird, though. regtest is all about being a solo network, AFAICT, and if I were writing mining software I'd probably want to test this way.

This has tripped me up before, but I think it makes sense to keep the current behaviour. Any regtest-specific behaviour means that regtest isn't testing the same code paths as a node running mainnet.

We work around this in our functional tests by just spinning up a second node to peer with the node-under-test. See for example https://github.com/bitcoin/bitcoin/blob/master/test/functional/mining.py#L31 where the second node is brought up for this purpose.

@rustyrussell Unfortunately even if we fix or workaround the connectivity requirement issue, I suspect that you'd likely still run into problems using getblocktemplate to test in this way, due to the caching of templates in gbt. (This has bitten me before.)

I'd be supportive of an rpc command that just called CreateNewBlock and returned the results, without supporting any of the getblocktemplate options / behaviors, in order to explicitly test transaction selection.

I have the same issue. How can I test getblocktemplate without using testnet, as waiting for sync takes too much time.

(saw this linked to complaints in IRC)

Would people still be interested in an alternative block-getter RPC?

We've had one in Elements for a few years now and it'd be an easy port.

@instagibbs A (hidden) testing-specific one, or a new one for users as well?

From the context here it would be offered as testing only, so hidden?

On Sun, Jun 2, 2019, 4:44 AM MarcoFalke notifications@github.com wrote:

@instagibbs https://github.com/instagibbs A (hidden) testing-specific one, or a new one for users as well?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/11379?email_source=notifications&email_token=ABMAFU37NSYQJENZ4XTP5NTPYOB6VA5CNFSM4D32WVQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXQ2PQ#issuecomment-498011454, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMAFUYEGLCILIM3INYD7RLPYOB6VANCNFSM4D32WVQA .

just hit this... yes please! Or at least we can improve the docs / error message for regtest

Perhaps a better solution would be to have a helper for starting N regtest nodes while testing? I could see starting up a mini test network with some fake traffic as being very useful...

95d5d5e6257825bb385cee318d5681597f7f7646

Odd -- I believe I have this patch but still not working for me... in any case should I PR this as a contrib script?

#!/bin/bash
TMP=$(mktemp -d)
echo $TMP
CONF=$TMP/extra.conf
RPCPASSWORD=$(echo $RANDOM |shasum -a 256 | cut -d ' ' -f1)
RPCUSER=$(echo $RPCPASSWORD |shasum -a 256 | cut -d ' ' -f1)
echo "regtest=1" >> $CONF
echo "[regtest]" >> $CONF
echo "server=1" >> $CONF
echo "txindex=1" >> $CONF
echo "daemon=1" >> $CONF
echo "debug=rpc" >> $CONF
echo "rpcuser=$RPCUSER" >> $CONF
echo "rpcpassword=$RPCPASSWORD" >> $CONF
echo $RPCUSER $RPCPASSWORD
create_node() {
    THIS=$TMP/$3/bitcoin.conf
    cp $CONF $THIS
    echo "port=$1">>$THIS
    echo "rpcport=$2">>$THIS
    ./src/bitcoind  -datadir=$TMP/$3 &
}
stop_node() {
    ./src/bitcoin-cli -conf=$TMP/$1/bitcoin.conf stop
}

peer() {
    ./src/bitcoin-cli -conf=$TMP/$1/bitcoin.conf addnode "127.0.0.1:$2" "add"
    ./src/bitcoin-cli -conf=$TMP/$1/bitcoin.conf addnode "127.0.0.1:$3" "add"
}

mkdir $TMP/A
mkdir $TMP/B
mkdir $TMP/C
create_node 8332 18332 A

create_node 8333 18333 B

create_node 8334 18334 C


cleanup() {
    stop_node A
    stop_node B
    stop_node C
}
trap 'cleanup' 2

echo "waiting"
sleep 3
echo "Node Started"


peer A 8333 8334
peer B 8332 8334
peer C 8332 8333

createwallet() {
    ./src/bitcoin-cli -conf=$TMP/$1/bitcoin.conf createwallet default
}

mine() {
    while true; do ./src/bitcoin-cli -conf=$TMP/$1/bitcoin.conf -generate 1; sleep 30; done
}
createwallet A
mine A &
tail -c+1 -f $TMP/A/regtest/debug.log

I believe I have this patch but still not working for me.

What are the steps to reproduce?

Works fine locally:

$ ./src/bitcoind -datadir=/tmp -regtest -noprinttoconsole &
$ ./src/bitcoin-cli -datadir=/tmp -regtest getblocktemplate '{"rules": ["segwit"]}' | wc -l
36