0.15: Backports

MarcoFalke commented at 2:16 pm on November 1, 2017: member

fanquake added this to the milestone 0.15.0.2 on Nov 1, 2017

fanquake added the label Tests on Nov 1, 2017

ryanofsky commented at 2:28 pm on November 1, 2017: member

utACK 6666d9524148931d261aed1be6700562876e8116

laanwj commented at 2:35 pm on November 1, 2017: member

utACK, though wouldn’t we in the long run be better off backporting the function name change to the 0.15 branch wholesale?

TheBlueMatt commented at 2:36 pm on November 1, 2017: member

Given we want to use the 0.15 branch for segwit-wallet, that seems like a wise decision to me.

[tests] remove direct testing on JSONRPCException from individual test cases

Github-Pull: #10853
Rebased-From: 5864e9c1611e6034a8d49b78231abe6bfade686d

5b728c8e98

[tests] do not allow assert_raises_message to be called with JSONRPCException

Github-Pull: #10853
Rebased-From: 677d893ff758b61f66203730da224bd7ef8f0b43

623de0acbb

scripted-diff: rename assert_raises_jsonrpc to assert_raises_rpc error

-BEGIN VERIFY SCRIPT-
sed -i 's/assert_raises_jsonrpc/assert_raises_rpc_error/g' test/functional/*py test/functional/test_framework/*py
-END VERIFY SCRIPT-

Github-Pull: #10853
Rebased-From: 47ba8cf71e9a3a6f7452488fc9ab3b350f0fae36

d94fc336c4

MarcoFalke force-pushed on Nov 1, 2017

MarcoFalke renamed this:
~~[qa] 0.15.0.2: Fix silent merge conflict (assert_raises_rpc_error)~~
[qa] 0.15: Backports (jnewbery)
on Nov 1, 2017

MarcoFalke commented at 2:50 pm on November 1, 2017: member

Right, #11376 (0.15.1) needs it as well.

ryanofsky commented at 2:54 pm on November 1, 2017: member

Seems fine, but if you wanted a more minimal change you could just add:

0assert_raises_rpc_error = assert_raises_jsonrpc

to the bottom of util.py to add an alias for the new function.

MarcoFalke renamed this:
~~[qa] 0.15: Backports (jnewbery)~~
0.15: Backports
on Nov 1, 2017

MarcoFalke force-pushed on Nov 1, 2017

Ensure backupwallet fails when attempting to backup to source file

Previous behaviour was to destroy the wallet (to zero-length)

Github-Pull: #11376
Rebased-From: 5d465e396249a0e2cc60b16984a2bdbe4c8993c3

fd79ed6b20

[wallet] Fix leak in CDB constructor

Now using a std::unique_ptr, the Db instance is correctly released
when CDB initialization fails.
The internal CDB state and mapFileUseCount are only mutated when
the CDB initialization succeeds.

Github-Pull: #11492
Rebased-From: 7104de8b1f3a31d3a60009b5dc376adbedac6a9c

de7053f114

Avoid opening copied wallet databases simultaneously

Make sure wallet databases have unique fileids. If they don't, throw an error.
BDB caches do not work properly when more than one open database has the same
fileid, because values written to one database may show up in reads to other
databases.

Bitcoin will never create different databases with the same fileid, but users
can create them by manually copying database files.

BDB caching bug was reported by Chris Moore <dooglus@gmail.com>
https://github.com/bitcoin/bitcoin/issues/11429

Fixes #11429

Github-Pull: #11476
Rebased-From: 478a89c1ef79a75275d1b508122c06eee9386b2d

9c8006dc33

qa: Make tmpdir option an absolute path

This should fix issues with the multiwallet test and symlinks
when the tmpdir is a relative path.

Rather than fixing os.symlink to work with paths relative to a
directory descriptor, which does not work on Windows, normalize
the path instead.

Github-Pull: #11472
Rebased-From: fa9de370b17297d6dd542da627c8dd2b31aec340

f3457d0e83

qa: Remove never used return value of sync_with_ping

Github-Pull: #11472
Rebased-From: fafa0039708e15d1067be091b2bfc10195afa480

41088795dd

MarcoFalke force-pushed on Nov 1, 2017

fanquake added the label Backport on Nov 1, 2017

MarcoFalke renamed this:
~~0.15: Backports~~
WIP 0.15: Backports
on Nov 2, 2017

Allow setting nMinimumChainWork on command line

Github-Pull: #10357
Rebased-From: 0311836f6927aec4ba5687ea12af35df3c509682

da4908c3a0

[qa] Test nMinimumChainWork

Nodes don't consider themselves out of "initial block download" until
their active chain has more work than nMinimumChainWork.

While in initial block download, nodes won't relay blocks to their
peers, so test that this parameter functions as intended by verifying
that block relay only succeeds past a given node once its
nMinimumChainWork has been exceeded.

Github-Pull: #10357
Rebased-From: eac64bb7a3b6aba747403b23b3b1d3609843f8db

0e9d04bf0a

Don't process unrequested, low-work blocks

A peer could try to waste our resources by sending us unrequested blocks with
low work, eg to fill up our disk.  Since
e2652002b6011f793185d473f87f1730c625593b we no longer request blocks until we
know we're on a chain with more than nMinimumChainWork (our anti-DoS
threshold), but we would still process unrequested blocks that had more work
than our tip.  This commit fixes that behavior.

Github-Pull: #11458
Rebased-From: ce8cd7a7da9174ab151172fc0ce97b5164637cf3

3acec38781

qa: add test for minchainwork use in acceptblock

Github-Pull: #11458
Rebased-From: 08fd822771bf8dae1c21698811f57aa691b2f25d

2df65eeb98

Add comment explaining forced processing of compact blocks

Github-Pull: #11458
Rebased-From: 01b52cedd42f50a93b40981c91af7c12de6e45ce

ffb6ea4e5e

morcos commented at 5:09 pm on November 2, 2017: member

PR’s backported here: #10853 #11376 #11492 #11476 #11472 #10357 #11458 57edc0b0 from #11456 #10756 #11326 #11490 #11568 #11578 #11531 (utACK from me, got same backport as 55b7abf through this PR) 6baa317 fixup for minchainwork.py from 11490 #11100 #11560 #11593 (utACK from me, got same backport as 8195cb0)

Rename fAddnode to a more-descriptive "manual_connection"

Github-Pull: #11456
Rebased-From: 57edc0b0c86549020a39cd65f96496e9771c4769

6f279652b0

net: pass CConnman via pointer rather than reference

There are a few too many edge-cases here to make this a scripted diff.

The following commits will move a few functions into PeerLogicValidation, where
the local connman instance can be used. This change prepares for that usage.

Github-Pull: #10756
Rebased-From: 28f11e9406b185dc87144f1f29af0d93eb115b4e

8aee55af3d

net: use an interface class rather than signals for message processing

Drop boost signals in favor of a stateful class. This will allow the message
processing loop to actually move to net_processing in a future step.

Github-Pull: #10756
Rebased-From: 8ad663c1fa88d68843e45580deced56112343183

dc897e53d8

net: drop unused connman param

The copy in PeerLogicValidation can be used instead.

Github-Pull: #10756
Rebased-From: 80e2e9d0cec890c5d2f81360ebb81e81c07ccb8c

b4136f21cf

net: stop both net/net_processing before destroying them

This should avoid either attempting to use an invalid reference/pointer to the
other.

Github-Pull: #10756
Rebased-From: 2525b972af6645ca239ac1078cffb132b402bfbb

0a5477c7e3

Fix uninitialized g_connman crash in Shutdown()

Github-Pull: #11326
Rebased-From: 77939f27f7dc42640ebfb9fe52490a2ddacc3ad4

d570aa4290

Disconnecting from bad outbound peers in IBD

When in IBD, we'd like to use all our outbound peers to help us
sync the chain.  Disconnect any outbound peers whose headers have
insufficient work.

Github-Pull: #11490
Rebased-From: c60fd71a65e841efe187992f46c583a704cc37f5

bf191a7183

Permit disconnection of outbound peers on bad/slow chains

Currently we have no rotation of outbound peers.  If an outbound peer
stops serving us blocks, or is on a consensus-incompatible chain with
less work than our tip (but otherwise valid headers), then we will never
disconnect that peer, even though that peer is using one of our 8
outbound connection slots.  Because we rely on our outbound peers to
find an honest node in order to reach consensus, allowing an
incompatible peer to occupy one of those slots is undesirable,
particularly if it is possible for all such slots to be occupied by such
peers.

Protect against this by always checking to see if a peer's best known
block has less work than our tip, and if so, set a 20 minute timeout --
if the peer is still not known to have caught up to a chain with as much
work as ours after 20 minutes, then send a single getheaders message,
wait 2 more minutes, and if a better header hasn't been received by then,
disconnect that peer.

Note:

- we do not require that our peer sync to the same tip as ours, just an
equal or greater work tip.  (Doing otherwise would risk partitioning the
network in the event of a chain split, and is also unnecessary.)

- we pick 4 of our outbound peers and do not subject them to this logic,
to be more conservative. We don't wish to permit temporary network
issues (or an attacker) to excessively disrupt network topology.

Github-Pull: #11490
Rebased-From: 5a6d00c6defc587e22c93e63029fdd538ce8858d

9961abf9e4

Add unit test for outbound peer eviction

Github-Pull: #11490
Rebased-From: e065249c014a070a8799b2ff947af5b8f012c5c1

e3272242e2

moveonly: factor out headers processing into separate function

ProcessMessages will now return earlier when processing headers
messages, rather than continuing on (and do nothing).

Github-Pull: #11568
Rebased-From: 4637f18522429473e68f6f512a03040e121a446d

fc966bbd2b

Disconnect outbound peers relaying invalid headers

Github-Pull: #11568
Rebased-From: 37886d5e2f9992678dea4b1bd893f4f10d61d3ad

59b210d9a7

net: Add missing lock in ProcessHeadersMessage(...)

Reading the variable mapBlockIndex requires holding the mutex cs_main.

The new "Disconnect outbound peers relaying invalid headers" code
added in commit 37886d5e2f9992678dea4b1bd893f4f10d61d3ad and merged
as part of #11568 two days ago did not lock cs_main prior to accessing
mapBlockIndex.

Github-Pull: #11578
Rebased-From: 2530bf27b72e53cc6ffec27de35f3b487984833d

ec8dedff46

Rewrite p2p-acceptblock in preparation for slight behavior changes

Removes checking whitelisted behavior (which will be removed, the
difference in behavior here makes little sense) and no longer
requires that blocks at the same work as our tip be dropped if not
requested (in part because we *do* request those blocks).

Github-Pull: #11531
Rebased-From: 3b4ac43bc371561b5fa559bccb17dc0702bd4b69

e976c36ddf

Stop always storing blocks from whitelisted peers

There is no reason to wish to store blocks on disk always just
because a peer is whitelisted. This appears to be a historical
quirk to avoid breaking things when the accept limits were added.

Github-Pull: #11531
Rebased-From: 3d9c70ca0fd5d42f31114b689714af1825f0ed30

c6e4d0ce82

Accept unrequested blocks with work equal to our tip

This is a simple change that makes our accept requirements the
same as our request requirements, (ever so slightly) further
decoupling our consensus logic from our FindNextBlocksToDownload
logic in net_processing.

Github-Pull: #11531
Rebased-From: 932f118e6a3779bb3d6c3cc83963cf34ac150e42

51001d684b

Reject headers building on invalid chains by tracking invalidity

This tracks the set of all known invalid-themselves blocks (ie
blocks which we attempted to connect but which were found to be
invalid). This is used to cheaply check if new headers build on an
invalid chain.

While we're at it we also resolve an edge-case in invalidateblock
on pruned nodes which results in them needing a reindex if they
fail to reorg.

Github-Pull: #11531
Rebased-From: 015a5258adffb0cf394f387a95ac9c8afc34cfc3

92d6105c4e

[qa] test that invalid blocks on an invalid chain get a disconnect

Github-Pull: #11531
Rebased-From: 00dcda60f6ee63d35bec640f27efe2338dd71270

5bec7744d1

Make p2p-acceptablock not an extended test

Github-Pull: #11531
Rebased-From: f3d4adfa6ff5db180ec09d93f78cdc8bfda26f96

55b7abfa8a

MarcoFalke force-pushed on Nov 2, 2017

MarcoFalke commented at 5:27 pm on November 2, 2017: member

@morcos missing #11326 😛

MarcoFalke force-pushed on Nov 2, 2017

Fix minchainwork test for 0.15 backport 6baa317b5f

Deprecate confusing blockmaxsize, fix getmininginfo output

* This removes block-size-limiting code in favor of GBT clients
  doing the limiting themselves (if at all).
* -blockmaxsize is deprecated and only used to calculate an implied
  blockmaxweight, addressing confusion from multiple users.
* getmininginfo's currentblocksize return value was returning
  garbage values, and has been removed, also removing a
  GetSerializeSize call in some block generation inner loops and
  potentially addressing some performance edge cases.

Github-Pull: #11100
Rebased-From: ba206d2c63a8d3cbd4a8dd47e9ef126af1bb3bb9

7871a7d3be

Use a sensible default for blockmaxweight

No sensible user will ever keep the default settings here, so not
having sensible defaults only serves to screw users who are
paying less attention, which makes for terrible defaults.

Github-Pull: #11100
Rebased-From: 3dc263c9b9068ee9793b6c7a0023eff16d70fb8f

4c82cea99b

Add release notes describing blockmaxweight deprecation

Github-Pull: #11100
Rebased-From: 6f703e9bf11c5f2fcb0fca0e4243fce6b4b9d35a

bb83fe1902

net: Allow connecting to extra outbound peers

Github-Pull: #11560
Rebased-From: 2d4327db1973a354e9e4153de6958d49120fcde8

49bf090185

Track tip update time and last new block announcement from each peer

Github-Pull: #11560
Rebased-From: db32a6589720e5b15931cef82e477118dfd92669

459f2db425

Connect to an extra outbound peer if our tip is stale

If our tip hasn't updated in a while, that may be because our peers are
not relaying blocks to us that we would consider valid. Allow connection
to an additional outbound peer in that circumstance.

Also, periodically check to see if we are exceeding our target number of
outbound peers, and disconnect the one which has least recently
announced a new block to us (choosing the newest such peer in the case
of tie).

Github-Pull: #11560
Rebased-From: ac7b37cd2bd612a64a4009ba82f1cd1d57f37434

a607a95d81

Add CConnmanTest to mutate g_connman in tests

Github-Pull: #11560
Rebased-From: 83df25736ebaa23777e965d265ed677a8d79f23f

2ed0647ac1

Add unit test for stale tip checking

Github-Pull: #11560
Rebased-From: 626291508c433488439b662f2e88882048fb59fb

fc308a6cdb

rpc: work-around an upstream libevent bug

A rare race condition may trigger while awaiting the body of a message, see
upsteam commit 5ff8eb26371c4dc56f384b2de35bea2d87814779 for details.

This may fix some reported rpc hangs/crashes.

Github-Pull: #11593
Rebased-From: 6b58360f9b64eb0b680a662fdfd590e47f115f44

34153a7e4a

rpc: further constrain the libevent workaround

The bug was introduced in 2.1.6-beta, versions before that don't need the
workaround.

Github-Pull: #11593
Rebased-From: 97932cd2689659addfbb58dc6148928b73af3bd0

8195cb0d7f

MarcoFalke renamed this:
~~WIP 0.15: Backports~~
0.15: Backports
on Nov 2, 2017

morcos approved

morcos commented at 7:54 pm on November 2, 2017: member

utACK 8195cb0 Did not thoroughly vet each change, but backported changes myself and verified conflict resolution.

ryanofsky commented at 8:06 pm on November 2, 2017: member

utACK 8195cb0d7fc4f8699b35aff7a43ed2fb3013608c. I verified that the cherry-pick commits were tagged with the right PR numbers and were previously merged to master, and I confirmed that all the merge conflict resolutions and diffs after cherry-picking looked good.

Used a little bash script to help with this

 0BRANCH=remotes/origin/0.15
 1BACKPORT=11592
 2
 3git reset --hard
 4git checkout $BRANCH
 5for commit in $(git rev-list --reverse --topo-order $BRANCH..remotes/origin/pull/$BACKPORT/head); do
 6    pr=$(git log -n1 --pretty=format:%B "$commit" | sed -n 's/Github-Pull: #//p')
 7    orig=$(git log -n1 --pretty=format:%B "$commit" | sed -n 's/Rebased-From: //p')
 8    echo "- Commit $commit orig $orig pr $pr"
 9    if [ -z "$pr" ]; then
10        # Commit is not from a pr, just cherry pick and add REVIEWME tag
11        git cherry-pick "$commit" || exit 1
12        GIT_EDITOR="sed -i '1 s/.*/REVIEWME\n\n&/'" git commit --amend
13        continue
14    fi
15    if ! git -c rerere.enabled=false cherry-pick $orig > /dev/null; then
16        EDITOR=true git commit -a
17    fi
18    if ! git diff --quiet "HEAD..$commit"; then
19        git read-tree "$commit"
20        git commit -m "REVIEWME from $commit orig $orig pr $pr"
21        while ! git reset --hard; do sleep 1; done
22    fi
23    if ! grep -q "$orig" <(git rev-list $BRANCH..remotes/origin/pull/$pr/head); then
24        echo "Error: Commit $orig not found in pr $pr" >&2
25        exit 1
26    fi
27    if [ "$(git merge-base $orig origin/master)" != $orig ]; then
28        echo "Error: Commit $orig from pr $pr not found in master" >&2
29        exit 1
30    fi
31done
32
33git log --grep REVIEWME -p $BRANCH..HEAD

theuni commented at 9:23 pm on November 2, 2017: member

utACK 8195cb0d7fc4f8699b35aff7a43ed2fb3013608c, assuming Travis is happy with it.

luke-jr commented at 10:29 pm on November 2, 2017: member

NACK #11100. Removing functionality has no place being snuck into a backport, and benefits nobody. It is only ever a reason for NOT upgrading.

gmaxwell commented at 10:37 pm on November 2, 2017: contributor

@luke-jr your opposition to #11100 was noted on the PR itself.

sdaftuar commented at 0:41 am on November 3, 2017: member

I manually verified the backports here; conflict resolution looks correct to me as well.

ACK 8195cb0d7fc4f8699b35aff7a43ed2fb3013608c

meshcollider commented at 2:25 am on November 3, 2017: contributor

utACK https://github.com/bitcoin/bitcoin/commit/8195cb0d7fc4f8699b35aff7a43ed2fb3013608c Thanks for the script @ryanofsky :) That should perhaps be put into contrib/devtools/

sipa commented at 10:31 pm on November 3, 2017: member

utACK 8195cb0d7fc4f8699b35aff7a43ed2fb3013608c, looked over the list of backported PRs, and verified using @ryanofsky’s script.

sipa merged this on Nov 3, 2017

sipa closed this on Nov 3, 2017

sipa referenced this in commit f518d9ae6a on Nov 3, 2017

MarcoFalke deleted the branch on Apr 30, 2018

codablock referenced this in commit b2c6b48031 on Sep 30, 2019

MarcoFalke locked this on Sep 8, 2021

0.15: Backports #11592