Bitcoin Unlimited Pull Request #9731

Commented out the code in main.cpp

Add command to get static compiled unix binaries

Due to the fact it takes too long building and that
won't let us terminate the `make check` and the
python qa tests.

This test pass if executed in isolation whereas it could fails
when it run together with all others due to the fact that
minRelayTxFee values could be changed by some test executed
before this one. This is happen because mineRelayTxFee is a global
variable and it is shared by all tests belonging to the suite.

TODO: add a unit test to exercise the {min,max}limitertxfee
machinery

Some of the regression tests cause a heap-use-after-free issue because
they use FindNode to find a CNode pointer and then attempt to set the
fDisconnet flag on that node, however the CNode instance has since been
destroyed inside of ThreadSocketHandler between FindNode returning the
pointer and attempting to set the fDisconnect flag.  This affects the
setban and disconnectnode functions in rpcnet.cpp.

1. Update all FindNode functions to assert that the cs_vNodes lock is
held, instead of taking the lock internally.
2. Update all calls to FindNode to add a cs_vNodes lock in the caller
scope.
2.a. For simple functionality, just hold the cs_vNodes lock until we are
done with the returned CNode pointer.
2.b. Anywhere the returned CNode pointer is used in/around potentially
expensive operations (UI updates, network communication, file I/O), take
the lock in a small scope block and use CNode::AddRef() to protect the
node from deletion prior to allowing the lock on cs_vNodes to go out of
scope.  Subsequently use CNode::Release() to remove the reference once
we are done with the CNode pointer.

it's not true, we like Linux :-)

Make Travis works as expected along the current BU codebase

[qa] add capability to disable/skip tests and have them show up as such in the log

Fix formatting code paragraph in static binaries section

Fix heap-use-after-free error that sometimes happens when running the nodehandling.py test.

Add unit tests for EraseOrphansByTime

ref. Core commit 5467820be5404fa736436a1202079e764a4308b4

This commit will fix the 3 following warnings:

a) net.cpp:893:13: warning: ‘bool ReverseCompareNodeTimeConnected ...  [-Wunused-function]
b) chainparams.cpp:41:15: warning: ‘CBlock CreateGenesisBlock ...  [-Wunused-function]
c) chainparamsseeds.h:10:18: warning: ‘pnSeed6_main’ defined but not used [-Wunused-variable]

All of them are related to declared but not used
functions (a,b) or variable (c)

add test_classes.py to EXTRA_DIST, for builds set up using distdir target

remove rpc-tests.sh which was replaced by Python script

Remove compiler warns

Update the bloom filter handling for this test.

Such function is spelled wrongly (in unlimited.cpp the name
ClearThinBlockTimer()) and more to the point is not used
anywhere but in unlimited.cpp.

also replace one occurence of Core with Unlimited and add $PWD
to the configure prefix, otherwise it may complain like this:

configure: error: expected an absolute directory name for --prefix:
./depends/x86_64-pc-linux-gnu

We do not know that a class of Network is reachable, only that it is not.

Github-Pull: #7553
Rebased-From: 110b62f06992d0fb989153afff2dc3aea62a674f

Fix some typos in build-unix.md

Fix README.md install instructions

Add Travis status icon to README.md

Add copyright and licence headers to requestManager.cpp

add -only-extended option to rpc-tests.py to run only extended tests

Adds a ValidationResourceTracker class that is passed to
CheckInputs() / CScriptCheck() to keep track of the exact number
of signature operations required to validate something, and the
exact number of bytes hashed to compute signature hashes.

Also extends CHashWriter to keep track of number of bytes hashed.

This is a subset of the original work by Gavin Andresen.

Cherry pick portions from b8a97498 in Core to properly display remote
node services offereing the Witness service.  This is only so that nodes
signalling for SegWit show as offering the "Witness" service instead of
"Unknown".

Github-Pull: #8317
Rebased-From: 477777f2503e3a56a267556f0fc5091042d93340

This change is needed to prevent sync_blocks timeouts in the mempool_reorg
test after the sync_blocks update in the upcoming commit
"[qa] Change sync_blocks to pick smarter maxheight".

This change was initially suggested by Suhas Daftuar <sdaftuar@chaincode.com>
in https://github.com/bitcoin/bitcoin/pull/8680#r78209060

Github-Pull: #9196
Rebased-From: 67c6326abd1788e6f411feb4f44b69774e76aae2

Changed http:// to https:// on some links in README.md

When mining large numbers of blocks and then syncing them to many
nodes it sometimes happens that a GET_XTHIN request can jump
ahead of another GET_XTHIN request or HEADER request resulting in blocks
showing up out of order and with no headers connected in the block index.
This results in a small DOS(10) and a subsequent node disconnect if this
happens 10 times during a regression test run.

When generating a new service key, explicitly request a RSA1024 one.

The bitcoin P2P protocol has no support for the longer hidden service names
that will come with ed25519 keys, until it does, we depend on the old
hidden service type so make this explicit.

See #9214.

Github-Pull: #9234
Rebased-From: 7d3b627395582ae7c9d54ebdbc68096d7042162b

This is added to avoid duplication of random seeding in tests that want to make use of randomness.

The test framework now initializes the RNG based on current time in seconds since the epoch unless --randomseed=X is specified.
If X is specified it is converted to an integer.

The seed used is made accessible through the attribute `self.randomseed` in BitcoinTestFramework.

Tests which use random numbers should print out their seed before doing so.
In this way, if they crash due to random-generated data this increases the chances that it will be possible to reproduce the fault, e.g. like this:
(assuming the `excessive` test crashed with a seed of 12345)

`qa/pull-tester/rpc-tests.py --randomseed=12345 excessive`

This parses the output of netstat -n to find the current ports that
are in use and then we make sure we don't select on of those when
we launch the test node.

This will be needed so that the test runner can distinguish tests that are invoked
with the same script, but different options.

This is a cross-port of some rpctests.py test runner changes made in the BTCfork project [1].

Overview:

- added a new `-list` option which, when given, only lists tests without executing them. This is useful for filtering a subset of tests without having to dissect the contents of the Python script. The `-list` option can be combined with `-extended` or `-extended-only` to list all tests or just the extended set of tests.
- command line option handling has been made a little more user-friendly by accepting double-hyphen forms as well for the single-hyphen options. Options like '-only-extended` have been given counterparts `-extended-only` to minimize friction.
- a warning message is printed for unrecognized options
- at the end of the test run, an overview table is printed, similar to what latest versions of the Core client software do. The overview shows executed tests along with their status and time taken, and also prints out aggregate information about the number of passed / failed / disabled / skipped tests. This is intended to facilitate a better overview and "feel" for how tests are affected during development, both in terms of performance and failures.

[1] https://github.com/BTCfork/hardfork_prototype_1_mvf-core/commits/master/qa/pull-tester/rpc-tests.py

Github-Pull: #9295
Rebased-From: 1a6eacbf3b7e3d5941fec1154079bbc4678ce861

Github-Pull: #9295
Rebased-From: c24a4f5981d47d55aa9e4eb40294832a4d38fb80

also add --mineblock as a recognized option since it is used by two tests

Two missing hyphens, one unnecessary "in".

Update the unit tests for thinblock_tests.cpp

Remove declaration of ClearThinblockTimer from unlimited.h

Remove vfReachable and modify IsReachable to only use vfLimited.

[Trival] - Update remote node UI display to show "WITNESS" instead of "UNKNOWN".

Don't allow undefined setminingmaxblock parameter

Changed http:// to https:// on some links

Do not queue jump GET_XTHIN in test environments

Add missing cs_orphancache lock to expedited block processing

add --randomseed option to Python test framework

Fix thinblock stats map interators from going out of range

Prevent the regression test from using a port that is arleady in use

Usability enhancements for rpc-tests.py: listing, better option handling, summary of test results

add lock for nLargestBlockSeen

Edits for grammar

Fix LargestBlockSeen to update properly in a lockless but atomic manner.

Implements #6211.

[skip ci]

d609895 [Wallet] Bugfix: FRT: don't terminate when keypool is empty (Jonas Schnelli)
8dee97f [QA] add fundrawtransaction test on a locked wallet with empty keypool (Jonas Schnelli)
82e29e8 torcontrol: Explicitly request RSA1024 private key (Wladimir J. van der Laan)
cca151b Send tip change notification from invalidateblock (Russell Yanofsky)
ad99a79 [rpcwallet] Don't use floating point (MarcoFalke)

On Windows10 this is a problem.  It appears that Netcleanup
and Maincleanup get called on nodeshutdown but also when Qt
exits.  The nodeshutdown completes but then QT wants to finish
and makes the second calls which can't be competed because the node
is already shutdown. Now the process hangs and leaves a partial process
in the tasklist but it isn't obvious to the user because they
can re-start bitcoin-qt without any issues. They will only notice
it if you try to delete the debug.log or look at your tasklist.

remove MainCleanup and NetCleanup from getting called twice

Fix wake from sleep issue with Boost 1.59.0

This reverts commit 1f8459f6607371d27a03af49c6eb4cee2f91215a, reversing
changes made to 12f2f937068bfc771d540fbec767bcbb6568c0f3.

Add a default seeder for the NOL network

Not returning the exit code may interfere with CI systems such as Travis.

return exit code 1 if any tests failed

Add emergent consensus to sigop and transaction size

Move Travis to Ubuntu Trusty
Introduce Out-of-tree build in Travis
Enable verbose log on Travis `make check`
Travis: Change OSX SDK provider
Make possible to build leveldb out of tree
Cherry pick from Core repo (ab95d5)
- Link pull-tester/rpc-tests.py to the build dir
- Add the build-dir's config to the python path so that tests can find it
- The tests themselves are in srcdir
- Clean up __pycache__ in 'make clean'

tests: Make proxy_test work on travis servers without IPv6

manual cherry pick Core 932aed

Manual cherry-pick of Core 92e37a3

This reverts commit 0933a79225c046c65afa69a31d553ef6c68cd6bf.

Manual import Core a4625ac

disable java based comparison test (take #2)
manual cherry-pick Core 92e37a3
fix qt out-of-tree build (manual cherry pick Core 932aed)
update leveldb out-of-tree build
rpc tests: fix extended flag

Manual cherry pick Core's diff 980e7eb9..b31e13e
Upgrade to QT 5.6.1
Update gitian linux recipe and add a separate one for arm
Use make install $DESTDIR rather than install-strip
bitcoin-qt is built only on linux 32 and 64 bit not arm
QT still depend on dbus
remove bdb part of fb0de07

Update travis, out of tree build and gitian ARM

And also move mapThinBlockTimer into CThinBlockStats

remove tab from thinblock.h

This is useful when cross-compiling for archs other
than the one we currently used to build the binaries.
In the case of aarch64 and armhf we need to use
/aarch64-linux-strip and arm-linux-gnueabihf-strip
rather than the usual strip program to successfully
remove debug symbols from the binaries.

Use http://savannah.gnu.org/projects/config repo to
udpat config.{guess,sub}. This is needed if you need
to compile bdb for aarch64.

Fix gitian descriptor for linux and arm

Add aarch64 to travis-ci

Refactor thinblock code

This will prevent any false positives from happening when  unit
tests are run which then gives a failed test result when it
wasn't a true failure.

Add non-deterministic tests to thinblock_tests.cpp

Previously we were forwarding all coinbase spends however this may
not be desireable by some users that want/need to completely prevent all
traffic below a certain fee rate by setting their -limitfreerelay=0.

There is also a potential attack vector here which is prevented in that
some attacker could get a hold of coinbase coins and use them to create
large amounts of spam transactions which would then get around any
rate limiting attempts by node operators.

Use a hardcoded seed value for the bloom filter

Subject coinbase spends to -limitfreerelay

We set the http keepalive timeout to zero to prevent a timeout from
happneing while we're waiting for a process to finish and then
we end up trying to request from a connection that no longer exists.
Instead we timout the connections right away so that each time a new
fresh connection is aquired.

This is something that has been coming up lately in very lately in long
running tests.

Conflicts:
	build-aux/m4/bitcoin_qt.m4

Conflicts:
	.travis.yml
	contrib/devtools/symbol-check.py

Remove references to LONG_LONG_MAX

do not allow HTTP timeouts to happen in excessive.py

Fix qt gitian windows and linux

fixes a coredump in bitcoin-qt when run with `-chain_nol`

fix a coredump in bitcoin-qt when run with `-chain_nol`

Fix compiler warnings in leakybucket.h

leveldb's buildsystem causes us a few problems:
- breaks out-of-tree builds
- forces flags used for some tools
- limits cross builds

Rather than continuing to add wrappers around it, simply integrate it into our
build.

Without this patch:

- When I compile the GUI from the bitcoin directory itself, it works as
  expected.

- When I build the GUI in an out-of-tree build, I cannot get it to
  select tabs. When I click, say the "Receive" tab nothing happens,
  the button selects but it doesn't switch the page. The rest - even
  the debug window - seems to work.

See full discussion here:
https://github.com/bitcoin/bitcoin/pull/7911#issuecomment-212413442

This turned out to be caused by a mismatch in the arguments to moc,
preventing it from finding `bitcoin-config.h`. Fix this by passing
`$(DEFAULT_INCLUDES)` to it, which gets set to the appropriate
path by autoconf itself.

Github-Pull: #7489
Rebased-From: 7539f1aae3b41279dc5d49e09f448a78a071e114

- 'make check' in parallel, since the log will take care of clean output
- 'make check' verbose, so that test failure causes aren't hidden

Now that caches are distinct (https://github.com/travis-ci/travis-ci/issues/4393),
we can use the Travis minimal image.
The minimal image should take less time to setup and lead to quicker builds.

Also addressed while I'm in here:
- No need to delete the broken google-chrome repo in the minimal image
- Set the hostname to work-around an openjdk bug
- Remove the non-functional apt-cache option
- Remove useless message at completion
- Install jre where the java tests are run

Don't glob the leveldb for dist. That means we need to enumerate the headers.

- Link pull-tester/rpc-tests.py to the build dir
- Add the build-dir's config to the python path so that tests can find it
- The tests themselves are in srcdir
- Clean up __pycache__ in 'make clean'

- Copy the qrc locale file to a temp location and remove it when finished
  (rcc expects everything to be in the same path)

The plist is generated, lives in builddir.

This does not break any existing prefix behavior, only makes new behavior work.

For example:
CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure --prefix=/

- update config.guess/config.sub for bdb for aarch64.
- temporarily disable symbol checks for arm/aarch64
- quit renaming to linux32/linux64 and use the host directly

This also adds a hack to work around an Ubuntu bug in the gcc-multilib package:
https://bugs.launchpad.net/ubuntu/+source/gcc-defaults-armhf-cross/+bug/1347820

The problem is that gcc-multilib conflicts with the aarch toolchain.
gcc-multilib installs a symlink that points
/usr/include/asm -> /usr/include/x86_64-linux-gnu/asm.

Without this link, gcc -m32 can't find asm/errno.h (and others), since
/usr/include/x86_64-linux-gnu isn't in its default include path. But
/usr/include/i386-linux-gnu is (though it doesn't exist on disk).

So work around the problem by linking
/usr/include/i386-linux-gnu/asm -> /usr/include/x86_64-linux-gnu/asm.

The symlink fix is actually quite reasonable, but echoing the password into
sudo is nasty, and should probably be addressed in gitian itself. It makes more
sense to enable passwordless sudo for the build user by default.

The only change picked from the commit
was the one about qt.mk. Since it has
nothing to do with the original commit
message it self I've changed it.

For the sake of completeness this is
the original commit message below:

   depends: bump OSX toolchain

   clang: 3.7.1
   cctools: 877.8
   ld64: 253.9

Partial cherry pick

C_INCLUDE_PATH and CPLUS_INCLUDE_PATH work globally as though -isystem was used
for each invocation.

Since that changes the build results, force a rebuild of x86 depends by adding
the value to $HOST_ID_SALT.

Redo of PR #190 via git cherry-pick

This is due to the fact zmq for some unknown reasons
stalls on ubuntu trusty system. Since travis is based
on such Ubunut ver every build conf that exercise the
python rpc tests suite will fail.

zmq test is not failing on ubuntu 16.04 (Xenial)

Release Branch - Fix maxuploadtarget.py so that it works with python3

Release Branch - Fix excessive.py python typeError

[Travis] temporarely disable zmq and py3

Release Branch - Fix txPerf.py for python3

When a block doesn't show up right away the request manager
will wait 5 seconds (on mainnet it is 30 seconds) before
re-requesting a block.  If that should happen during a test
we need to check whether the block was successfully re-requested
and if so mark the test as passed.  Previously we were waiting
for the 5 seconds but then returning with a False if the correct
block showed up when it should be True.

When a blockhash begins with a zero, the zero will be missing when
calling block.sha256.  We need to replace those zero's before
doing our blockhash comparison.

We shave about 70 seconds of run time by taking out the sleep.
Since the comptool issue was resolved, we no longer need the
sleep.

Reanable zmq test on travis

Release Branch - Comptool fix for the requestmanager re-request interval

Release branch - improve perf of bip68 python test

Most of this test runs just fine.  There is one test that
is commented out that requires RBF

Since we don't have included ee06e043 in our tree
ServiceBits is still a plain uint64_t enum, this is
not a plain cherry pick because we had to fix
reference to enum servicebits.

The actual "depends" machinery does not produce bitcoin-qt
while compiling static buids (i.e. `make -C depends
HOST=x86_64-linux-gnu). This is a temporary fix that let us
produce the gitian builds (who depends on the static build)
without wasting time. As soon as we have the release out
of the door we are going to commit a proper fix.

The problem is that Qt binaries (uic, lrelease, /qdbuscpp2xml and
qdbusxml2cpp) are installed under depends/x86_64-linux-gnu/bin
rather than depends/x86_64-linux-gnu/native/bin.

Misc gitian fixes (release branch)

Add support for seeders able to filter by servicebits (release branch)

Release Branch - Fix maxblocksinflight

Release Branch - Renable bip68-sequence.py

testExcessiveTx was not setting the maxblocksize and as a result
was on occasion  hanging after running the testExcessiveBlock.

Also put in a sleep time to give chance for txns to propagate.

Release Branch - fix for random excessive.py hang

fix issue where a block's coinbase can make it exceed the configured val

Update Travis-ci status icon (release branch)

Increase travis time out from 10 to 20 mins using travis_wait
on windows platform.

Add release note for BU 1.0.0

[travis] Backport of BU PR #261 to release branch

Add <h1> title to the doc and fix download URL

Fix a few more typos and URLs

small features in tweaks

[QT] Bugfix: ensure tray icon menu is not empty