test: wait_until() failure in p2p_permissions.py #22714

Seen in https://cirrus-ci.com/task/6730209715027968:

Remaining jobs: [p2p_node_network_limited.py, p2p_permissions.py]                                                                                                      
Remaining jobs: [p2p_permissions.py]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
217/217 - p2p_permissions.py failed, Duration: 2403 s
stdout:
2021-08-16T05:03:11.839000Z TestFramework (INFO): Initializing test directory /tmp/cirrus-ci-build/ci/scratch/test_runner/test_runner_₿_🏃_20210816_044948/p2p_permissions_13
2021-08-16T05:43:14.597000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: ''''
            self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
'''
2021-08-16T05:43:14.598000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
    self.run_test()
  File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/p2p_permissions.py", line 34, in run_test
    self.check_tx_relay()
  File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/p2p_permissions.py", line 127, in check_tx_relay
    self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
  File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 672, in wait_until
    return wait_until_helper(test_function, timeout=timeout, timeout_factor=self.options.timeout_factor)
  File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/util.py", line 256, in wait_until_helper
    raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
AssertionError: Predicate ''''
            self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
''' not true after 2400.0 seconds
2021-08-16T05:43:14.649000Z TestFramework (INFO): Stopping nodes

ran this a few times locally and it passed, so it looks like the mempool sync is taking extra long? i replaced self.wait_until locally with a call to self.sync_mempools (as that seems to be the actual purpose of the wait until?). ran 10 times without any failures.

looking at sync_mempools, tho, it would still be at risk of a timeout error: https://github.com/bitcoin/bitcoin/blob/b935abb9eb2eeaceb5621e110e1889c2ce1b7734/test/functional/test_framework/test_framework.py#L662-L665

but i think the error message and the use of sync_mempools makes it more clear in the logs what exactly went wrong and makes the intent of the test more clear. happy to open a PR for that if it would help, but perhaps there is a better solution that would completely eliminate the risk of a timeout?

I did also run the test, p2p_permission.py, ran a couple of times. It passed through.

Also can't reproduce locally, the test passes as expected.

Not sure how related, but I came across a similar problem on this same 32-bit + dash [gui] [CentOS 8] CI. See these for more context #22543#pullrequestreview-730761475, #22543 (comment), and #22543#pullrequestreview-730921885

Also can't reproduce locally, the test passes as expected.

Not sure how related, but I came across a similar problem on this same 32-bit + dash [gui] [CentOS 8] CI. See these for more context #22543 (review), #22543 (comment), and #22543 (review)

I tried the test on Mac M1 with 11.5.2

https://cirrus-ci.com/task/4566361711050752

 test  2022-06-17T10:02:22.504000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: '''' 
                                               self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
                                   '''
 test  2022-06-17T10:02:22.504000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-build-449516745/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 133, in main
                                       self.run_test()
                                     File "/tmp/cirrus-build-449516745/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_permissions.py", line 34, in run_test
                                       self.check_tx_relay()
                                     File "/tmp/cirrus-build-449516745/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_permissions.py", line 126, in check_tx_relay
                                       self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
                                     File "/tmp/cirrus-build-449516745/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 715, in wait_until
                                       return wait_until_helper(test_function, timeout=timeout, timeout_factor=self.options.timeout_factor)
                                     File "/tmp/cirrus-build-449516745/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 276, in wait_until_helper
                                       raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
                                   AssertionError: Predicate ''''
                                               self.wait_until(lambda: txid in self.nodes[0].getrawmempool())
                                   ''' not true after 2400.0 seconds
 test  2022-06-17T10:02:22.507000Z TestFramework (DEBUG): Closing down network thread 

I saw the same in https://cirrus-ci.com/task/6553923304554496