Intermittent failure in feature_fee

mzumsande commented at 6:59 pm on April 11, 2022: member

Observed in #24827 (https://cirrus-ci.com/task/6466255107391488) but very likely unrelated. Looks to me that a testmempoolaccept RPC call issued by the mini-wallet timed out - so it seems that it occurs after the #24817 rework but has a root cause in bitcoind?

0 node1 2022-04-11T17:04:58.456117Z [httpworker.3] [rpc/request.cpp:179] [parse] ThreadRPCServer method=testmempoolaccept user=__cookie__ 
1 node1 2022-04-11T17:05:25.635870Z [msghand] [net.cpp:3041] [PushMessage] sending ping (8 bytes) peer=0 
2 node1 2022-04-11T17:06:50.669591Z [scheduler] [random.cpp:520] [SeedPeriodic] Feeding 32422 bytes of dynamic environment data into RNG 
3 node0 2022-04-11T17:06:50.669699Z [net] [net.cpp:1651] [SocketHandlerConnected] socket closed for peer=1 
4 node0 2022-04-11T17:06:50.671075Z [net] [net.cpp:567] [CloseSocketDisconnect] disconnecting peer=1

mzumsande added the label Bug on Apr 11, 2022

MarcoFalke commented at 6:43 am on April 12, 2022: member

I think node2 and node1 got sniped by the OS

Container errored with 'OOMKilled'

mzumsande commented at 9:56 am on April 12, 2022: member

Does that tend to happen randomly, so that it would likely have nothing to do with feature_fee _estimation? Or could the functional test have caused the OOM error?

MarcoFalke commented at 10:07 am on April 12, 2022: member

Yes, it is likely unrelated to a specific test, but the functional tests are still responsible for the OOM.

The last four runs:

https://cirrus-ci.com/task/5923607833477120 https://cirrus-ci.com/task/6107969036222464 https://cirrus-ci.com/task/6231558163005440 https://cirrus-ci.com/task/6674314664607744

The memory usage is (obviously) not deterministic, but never exceeds 8 GB. Though, it can get pretty close to it (7.4).

Not sure how to fix that. Increasing the memory isn’t possible without increasing the CPU as well, which means that CI task scheduling may take longer.

Decreasing the MAKEJOBS for the functional tests may help, but obviously means they run slower.

mzumsande commented at 10:20 am on April 12, 2022: member

ok, makes sense, thanks. If this starts to happens more frequently, it would be best discussed in another issue not related to a specific test. Closed.

mzumsande closed this on Apr 12, 2022

mzumsande commented at 4:17 pm on April 13, 2022: member

I’m seeing this a lot currently: https://cirrus-ci.com/task/4740331357863936 https://cirrus-ci.com/task/6471327732203520 https://cirrus-ci.com/task/5180616776876032

MarcoFalke commented at 5:54 pm on April 14, 2022: member

Maybe we could measure the memory usage of the test before and after the above change?

Though, I suspect that most likely the test changed runtime and thus is now shuffled earlier or later in the timeline with other tests that happen to consume more memory.

MarcoFalke commented at 5:54 pm on April 14, 2022: member

If the runtime is the “cause”, maybe reshuffling the test list a bit could help?

MarcoFalke commented at 11:44 am on April 15, 2022: member

Looks like the run-time doubled after commit 494455f8a56c93cef4162a0cedc792901577314e

MarcoFalke reopened this on Apr 15, 2022

mzumsande commented at 7:14 pm on April 15, 2022: member

Yes, it’s twice as slow for me too. Plus it has 3 parallel nodes. Shuffling the tests might work as a short-term solution. Would it make sense to parallelize (-jobs option of the testrunner) not by restricting the number of functional tests that may run at the same time, but by restricting the number of nodes that may run at the same time?

jnewbery deleted a comment on Apr 22, 2022

MarcoFalke referenced this in commit d24318a40c on May 3, 2022

MarcoFalke closed this on May 3, 2022

sidhujag referenced this in commit 1e4a8f3ce3 on May 4, 2022

DrahtBot locked this on May 3, 2023

Intermittent failure in feature_fee_estimation.py #24828