There is a persistently high amount of intermittent test failures, which are a burden to development and slowing everything down. I suggest to adapt the well established Volkswagen-technique to counter those annoyances.
The implementation is quite straightforward: Tests, to an outside observer, will appear to be running as designed. They emit logs to please the observer and, as desired, will always exit successfully. The tested bitcoind itself is not modified and thus does not change behaviour.
🏎 💨
Can you please update the linters to return success as well? Thx
Good catch! Updated linters as well.
Yeah. Makes sense. utACK fac32125a3779529216257961127ad8c247b243b 🤥
This is perfect, it starts off a complete new era of testing, thank you!
Concept ACK. :100:
NACK regarding the linter change.
Rationale:
Formal proof:
# Some Z3 intricacies.
>>> import z3
>>> s = z3.Solver()
>>> t = z3.DeclareSort("t")
>>> is_test = z3.Function("is_test", t, z3.BoolSort())
>>> is_important = z3.Function("is_important", t, z3.BoolSort())
>>> linter = z3.Const("linter", t)
>>> _ = z3.Const("_", t)
# Axiom: Tests are important.
>>> s.add(z3.ForAll([_], z3.Implies(is_test(_), is_important(_))))
# Axiom: A linter is a test.
>>> s.add(is_test(linter))
>>> s.check()
sat
# Try to claim that linters are not important.
>>> s.add(z3.Not(is_important(linter)))
>>> s.check()
unsat
@practicalswift thanks for the proof, very clear However it does not conflict with the intent of this PR, we think linters are very important that's why they should always be happy and return success!
@laanwj You have a very good point! I'm happy if and only if the linters are happy. Concept ACK!
I think the PR title is a bit misleading. To fully conform to the Volkswagen methodolgy, the code should detect CI platforms and only only kick in then
its a good start, but can you figure out some way that this could also be used to gas people? you know, to prove it works or whatever.
This makes me deeply sad now, closing