test: make feature_init more robust #23737

1. 
   jamesob commented at 7:39 pm on December 10, 2021: member

   Hopefully fixes #23646.

   This makes a few changes to feature_init (along the lines of the discussion here: #23289 (review)). The changes are detailed in the specific commits, but a summary might be:

   • more robust log-line detection (at the expense of maybe being slower)
   • retain all debug.log content
   • perturb .ldb files in a more complete way
2. jamesob force-pushed on Dec 10, 2021
3. DrahtBot added the label Tests on Dec 10, 2021
4. jamesob force-pushed on Dec 10, 2021
5. 
   MarcoFalke commented at 12:41 pm on December 13, 2021: member

   Could rebase over commit bf66e258a84e18935fde3ebb9a4b0392bf883222?
6. jamesob force-pushed on Dec 13, 2021
7. 
   jamesob commented at 1:22 pm on December 13, 2021: member

   Could rebase over commit bf66e25?

   Rebased and re-enabled the txindex portion of the test.

8. 
   MarcoFalke commented at 4:12 pm on December 15, 2021: member

   Thanks, concept ACK
9. 
   DrahtBot commented at 1:19 am on December 16, 2021: member

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #10102 (Multiprocess bitcoin by ryanofsky)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

10. DrahtBot added the label Needs rebase on Dec 16, 2021
11. 
    MarcoFalke commented at 5:07 pm on December 29, 2021: member

    Are you still working on this? If not, that is fine, just let us know, so that it can be picked up.
12. 
    jamesob commented at 5:57 pm on December 29, 2021: member

    Oh whoops, didn’t see the rebase label.
13. test: introduce TestNode.debug_log_bytes a2fb62b632
14. test: add TestNode.wait_for_debug_log a8ffbc01db
15. test: feature_init: tweak all .ldb files of a certain type

    This part of the test sporadically fails on CI infrastructure. Instead
    of perturbing a single .ldb file of each type, move all .ldb files of a
    given type to ensure a bad startup.

    24fcf6e435
16. test: feature_init: retain debug.log and improve detection

    This test sporadically fails due to the Python test missing log lines
    for reasons that are poorly understood. The problem is made worse by the
    fact that this test does not retain the log files from iteration to
    iteration.
    
    Change the test to do logline detection in a more robust manner (by
    using `re.search` on the whole log content) in a way that is comparable
    to the existing `assert_debug_log` utility, and retain all debug.log
    content from case to case.

    93db6d8422
17. test: enable txindex in feature_init

    Now that #23365 is merged.

    8904f17ea7
18. jamesob force-pushed on Dec 29, 2021
19. DrahtBot removed the label Needs rebase on Dec 29, 2021
20. 
    MarcoFalke commented at 7:52 am on January 3, 2022: member

    Merging this to unbreak the CI. I am resetting too many CI build manually every day
21. MarcoFalke merged this on Jan 3, 2022
22. MarcoFalke closed this on Jan 3, 2022

    ---

23. sidhujag referenced this in commit 895dd37712 on Jan 4, 2022
24. in test/functional/feature_init.py:104 in 8904f17ea7

    103@@ -132,7 +104,8 @@ def check_clean_start():
    104                 if line:
    

    ---

    ---

    

    MarcoFalke commented at 2:15 pm on January 6, 2022:

    0                                       line = logfile.readline()
    1                                     File "/usr/lib/python3.8/codecs.py", line 322, in decode
    2                                       (result, consumed) = self._buffer_decode(data, self.errors, final)
    3                                   UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data
    

    https://cirrus-ci.com/task/6633293054476288?logs=ci#L5910

25. Fabcien referenced this in commit 8dffcaf381 on Nov 25, 2022
26. Fabcien referenced this in commit b8b10db652 on Nov 25, 2022
27. Fabcien referenced this in commit 7a31619610 on Nov 25, 2022
28. DrahtBot locked this on Jan 6, 2023

Contributors
jamesob MarcoFalke DrahtBot

Labels
Tests