The young person's guide to the test_framework #10612

pull jnewbery wants to merge 2 commits into bitcoin:master from jnewbery:templatefunctionaltest changing 4 files +441 −107
  1. jnewbery commented at 4:01 PM on June 16, 2017: member

    Variations on a theme of Nakamoto

    • Adds a heavily commented example_test.py test script that can be used as a template for writing new functional tests.
    • Expands test/README.md and test/functional/README.md:
      • test/README.md documents how to run tests locally, as well as giving information about logging and troubleshooting failing test cases
      • test/functional/README.md documents how to write tests, including a style guide and general tips on writing good test cases.

    Addresses #10601

    The style guide is of course completely up for grabs at the moment. This PR contains my first attempt, but I'm very happy for feedback including expanding/changing/reducing the guide.

  2. jimmysong approved
  3. jimmysong commented at 4:07 PM on June 16, 2017: contributor

    ACK

    I wish this documentation existed when I started writing tests! Only thing I would add is maybe some documentation on finding rpc commands that haven't really been covered yet (read: --coverage). Also, would be useful to point people to the actual rpc commands in C++ that can be used to find things to test.

  4. in test/README.md:33 in a001a92dba outdated
      33 | -Running tests locally
      34 | -=====================
      35 | +#### Running the tests
      36 |  
      37 | -Build for your system first. Be sure to enable wallet, utils and daemon when you configure. Tests will not run otherwise.
      38 | +individual tests can be run by directly calling the test script, eg:
    


    practicalswift commented at 9:55 PM on June 16, 2017:

    Nit: s/^individual/Individual/ :-)


    jnewbery commented at 11:35 PM on June 16, 2017:

    Thanks! Fixed

  5. jnewbery force-pushed on Jun 16, 2017
  6. jnewbery commented at 11:37 PM on June 16, 2017: member

    Thanks @jimmysong - added sections on Coverage and RPC and p2p definitions to test/functional/README.md

  7. laanwj commented at 11:11 AM on June 17, 2017: member

    Awesome! Thanks for adding a guide to make tests.

  8. fanquake added the label Docs and Output on Jun 18, 2017
  9. fanquake added the label Tests on Jun 18, 2017
  10. in test/functional/example_test.py:208 in 5a0134a5de outdated
     197 | +            getdata_request.inv.append(CInv(2, block))
     198 | +            node2.send_message(getdata_request)
     199 | +
     200 | +        # wait_until() will loop until a predicate condition is met. Use it to test properties of the
     201 | +        # ConnNodeCB objects.
     202 | +        assert wait_until(lambda: sorted(blocks) == sorted(list(node2.block_receive_map.keys())), timeout=5)
    


    MarcoFalke commented at 2:40 PM on June 18, 2017:

    I'd prefer if the assert was moved into wait_until and the return statement removed from wait_until.

    The places in the code where it is missing right now are likely oversights and should be fixed.


    jnewbery commented at 3:39 PM on June 18, 2017:

    Indeed. I'd also like to see an assert within wait_until(). I've previously tried adding an assert into that function and it causes several tests to fail.

    Fixing those cases where wait_until() isn't actually returning True is a job for another PR. When that happens, a scripted diff to s/assert wait_until/wait_until/ should be run, which will update this example test script.


    sdaftuar commented at 6:11 PM on June 22, 2017:

    I think it'd be nice to mention the locking issues with reaching into a NodeConnCB from the main thread. We could mention that wait_until() grabs the global lock before checking the predicate, or perhaps better still would be to add an example of acquiring the lock in this example before inspecting an object that is accessed from the other thread.


    jnewbery commented at 9:07 AM on June 27, 2017:

    Good idea. Added an example of accessing the NodeConnCB data using the mininode_lock

  11. in test/functional/example_test.py:158 in 5a0134a5de outdated
     147 | +        # node object. Instead there's some __getattr__() magic going on under
     148 | +        # the covers to dispatch unrecognised attribute calls to the RPC
     149 | +        # interface.
     150 | +
     151 | +        # Logs are nice. Do plenty of them. They can be used in place of comments for
     152 | +        # breaking the test into sub-sections.
    


    MarcoFalke commented at 2:40 PM on June 18, 2017:

    👍

  12. in test/README.md:122 in 5a0134a5de outdated
     154 | +
     155 | +- when run through the test_runner harness, *all* logs are written to
     156 | +  `test_framework.log` and no logs are output to the console.
     157 | +- when run directly, *all* logs are written to `test_framework.log` and INFO
     158 | +  level and above are output to the console.
     159 | +- when run on Travis, no logs are output to the console. However, if a test
    


    MarcoFalke commented at 2:45 PM on June 18, 2017:

    Since travis supports color, could we run combine_logs.py on travis as well?


    jnewbery commented at 3:41 PM on June 18, 2017:

    Potentially, yes. combine_logs.py was written to be portable (and even usable on machines where test_framework is not installed).

    Again, I think this is a job for another PR.

  13. MarcoFalke commented at 2:47 PM on June 18, 2017: member

    Concept ACK! Just some meta comments.

  14. Christewart commented at 2:55 PM on June 18, 2017: member

    Awesome! Concept ACK x2

  15. jnewbery commented at 3:43 PM on June 18, 2017: member

    @laanwj @Marcofalke @christewart - thanks for the feedback.

    Any comments on the suggested style guide? I was expecting that to be the area where people would have opinions.

  16. in test/functional/README.md:71 in 5a0134a5de outdated
      99 | +a callback class that derives from `NodeConnCB` and pass that to the
     100 | +`NodeConn` object, your code will receive the appropriate callbacks when
     101 |  events of interest arrive.
     102 |  
     103 | -* You can pass the same handler to multiple ```NodeConn```'s if you like, or pass
     104 | +- You can pass the same handler to multiple `NodeConn`'s if you like, or pass
    


    sdaftuar commented at 3:40 PM on June 20, 2017:

    This is true, but no one does this anymore I think, and we should probably discourage that style?


    jnewbery commented at 9:06 AM on June 27, 2017:

    Yes, you're right. I've removed this line.

  17. in test/functional/example_test.py:109 in 5a0134a5de outdated
     104 | +        self.setup_nodes()
     105 | +
     106 | +        # In this test, we're not connecting node2 to node0 or node1. Calls to
     107 | +        # sync_all() should not include node2, since we're not expecting it to
     108 | +        # sync.
     109 | +        connect_nodes_bi(self.nodes, 0, 1)
    


    sdaftuar commented at 4:14 PM on June 20, 2017:

    Should we try to discourage connect_nodes_bi?


    jnewbery commented at 9:06 AM on June 27, 2017:

    Yes - removed from the example test.

  18. in test/functional/example_test.py:147 in 5a0134a5de outdated
     137 | +        NetworkThread().start()
     138 | +        # wait_for_verack ensures that the p2p connection is fully up.
     139 | +        node0.wait_for_verack()
     140 | +
     141 | +        # Generating a block on one of the nodes will get us out of IBD
     142 | +        blocks = [int(self.nodes[0].generate(nblocks=1)[0], 16)]
    


    sdaftuar commented at 4:15 PM on June 20, 2017:

    Should sync the (connected) nodes after this call, I think?

  19. in test/README.md:135 in 5a0134a5de outdated
     171 | +combine_logs.py -c <test data directory> | less -r
     172 | +```
     173 | +
     174 | +will pipe the colorized logs from the test into less.
     175 | +
     176 | +Use --tracerpc to trace out all the RPC calls and responses to the console. For
    


    achow101 commented at 1:45 AM on June 22, 2017:

    --tracerpc should be surrounded by backticks.


    jnewbery commented at 9:05 AM on June 27, 2017:

    thanks. Done

  20. achow101 approved
  21. achow101 commented at 1:51 AM on June 22, 2017: member

    ACK 5a0134a5deb09103e4ba70ebf519c26c1557985b

  22. in test/functional/README.md:54 in 5a0134a5de outdated
      73 |  
      74 | -P2P test design notes
      75 | ----------------------
      76 | +- `/src/rpc/*` for RPCs
      77 | +- `/src/wallet/rpc*` for wallet RPCs
      78 | +- `ProcessMessage()` in `/sfrc/net_processing.cpp` for parsing p2p messages
    


    MarcoFalke commented at 10:58 AM on June 22, 2017:

    nit: sfrc/src


    jnewbery commented at 9:05 AM on June 27, 2017:

    fixed

  23. in test/functional/test_runner.py:156 in 5a0134a5de outdated
     152 | @@ -153,6 +153,7 @@
     153 |      # These are python files that live in the functional tests directory, but are not test scripts.
     154 |      "combine_logs.py",
     155 |      "create_cache.py",
     156 | +    "example_test.py",
    


    MarcoFalke commented at 11:00 AM on June 22, 2017:

    Can we put this in the extended scripts, just to make sure it has correct syntax?


    jnewbery commented at 9:06 AM on June 27, 2017:

    Yes, good idea. example_test should be run in the extended test run to test the test.

  24. MarcoFalke commented at 11:03 AM on June 22, 2017: member

    utACK 5a0134a, one nit.

  25. in test/functional/example_test.py:201 in 5a0134a5de outdated
     196 | +            getdata_request = msg_getdata()
     197 | +            getdata_request.inv.append(CInv(2, block))
     198 | +            node2.send_message(getdata_request)
     199 | +
     200 | +        # wait_until() will loop until a predicate condition is met. Use it to test properties of the
     201 | +        # ConnNodeCB objects.
    


    sdaftuar commented at 6:07 PM on June 22, 2017:

    NodeConnCB, not ConnNodeCB


    jnewbery commented at 9:07 AM on June 27, 2017:

    thanks! Fixed.

  26. sdaftuar approved
  27. sdaftuar commented at 6:13 PM on June 22, 2017: member

    Thanks for doing this! Left some nits for you, but looks good to me.

  28. in test/README.md:73 in 5a0134a5de outdated
      96 | -                        (default: ../../src)
      97 | -  --tmpdir=TMPDIR       Root directory for datadirs
      98 | -  --tracerpc            Print out all RPC calls as they are made
      99 | -  --coveragedir=COVERAGEDIR
     100 | -                        Write tested RPC commands into this directory
     101 | +The p2p and RPC ports used by the bitcoind nodes-under-test are chosen to make
    


    laanwj commented at 8:38 AM on June 23, 2017:

    Nit: P2P


    jnewbery commented at 9:07 AM on June 27, 2017:

    fixed

  29. in test/README.md:89 in 5a0134a5de outdated
     109 | +bitcoind process running when the tests are started.
     110 | +
     111 | +If there are zombie bitcoind processes after test failure, you can kill them by running:
     112 | +
     113 | +```bash
     114 | +killall bitcoind
    


    laanwj commented at 8:40 AM on June 23, 2017:

    This could potentially be risky if any non-test bitcoinds are running as the same user. Not sure it warrants a warning or not.


    jnewbery commented at 9:07 AM on June 27, 2017:

    Added a warning

  30. in test/README.md:152 in 5a0134a5de outdated
     184 | +##### Attaching a debugger
     185 | +
     186 | +A python debugger can be attached to tests at any point. Just add the line:
     187 | +
     188 | +```py
     189 | +import pdb; pdb.set_trace()
    


    laanwj commented at 8:48 AM on June 23, 2017:

    Nice one to mention. After ~15 years of using Python I still forgot about this all the time when I need it.


    jnewbery commented at 9:08 AM on June 27, 2017:

    🙂 hooray for pdb!


    instagibbs commented at 1:24 PM on June 27, 2017:

    how is this such a secret!

    Another pro-tip that might be worthwhile is if you need to introspect bitcoind itself during a test you set a pdb break, then gdb path/to/bitcoind <pid> to attach to it.

    Sorry I never saw this PR while it was live :/


    jnewbery commented at 1:54 PM on June 27, 2017:

    Good tip @instagibbs . I'll happily ACK a PR that adds that to test/README.md

  31. MarcoFalke commented at 8:50 AM on June 23, 2017: member

    ​Another nit I forgot to mention earlier: ​ ​ ​Since you remove the mention of PYTHON_DEBUG, you might as well remove it in the code as well and replace it by os.getenv('TRAVIS') == 'true'

  32. [tests] Update functional tests documentation 76859e6a76
  33. [tests] add example test e7ba6c16b3
  34. in test/functional/README.md:47 in 5a0134a5de outdated
      64 | +  arguments instead of positional arguments to make the intent of the call
      65 | +  clear to readers.
      66 |  
      67 | -### [test_framework/blocktools.py](test_framework/blocktools.py)
      68 | -Helper functions for creating blocks and transactions.
      69 | +#### RPC and p2p definitions
    


    laanwj commented at 9:05 AM on June 23, 2017:

    also here: P2P uppercase please


    jnewbery commented at 9:08 AM on June 27, 2017:

    fixed

  35. jnewbery force-pushed on Jun 27, 2017
  36. jnewbery commented at 9:09 AM on June 27, 2017: member

    Since you remove the mention of PYTHON_DEBUG, you might as well remove it in the code as well and replace it by os.getenv('TRAVIS') == 'true'

    Absolutely agree, although this should be done in a separate PR.

  37. jnewbery commented at 9:09 AM on June 27, 2017: member

    Nits fixed

  38. laanwj commented at 10:07 AM on June 27, 2017: member

    ACK e7ba6c1

  39. laanwj merged this on Jun 27, 2017
  40. laanwj closed this on Jun 27, 2017

  41. laanwj referenced this in commit 7c87a9c748 on Jun 27, 2017
  42. jnewbery deleted the branch on Jun 27, 2017
  43. PastaPastaPasta referenced this in commit 809e67ff79 on Jul 6, 2019
  44. PastaPastaPasta referenced this in commit d094a5695c on Jul 6, 2019
  45. PastaPastaPasta referenced this in commit c56b2bf8ac on Jul 6, 2019
  46. PastaPastaPasta referenced this in commit b02e697865 on Jul 8, 2019
  47. PastaPastaPasta referenced this in commit 46193cf9fa on Jul 9, 2019
  48. PastaPastaPasta referenced this in commit 2708a7be06 on Jul 11, 2019
  49. barrystyle referenced this in commit 67ba69edbc on Jan 22, 2020
  50. MarcoFalke locked this on Sep 8, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-14 15:15 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me