tests: add utility to easily profile node performance with perf #14519

jamesob commented at 4:49 PM on October 19, 2018: member

Adds a context manager to easily (and selectively) profile node performance during functional test execution using perf.

While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (TestNode.profile_with_perf) that generates performance diagnostics for a node by running perf during the execution of a particular region of test code.

perf usage is detailed in the excellent (and sadly unmerged) #12649; all due props to @eklitzke.

Example

with node.profile_with_perf("large-msgs"):
    for i in range(200):
        node.p2p.send_message(some_large_msg)
    node.p2p.sync_with_ping()

This generates a perf data file in the test node's datadir (/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data).

Running perf report generates nice output about where the node spent most of its time while running that part of the test:

$ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \
  | c++filt \
  | less

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 135  of event 'cycles:pp'
# Event count (approx.): 1458205679493582
#
# Children      Self  Command          Shared Object        Symbol
# ........  ........  ...............  ...................  ........................................................................................................................................................................................................................................................................
#
    70.14%     0.00%  bitcoin-net      bitcoind             [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)                                                                                                                                                                                                                        
                |
                ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

    70.14%     0.00%  bitcoin-net      bitcoind             [.] CNetMessage::readData(char const*, unsigned int)                                                                                                                                                                                                                                
                |
                ---CNetMessage::readData(char const*, unsigned int)
                   CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

    35.52%     0.00%  bitcoin-net      bitcoind             [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)                                                                                      
                |
                ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
                   CNetMessage::readData(char const*, unsigned int)
                   CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

...

jamesob renamed this:
~~test: add utility to easily profile node performance with perf~~
tests: add utility to easily profile node performance with perf
on Oct 19, 2018

jamesob renamed this:
~~tests: add utility to easily profile node performance with perf~~
test: add utility to easily profile node performance with perf
on Oct 19, 2018

jamesob force-pushed on Oct 19, 2018

jamesob renamed this:
~~test: add utility to easily profile node performance with perf~~
tests: add utility to easily profile node performance with perf
on Oct 19, 2018

jamesob commented at 6:10 PM on October 19, 2018: member

An example of running the generated perf data file through hotspot:

selection_109

in test/functional/test_framework/test_node.py:322 in b93a17c315 outdated

 317 | +
 318 | +            cmd = [
 319 | +                'perf', 'record',
 320 | +                '-g',
 321 | +                '--call-graph', 'dwarf',
 322 | +                '-F', '101',

ryanofsky commented at 6:15 PM on October 19, 2018:

Maybe comment # Frequency in Hz

in test/functional/test_framework/test_node.py:307 in b93a17c315 outdated

 302 | +        def test_success(cmd):
 303 | +            return subprocess.run(
 304 | +                cmd, shell=True,
 305 | +                stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL).returncode == 0
 306 | +
 307 | +        if not test_success('readelf -S {} | grep .debug_str'.format(self.binary)):

ryanofsky commented at 6:22 PM on October 19, 2018:

Escaping self.binary with shlex.quote would let it work with any filename.

ryanofsky approved

ryanofsky commented at 6:32 PM on October 19, 2018: member

utACK b93a17c3153f050eb37292b51463b0aaa5860f1a. Context manager seems nice for writing new tests. This is just an idea, but maybe it would be useful if there were an option to collect perf data for all nodes from start to stop to gain insight into performance of existing tests.

jamesob force-pushed on Oct 19, 2018

jamesob commented at 8:38 PM on October 19, 2018: member

Thanks for the review, @ryanofsky. I really like your suggestions and so I've incorporated them all into a rebase.

You can now pass --perf when running an individual test file and all nodes will be profiled for the duration of the test. Here's an example run:

$ ./test/functional/p2p_unrequested_blocks.py --perf

2018-10-19T20:36:06.542000Z TestFramework (INFO): Initializing test directory /tmp/test71_cfvsc
...
2018-10-19T20:36:09.206000Z TestFramework (INFO): Stopping nodes
2018-10-19T20:36:09.384000Z TestFramework.node0 (INFO): Wrote perf output to '/tmp/test71_cfvsc/node0/node-0-AcceptBlockTest.perf.data'
2018-10-19T20:36:09.558000Z TestFramework.node1 (INFO): Wrote perf output to '/tmp/test71_cfvsc/node1/node-1-AcceptBlockTest.perf.data'
2018-10-19T20:36:09.608000Z TestFramework (WARNING): Not cleaning up dir /tmp/test71_cfvsc due to perf data
2018-10-19T20:36:09.608000Z TestFramework (INFO): Tests successful

jamesob force-pushed on Oct 19, 2018

promag commented at 8:28 AM on October 20, 2018: member

Concept ACK, very nice!

DrahtBot commented at 9:50 AM on October 20, 2018: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

jtimon commented at 4:40 PM on October 22, 2018: contributor

Concept ACK

in test/functional/test_framework/test_node.py:66 in 358e757d56 outdated

  59 | @@ -59,7 +60,15 @@ class TestNode():
  60 |      To make things easier for the test writer, any unrecognised messages will
  61 |      be dispatched to the RPC connection."""
  62 |  
  63 | -    def __init__(self, i, datadir, *, rpchost, timewait, bitcoind, bitcoin_cli, mocktime, coverage_dir, extra_conf=None, extra_args=None, use_cli=False):
  64 | +    def __init__(self, i, datadir, *, rpchost, timewait, bitcoind, bitcoin_cli, mocktime, coverage_dir, extra_conf=None, extra_args=None, use_cli=False, start_perf=False, test_class_name=None):
  65 | +        """
  66 | +        Kwargs:
  67 | +            test_class_name (str): The name of the test class which has spawned this node.

ryanofsky commented at 6:09 PM on October 22, 2018:

I don't think I see advantage of the test_class_name plumbing. It seems like it makes resulting filenames less predictable I run a test with --perf and the resulting filenames have a python class name in the middle. It also doesn't seem like it helps with disambiguation either if every node has the same test name.

jamesob commented at 7:26 PM on October 22, 2018:

Yeah, that's a good point. I thought we needed to do this in order to maintain compatibility with test/functional/test_runner.py (to avoid perf data files from getting clobbered), but I forgot that each test gets its own nested tmp directory. Will do.

ryanofsky approved

ryanofsky commented at 6:15 PM on October 22, 2018: member

utACK 358e757d566d5f659901acc6f15fa9eb6eb3e741

fanquake added the label Tests on Oct 22, 2018

in test/functional/test_framework/test_node.py:332 in 358e757d56 outdated

 330 | +        Returns the subprocess running perf.
 331 | +        """
 332 | +        subp = None
 333 | +
 334 | +        def test_success(cmd):
 335 | +            return subprocess.run(

conscott commented at 6:25 AM on October 23, 2018:

Might want to add a try/except here with CalledProcessError

jamesob force-pushed on Oct 23, 2018

jamesob commented at 3:28 PM on October 23, 2018: member

Addressed feedback from @ryanofsky @conscott.

in test/functional/test_framework/test_node.py:368 in dd22df8506 outdated

 367 | +        self.log.info("Wrote perf output to '{}'".format(
 368 | +            self._get_perf_data_path(profile_name)))
 369 | +
 370 | +    def _get_perf_data_path(self, profile_name):
 371 | +        """Return a suitable path for perf data given some descriptive name."""
 372 | +        suffix = "-{}".format(profile_name) if profile_name else ""

MarcoFalke commented at 11:28 AM on October 24, 2018:

profile_name is always given, so no need for the else?

jamesob commented at 7:07 PM on October 24, 2018:

It's an empty string when we specify --perf to the runner: https://github.com/bitcoin/bitcoin/blob/358e757d566d5f659901acc6f15fa9eb6eb3e741/test/functional/test_framework/test_node.py#L184

MarcoFalke commented at 12:10 PM on October 26, 2018:

Wouldn't that overwrite the perf file on restart?

jamesob commented at 2:46 PM on October 30, 2018:

Ah, good catch. Will verify & fix.

in test/functional/test_framework/test_node.py:333 in dd22df8506 outdated

 328 | +        """
 329 | +        subp = None
 330 | +
 331 | +        def test_success(cmd):
 332 | +            try:
 333 | +                return subprocess.run(

MarcoFalke commented at 11:31 AM on October 24, 2018:

We don't support this function yet: https://docs.python.org/3.4/library/subprocess.html#subprocess.run

MarcoFalke approved

MarcoFalke commented at 11:34 AM on October 24, 2018: member

utACK

in test/functional/test_framework/test_framework.py:215 in dd22df8506 outdated

 211 | +        )
 212 | +        if should_clean_up:
 213 |              self.log.info("Cleaning up {} on exit".format(self.options.tmpdir))
 214 |              cleanup_tree_on_exit = True
 215 | +        elif self.options.perf:
 216 | +            self.log.warning("Not cleaning up dir %s due to perf data" % self.options.tmpdir)

MarcoFalke commented at 11:36 AM on October 24, 2018:

nit: Could use .format(... like two lines above

jamesob force-pushed on Oct 24, 2018

jamesob commented at 7:12 PM on October 24, 2018: member

Pushed feedback from @MarcoFalke.

in test/functional/test_framework/test_framework.py:215 in ac6a6b3ab5 outdated

 211 | +        )
 212 | +        if should_clean_up:
 213 |              self.log.info("Cleaning up {} on exit".format(self.options.tmpdir))
 214 |              cleanup_tree_on_exit = True
 215 | +        elif self.options.perf:
 216 | +            self.log.warning("Not cleaning up dir %s due to perf data".format(self.options.tmpdir))

MarcoFalke commented at 7:13 PM on October 24, 2018:

needs to use {} in the format string ;)

jamesob commented at 7:14 PM on October 24, 2018:

Oops! Thanks :)

jamesob force-pushed on Oct 24, 2018

ryanofsky approved

ryanofsky commented at 3:59 PM on October 25, 2018: member

utACK 5d8ac69be53a876d2b15ba703603cea8f4efdb1a. Just change in string formatting and simplification to file naming since last review.

fanquake commented at 9:26 AM on October 28, 2018: member

utACK 5d8ac69

Can't comment on the usage of perf, but if this leads to more profiling, and performance related discussion/results in reviews, sounds great.

Also pro reviving the flamegraph docs out of #12649, however that shouldn't hold this up.

in test/functional/test_framework/test_node.py:201 in 5d8ac69be5 outdated

 176 | @@ -167,6 +177,9 @@ def start(self, extra_args=None, *, stdout=None, stderr=None, **kwargs):
 177 |          self.running = True
 178 |          self.log.debug("bitcoind started, waiting for RPC to come up")
 179 |  
 180 | +        if self.start_perf:
 181 | +            self._start_perf("")

MarcoFalke commented at 9:20 PM on October 29, 2018:

That will overwrite previous perf results.

jamesob commented at 6:42 PM on November 9, 2018:

Fixed and tested by running ./test/functional/p2p_disconnect_ban.py --perf (which restarts node1):

 $ ./test/functional/p2p_disconnect_ban.py --perf
2018-11-09T18:41:23.959000Z TestFramework (INFO): Initializing test directory /tmp/testk5t2fuyc
[...]
2018-11-09T18:41:25.250000Z TestFramework (INFO): setban: test persistence across node restart
2018-11-09T18:41:25.616000Z TestFramework.node1 (INFO): Wrote perf output to '/tmp/testk5t2fuyc/node1/perf.data'
[...]
2018-11-09T18:41:26.438000Z TestFramework (INFO): Stopping nodes
2018-11-09T18:41:26.756000Z TestFramework.node0 (INFO): Wrote perf output to '/tmp/testk5t2fuyc/node0/perf.data'
2018-11-09T18:41:27.040000Z TestFramework.node1 (INFO): Wrote perf output to '/tmp/testk5t2fuyc/node1/1.perf.data'
2018-11-09T18:41:27.041000Z TestFramework (WARNING): Not cleaning up dir /tmp/testk5t2fuyc due to perf data
2018-11-09T18:41:27.041000Z TestFramework (INFO): Tests successful

DrahtBot added the label Needs rebase on Nov 6, 2018

jamesob force-pushed on Nov 9, 2018

DrahtBot removed the label Needs rebase on Nov 9, 2018

jamesob force-pushed on Nov 9, 2018

in test/functional/test_framework/test_node.py:342 in a94c68b3f4 outdated

 337 | +    def profile_with_perf(self, profile_name):
 338 | +        """
 339 | +        Context manager that allows easy profiling of node activity using `perf`.
 340 | +
 341 | +        Perf will sample the running node and will generate profile data in the node's
 342 | +        datadir. The profile data can then be presented using `perf report` or a graphical

jnewbery commented at 11:15 PM on November 21, 2018:

Since this is now a user activated feature (with --perf, I think most of these usage notes should go in /test/README/md)

jnewbery commented at 11:16 PM on November 21, 2018: member

Concept ACK.

This seems cool. I haven't had a chance to play around with it yet. I've given the code a cursory glance and will review more thoroughly next week.

in test/functional/test_framework/test_node.py:108 in a94c68b3f4 outdated

 102 | @@ -95,6 +103,8 @@ def __init__(self, i, datadir, *, rpchost, timewait, bitcoind, bitcoin_cli, mock
 103 |          self.url = None
 104 |          self.log = logging.getLogger('TestFramework.node%d' % i)
 105 |          self.cleanup_on_exit = True # Whether to kill the node when this object goes away
 106 | +        # Cache perf subprocesses here by their data output filename.
 107 | +        self.perf_subprocesses = {}

jnewbery commented at 9:45 PM on November 30, 2018:

Are you ever expecting to have more than one perf process profiling the node? That seems like it's probably unnecessary.

jamesob commented at 3:28 PM on January 2, 2019:

It's conceivable if you want to do nested benchmarking, or have specific regions benchmarked under different perf output files. There's very little overhead here to keeping the subprocess references around, so I can't think of a good reason to rejigger this.

jnewbery commented at 10:43 PM on January 7, 2019:

My point is that there should only ever be one perf_subprocess at any time (it gets popped in _stop_perf). I think it's fine to have a perf_subrocess reference in the TestNode object, but making it a dictionary when there can only be one reference at a time seems odd.

Or maybe I'm misunderstanding. Is it possible to attach more than one perf instance to the same process? Can you give an example of when you'd want that?

jamesob commented at 1:34 PM on January 22, 2019:

My point is that there should only ever be one perf_subprocess at any time

Yeah, this isn't the case: you might have nested context managers, or a context manager used in conjunction with --perf.

in test/functional/test_framework/test_node.py:391 in a94c68b3f4 outdated

 386 | +        if not test_success('readelf -S {} | grep .debug_str'.format(shlex.quote(self.binary))):
 387 | +            self.log.warning(
 388 | +                "perf output won't be very useful without debug symbols compiled into bitcoind")
 389 | +
 390 | +        if not test_success('which perf'):
 391 | +            self.log.warning("Can't profile with perf; must install perf-tools")

jnewbery commented at 9:47 PM on November 30, 2018:

This might just be personal taste, but I'd prefer to have a return None in this if block, so that the main body of the function isn't indented in an else block.

in test/functional/test_framework/test_node.py:376 in a94c68b3f4 outdated

 381 | +            return subprocess.call(
 382 | +                # shell=True required for pipe use below
 383 | +                cmd, shell=True,
 384 | +                stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL) == 0
 385 | +
 386 | +        if not test_success('readelf -S {} | grep .debug_str'.format(shlex.quote(self.binary))):

jnewbery commented at 9:50 PM on November 30, 2018:

This fails for macOS. Do you want to make this portable or are you just targetting linux? You could make it work on macOS by installing binutils and using greadelf.

jamesob commented at 3:31 PM on January 2, 2019:

Perf isn't available for macOS since it's Linux-specific. I'll add a warning and early-exit for non-Linux systems.

in test/functional/test_framework/test_node.py:414 in a94c68b3f4 outdated

 409 | +        """Stop (and pop) a perf subprocess."""
 410 | +        subp = self.perf_subprocesses.pop(profile_name)
 411 | +        output_path = subp.args[subp.args.index('-o') + 1]
 412 | +
 413 | +        subp.terminate()
 414 | +        subp.wait()

jnewbery commented at 9:53 PM on November 30, 2018:

Should this have a timeout so the test doesn't hang if the subprocess doesn't terminate?

in test/functional/test_framework/test_node.py:410 in a94c68b3f4 outdated

 414 | +        subp.wait()
 415 | +
 416 | +        stderr = subp.stderr.read().decode()
 417 | +        if 'Consider tweaking /proc/sys/kernel/perf_event_paranoid' in stderr:
 418 | +            self.log.warning(
 419 | +                "perf couldn't collect data! Try "

jnewbery commented at 9:54 PM on November 30, 2018:

This was very helpful!

in test/functional/test_framework/test_node.py:424 in a94c68b3f4 outdated

 419 | +                "perf couldn't collect data! Try "
 420 | +                "'sudo sysctl -w kernel.perf_event_paranoid=-1'")
 421 | +        else:
 422 | +            self.log.info("Wrote perf output to '{}'".format(output_path))
 423 | +
 424 | +    def _get_unique_perf_data_path(self, profile_name):

jnewbery commented at 9:55 PM on November 30, 2018:

Can you replace all this with a call to tempfile.NamedTemporaryFile? See above in the same file for an example:

            stderr = tempfile.NamedTemporaryFile(dir=self.stderr_dir, delete=False)

in test/functional/test_framework/test_node.py:21 in a94c68b3f4 outdated

  17 | @@ -18,6 +18,7 @@
  18 |  import time
  19 |  import urllib.parse
  20 |  import collections
  21 | +import shlex

jnewbery commented at 9:56 PM on November 30, 2018:

Why no sorting? :cry:

jamesob commented at 4:12 PM on January 2, 2019:

If we want to enforce alphabetical sorting for imports, we should have a linter for it!

in test/functional/example_test.py:196 in a94c68b3f4 outdated

 192 | @@ -193,10 +193,14 @@ def run_test(self):
 193 |  
 194 |          self.log.info("Wait for node2 reach current tip. Test that it has propagated all the blocks to us")
 195 |  
 196 | -        getdata_request = msg_getdata()
 197 | -        for block in blocks:
 198 | -            getdata_request.inv.append(CInv(2, block))
 199 | -        self.nodes[2].p2p.send_message(getdata_request)
 200 | +        # If we have Linux perf-tools installed, we can easily profile part of the

jnewbery commented at 10:01 PM on November 30, 2018:

I think this example code should be in test/README.md, which is documentation of how to run tests and contains debugging/troubleshooting tips.

This is meant as an example of how to write a test. I don't think the expectation is that people leave profile_with_perf contexts in test that are checked into the repo.

jamesob commented at 5:38 PM on January 2, 2019:

Usage here is required to get around vulture's dead code check, but I'll add documentation to test/functional/README.md.

jnewbery commented at 10:40 PM on January 7, 2019:

If this is just to get around vulture, then I'd prefer to just update lint-python-dead-code.sh to ignore the profile_with_perf name. This file is supposed to be a guide for writing a new testcase. As far as I can tell, profile_with_perf shouldn't be included in new testcases to be checked into master.

jnewbery commented at 10:04 PM on November 30, 2018: member

This is really cool. I've now had a chance to play around with it and it's great. I think doing the following would make it even more useful and accessible for developers:

move the documentation about installing and using perf to doc/developer-notes.md
move the documentation about using profile_with_perf() to test/README.md, with a reference to doc/developer-notes.md for notes on perf.

DrahtBot added the label Needs rebase on Dec 13, 2018

laanwj commented at 2:16 PM on January 2, 2019: member

Concept ACK

jamesob force-pushed on Jan 2, 2019

jamesob commented at 7:09 PM on January 2, 2019: member

Thanks for the review, @jnewbery. I've pushed a rebased tip that addresses the feedback.

Here's the diff (excluding the rebase).

jamesob force-pushed on Jan 2, 2019

DrahtBot removed the label Needs rebase on Jan 3, 2019

in doc/developer-notes.md:283 in 27a895d72a outdated

 278 | +$ sudo sysctl -w kernel.perf_event_paranoid=-1
 279 | +$ sudo sysctl -w kernel.kptr_restrict=0
 280 | +```
 281 | +
 282 | +Make sure you [understand the security
 283 | +trade-offs](https://lwn.net/Articles/420403/) of setting these kernel

ryanofsky commented at 7:07 PM on January 4, 2019:

In commit "docs: add perf section to developer docs" (27a895d72a385bcaa861e08b03bb5c817f8e253f)

Slightly strange the link text is "understand the security tradeoffs" instead of just "security tradeoffs."

ryanofsky approved

ryanofsky commented at 7:13 PM on January 4, 2019: member

utACK 27a895d72a385bcaa861e08b03bb5c817f8e253f. Changes since last review: rebase, moving & adding documentation, tweaking output paths.

MarcoFalke added this to the milestone 0.18.0 on Jan 5, 2019

in test/functional/README.md:122 in 27a895d72a outdated

 117 | @@ -118,3 +118,36 @@ Helpers for script.py
 118 |  
 119 |  #### [test_framework/blocktools.py](test_framework/blocktools.py)
 120 |  Helper functions for creating blocks and transactions.
 121 | +
 122 | +### Benchmarking with perf

jnewbery commented at 10:26 PM on January 7, 2019:

This documentation is excellent. I think it should live in /test/README.md, which is documentation for running the functional tests, and already includes hints on logging, attaching a debugger, etc. /test/functional/README.md is documentation for writing test cases (we could probably add a note to the top of each file explaining that).

in test/functional/test_framework/test_node.py:201 in 27a895d72a outdated

 196 | @@ -186,6 +197,9 @@ def start(self, extra_args=None, *, stdout=None, stderr=None, **kwargs):
 197 |          self.running = True
 198 |          self.log.debug("bitcoind started, waiting for RPC to come up")
 199 |  
 200 | +        if self.start_perf:
 201 | +            self._start_perf("")

jnewbery commented at 10:51 PM on January 7, 2019:

I think it'd be better to pass None here (or even better have None as the default argument). "" and None will get treated the same in (profile_name or 'test').

EDIT: I see you're using profile_name as the key in the perf_subprocesses dictionary so this wouldn't work. Is perf_subprocesses safe from having entries overwritten if the same profile_name is passed in twice?

in test/functional/test_framework/test_node.py:402 in 27a895d72a outdated

 397 | +        return subp
 398 | +
 399 | +    def _stop_perf(self, profile_name):
 400 | +        """Stop (and pop) a perf subprocess."""
 401 | +        subp = self.perf_subprocesses.pop(profile_name)
 402 | +        output_path = subp.args[subp.args.index('-o') + 1]

jnewbery commented at 10:55 PM on January 7, 2019:

This seems a bit brittle. Probably ok to leave as it is, but if anyone updates the arguments or ordering of arguments, then this will break.

jamesob commented at 1:36 PM on January 22, 2019:

I don't think this is as brittle as it seems; you're always going to have the filename following -o otherwise the argument doesn't make any sense.

in test/functional/README.md:129 in 27a895d72a outdated

 124 | +An easy way to profile node performance during functional tests is provided
 125 | +for Linux platforms using `perf`.
 126 | +
 127 | +Perf will sample the running node and will generate profile data in the node's
 128 | +datadir. The profile data can then be presented using `perf report` or a graphical
 129 | +tool like `hotspot`<https://github.com/KDAB/hotspot>.

jnewbery commented at 10:57 PM on January 7, 2019:

Make this a markdown link: [hotspot](https://github.com/KDAB/hotspot).

jnewbery commented at 10:58 PM on January 7, 2019: member

Looks great James. A few comments inline.

tests: add utility to easily profile node performance with perf

Introduces `TestNode.profile_with_perf()` context manager which
samples node execution to produce profiling data.

Also introduces a test framework flag, `--perf`, which will run
perf on all nodes for the duration of a given test.

58180b5fd4

docs: add perf section to developer docs 13782b8ba8

jamesob force-pushed on Jan 22, 2019

jamesob commented at 2:04 PM on January 22, 2019: member

Pushed an update addressing feedback from @jnewbery.

ryanofsky approved

ryanofsky commented at 7:54 PM on February 4, 2019: member

utACK 13782b8ba84c5033a59a5234410a763393eafb8d. Main change is removing code in example test, and adding to test README instead.

MarcoFalke merged this on Feb 5, 2019

MarcoFalke closed this on Feb 5, 2019

MarcoFalke referenced this in commit 5029e94f85 on Feb 5, 2019

pravblockc referenced this in commit afaaff4669 on Aug 10, 2021

pravblockc referenced this in commit 3018175236 on Aug 14, 2021

pravblockc referenced this in commit cb8f2265a3 on Aug 17, 2021

pravblockc referenced this in commit 891c1c6298 on Aug 17, 2021

pravblockc referenced this in commit 693e1bb4bb on Aug 18, 2021

pravblockc referenced this in commit 33384816b5 on Aug 19, 2021

DrahtBot locked this on Dec 16, 2021