Add documentation about generating flame graphs. #12649

eklitzke commented at 6:23 PM on March 8, 2018: contributor

This adds documentation for generating flame graphs (a profiling format I've found to be very useful). It's pretty straightforward: I've added a new file in doc/, updated doc/developer-notes.md, and checked in a patch and example flame graph to contrib.

You can see how it looks when rendered on GitHub here: https://github.com/eklitzke/bitcoin/blob/flamegraphs/doc/flamegraphs.md

The one thing that's kind of weird is that when you look at an actual flame graph file you get hover events and can interact with the graph (for example, here's the same example.svg file I checked in on an actual web page). The way GitHub renders the markdown file (which refers to the example.svg I added to contrib) prevent this from happening, it just gets rendered as a static image. It would be kind of cool if we could host the example file on bitcoincore.org or something.

Add documentation about generating flame graphs. 9c25c1b76c

eklitzke force-pushed on Mar 8, 2018

MarcoFalke commented at 7:48 PM on March 8, 2018: member

Concept ACK 🔥, just some notes:

I'd prefer if we didn't check in the svg file (less files, less to maintain). If someone is into flames, they can generate the svg themselfes.
My preference would be to move the patch file into the devtools subfolder, since it seems this is only useful for devs.
Unrelated: You could add an :(exclude) for .patch files in line 19 of ./contrib/devtools/lint-whitespace.sh if you feel like it.

in doc/flamegraphs.md:8 in 9c25c1b76c outdated

   0 | @@ -0,0 +1,108 @@
   1 | +## Flame Graphs
   2 | +
   3 | +Flame graphs are a way of visualizing software CPU and heap profiles. Any tool
   4 | +that can create stack data in the correct format can be used to generate a flame
   5 | +graph. You can learn more about flame graphs
   6 | +[here](http://www.brendangregg.com/flamegraphs.html).
   7 | +
   8 | +Example flame graph (click to zoom):

Sjors commented at 8:13 PM on March 8, 2018:

When I preview this page on Github, clicking on the image takes me to the image page, rather than zoom. An animated gif might be a better way to illustrate the behavior.

in doc/flamegraphs.md:42 in 9c25c1b76c outdated

  37 | +sudo apt-get install linux-perf
  38 | +
  39 | +# Install perf on Fedora.
  40 | +sudo dnf install perf
  41 | +```
  42 | +

Sjors commented at 8:18 PM on March 8, 2018:

# Install on macOS
brew install google-perftools

eklitzke commented at 5:35 PM on March 9, 2018:

This is a totally different project.

in doc/flamegraphs.md:46 in 9c25c1b76c outdated

  41 | +```
  42 | +
  43 | +### Generating Flame Graphs On Linux Using Perf
  44 | +
  45 | +To generate flame graphs with labeled function names you'll need a build of
  46 | +Bitcoin that has debugging symbols enabled (this should be the case by default).

Sjors commented at 8:23 PM on March 8, 2018:

Are you sure? ./configure --enable-debug is off by default.

in doc/flamegraphs.md:52 in 9c25c1b76c outdated

  47 | +You can check that your executable has debugging symbols by checking for a
  48 | +`.debug_str` section in the ELF file:, e.g.
  49 | +
  50 | +```bash
  51 | +# If this produces output your binary has debugging symbols.
  52 | +$ readelf -S src/bitcoind | grep .debug_str

Sjors commented at 8:26 PM on March 8, 2018:

For OSX, use greadelf instead of readelf, which can be installed using brew install binutils

in doc/flamegraphs.md:60 in 9c25c1b76c outdated

  55 | +If you want kernel annotations (optional) then you should also make sure that
  56 | +the `kernel.perf_event_paranoid` sysctl option is set to -1:
  57 | +
  58 | +```bash
  59 | +# Optional, enable kernel annotations
  60 | +sudo sysctl kernel.perf_event_paranoid=-1

Sjors commented at 8:53 PM on March 8, 2018:

On macOS this complains with:

sysctl: unknown oid 'kernel.perf_event_paranoid'

in doc/flamegraphs.md:68 in 9c25c1b76c outdated

  63 | +Now you're ready to generate a profile. To profile a running bitcoind process
  64 | +for 60 seconds you can use the following command:
  65 | +
  66 | +```bash
  67 | +# Profile bitcoind for 60 seconds
  68 | +perf record -g --call-graph dwarf -F 101 -p $(pidof bitcoind) -- sleep 60

Sjors commented at 8:55 PM on March 8, 2018:

pidof on macOS can be installed with brew install pidof. perf is gperf on macOS, although it doesn't recognize --call-graph.

Sjors commented at 8:56 PM on March 8, 2018: member

Concept ACK, but having trouble getting this to work on macOS. That could be "fixed" with a note saying this doesn't work on macOS, but it would be nice if it did work.

greadelf returns:

 Not an ELF file - it has the wrong magic bytes at the start

file src/bitcoind
src/bitcoind: Mach-O 64-bit executable x86_64

meshcollider added the label Docs on Mar 9, 2018

Address GitHub review feedback. 517b380680

eklitzke commented at 5:34 PM on March 9, 2018: contributor

Changes in my latest commit:

Generally reword a lot of the flamegraphs.md file for more detail and clarity
Make the Linux-specific stuff more explicit
Remove the checked in example SVG file
Move the patch to devtools

The other change is to link to a copy of the SVG file I am hosting on monad.io (a domain I own). I feel kind of weird about this but it's the only way I could figure out for hosting the example file in a way where people can click it and actually zoom, use mouseover events, etc. I feel like that's a really compelling part of the flamegraph tool, and is important to demonstrate why it's so much cooler than other profiling tools. If someone who has access to bitcoincore.org or another more official resource (maybe the wiki?) could host the file there (it's a small static cacheable file) I'd be happy to change the link.

exclude .patch files from whitespace linter e22e7af2dd

eklitzke commented at 6:00 PM on March 9, 2018: contributor

I opened this PR on bitcoincore.org to see if I can get the SVG file hosted here.

add todo about hosting on bitcoincore.org 3c9ddd3ec7

fix broken link e3b29acf95

laanwj commented at 12:36 PM on March 10, 2018: member

Thanks for adding documentation on this!

What about adding this to the maintainer tools or docs repository?

This not bound to bitcoin core's release cycle, after all.

in doc/flamegraphs.md:73 in e3b29acf95

  68 | +Perf is available on most Linux distros in a package named `perf` or some
  69 | +variant thereof:
  70 | +
  71 | +```bash
  72 | +# Install perf on Debian.
  73 | +sudo apt-get install linux-perf

sipa commented at 5:38 PM on March 10, 2018:

Seems on Ubuntu you need linux-tools-generic, linux-cloud-tools-generic, linux-tools-common.

eklitzke commented at 12:22 AM on March 11, 2018:

It seems that linux-tools is the old package name, I can update.

eklitzke commented at 12:25 AM on March 11, 2018: contributor

@laanwj I didn't know about those repositories until you pointed them out to me. I'm happy to move the docs if we can still keep a link to them in developer-notes.md. Does that seem like a reasonable compromise?

fanquake commented at 10:09 AM on March 11, 2018: member

@eklitzke Yes, that sounds fine.

martinus commented at 10:46 AM on March 13, 2018: contributor

Flamegraphs are awesome, but I have personally switched from using the flamegraph scripts to the KDAB/hotspot profiler, which basically is a flamegraph GUI for perf: https://github.com/KDAB/hotspot

laanwj commented at 4:19 PM on March 19, 2018: member

I'm happy to move the docs if we can still keep a link to them in developer-notes.md. Does that seem like a reasonable compromise?

Yes, that'd be the way to do it.

in doc/flamegraphs.md:5 in e3b29acf95

   0 | @@ -0,0 +1,176 @@
   1 | +## Flame Graphs
   2 | +
   3 | +Flame graphs are a way of visualizing software CPU and heap profiles. Any tool
   4 | +that can create stack data in the correct format can be used to generate a flame
   5 | +graph. The flame graphs themselves are SVG files that have a small have a small

jnewbery commented at 7:26 PM on March 19, 2018:

s/have a small have a small/have a small/

eklitzke commented at 8:56 AM on March 20, 2018: contributor

Apologies for the delay on this, I was asked on IRC if I still plan to move this to the docs repo, and I do intend to. I will update this tomorrow.

eklitzke closed this on Mar 22, 2018

MarcoFalke referenced this in commit 5029e94f85 on Feb 5, 2019

GChuf commented at 12:41 PM on October 9, 2019: contributor

@eklitzke the links you posted at doc/flamegraphs are dead - can you generate/upload the pic and svg file again?

MarcoFalke commented at 1:31 PM on October 9, 2019: member

@GChuf You can generate them yourself. If you are looking for an illustrative example, there are plenty: https://duckduckgo.com/?q=flamegraph&t=ffab&iax=images&ia=images

GChuf commented at 2:12 PM on October 9, 2019: contributor

Just wanted to point out the broken links. Anyone can find examples on the internet - if I had found a bitcoin example i'd already propose to replace the picture.

achow101 commented at 2:29 PM on October 9, 2019: member

Here's a flame graph I generated for doing some of my wallet load time stuff: https://achow101.com/bitcoin-master-flame.svg

jasonbcox referenced this in commit 0d7f8ccb11 on Jul 30, 2020

pravblockc referenced this in commit afaaff4669 on Aug 10, 2021

pravblockc referenced this in commit 3018175236 on Aug 14, 2021

pravblockc referenced this in commit cb8f2265a3 on Aug 17, 2021

pravblockc referenced this in commit 891c1c6298 on Aug 17, 2021

pravblockc referenced this in commit 693e1bb4bb on Aug 18, 2021

pravblockc referenced this in commit 33384816b5 on Aug 19, 2021

DrahtBot locked this on Dec 16, 2021