Automatic Flamegraphs for Benchmarks in Rust

Introduction

Criterion is a well-known and often-used benchmarking framework in the Rust ecosystem. If you’ve not yet heard of it, definitely check it out!

Background

One question I often ask myself when looking at benchmarks is:

Setup

We will now create a custom profiler for criterion with pprof that will print flamegraphs for each benchmark.

[dev-dependencies]
pprof = { version = "0.3", features = ["flamegraph"] }
criterion = "0.3"
# criterion-macro = "0.3" # if using custom test frameworks
use std::{fs::File, os::raw::c_int, path::Path};use criterion::profiler::Profiler;
use pprof::ProfilerGuard;
/// Small custom profiler that can be used with Criterion to create
/// a flamegraph for benchmarks.
/// Also see [the Criterion documentation on this][custom-profiler].
///
/// ## Example on how to enable the custom profiler:
///
/// ```
/// mod perf;
/// use perf::FlamegraphProfiler;
///
/// fn fibonacci_profiled(criterion: &mut Criterion) {
/// // Use the criterion struct as normal here.
/// }
///
/// fn custom() -> Criterion {
/// Criterion::default()
/// .with_profiler(FlamegraphProfiler::new())
/// }
///
/// criterion_group! {
/// name = benches;
/// config = custom();
/// targets = fibonacci_profiled
/// }
/// ```
///
/// The neat thing about this is that it will sample _only_ the
/// benchmark, and not other stuff like
/// the setup process.
///
/// Further, it will only kick in if `--profile-time <time>` is
/// passed to the benchmark binary.
/// A flamegraph will be created for each individual benchmark in
/// its report directory under
/// `profile/flamegraph.svg`.
///
/// [custom-profiler]: https://bheisler.github.io/criterion.rs/book/user_guide/profiling.html#implementing-in-process-profiling-hooks
pub struct FlamegraphProfiler<'a> {
frequency: c_int,
active_profiler: Option<ProfilerGuard<'a>>,
}
impl<'a> FlamegraphProfiler<'a> {
#[allow(dead_code)]
pub fn new(frequency: c_int) -> Self {
FlamegraphProfiler {
frequency,
active_profiler: None,
}
}
}
impl<'a> Profiler for FlamegraphProfiler<'a> {
fn start_profiling(&mut self, _benchmark_id: &str, _benchmark_dir: &Path) {
self.active_profiler = Some(ProfilerGuard::new(self.frequency).unwrap());
}
fn stop_profiling(&mut self, _benchmark_id: &str, benchmark_dir: &Path) {
std::fs::create_dir_all(benchmark_dir).unwrap();
let flamegraph_path = benchmark_dir.join("flamegraph.svg");
let flamegraph_file = File::create(&flamegraph_path)
.expect("File system error while creating flamegraph.svg");
if let Some(profiler) = self.active_profiler.take() {
profiler
.report()
.build()
.unwrap()
.flamegraph(flamegraph_file)
.expect("Error writing flamegraph");
}
}
}
#![feature(custom_test_frameworks)]
#![test_runner(criterion::runner)]
use criterion::{Criterion, black_box};
use criterion_macro::criterion;
mod perf;
fn fibonacci(n: u64) -> u64 {
match n {
0 | 1 => 1,
n => fibonacci(n - 1) + fibonacci(n - 2),
}
}
fn custom_criterion() -> Criterion {
Criterion::default()
.with_profiler(perf::FlamegraphProfiler::new(100))
}
#[criterion(custom_criterion())]
fn bench_custom(c: &mut Criterion) {
c.bench_function("Fibonacci-Custom", |b| b.iter(|| fibonacci(black_box(20))));
}
mod perf;criterion_group!{
name = benches;
// This can be any expression that returns a `Criterion` object.
config = Criterion::default().with_profiler(perf::FlamegraphProfiler::new(100);
targets = bench
}

Running the Benchmarks and Getting Results

Enable Performance Profiling for Unprivileged Users

To enable performance profiling without running the benchmarks as root, you may need to adjust the value of perf_event_paranoid in the Linux kernel to an appropriate value for your environment. The most permissive value is -1.

echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid

Running the Benchmarks

Now that everything is set up, we can run our benchmark with

cargo bench --bench my_bench -- --profile-time=5

Viewing the Flamegraph

We will now find a file called flamegraph.svg in target/criterion/<name-of-benchmark>/profile/flamegraph.svg.

Flamegraph for the fibonacci benchmark

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store