Beyond p99: Observing Async Rust

Async Rust programs are hard to observe. Runtimes schedule your work across threads and move tasks between them, so by the time a problem shows up as a latency spike on a dashboard, most of the context that would explain it is long gone.

The usual response is to add more aggregate metrics or to fall back on tedious print debugging locally. Aggregates throw away the detail you need to explain a single slow request, and sometimes problems don't reproduce on your machine.

There's a way around this: instrumentation cheap enough that you can leave it running in production, recording rich per-event detail to dig through afterwards. That's what dial9 does, and we'll use it to work out what an application is really doing.

You'll start with an application that has almost no observability and add instrumentation a step at a time, each step answering a question you couldn't answer before.

We'll build up a picture of what a Tokio runtime is doing with dial9, bringing in tracing spans and metrique metrics as we go.

By the end, you'll be able to:

• Instrument an async application so spans and contextual data survive await points and spawn boundaries
• Break a slow request down into the polls it took, and see where the time actually went
• Tell a task waiting to be polled apart from a thread the kernel hasn't scheduled yet
• Tell when an aggregate is enough and when only per-event data will do
• Judge what each piece of instrumentation costs, and pick what's worth leaving on in production

You should be comfortable with Rust and ideally have written some async code. No prior experience with these tools is needed.

Bring a laptop set up for Rust development. We'll email setup instructions ahead of time.