October 15, 20264 min readAI systems

What I learned at the vLLM Toronto meetup

Notes from October 15 on agentic workloads, KV cache management, sampler orchestration, and validating agent fixes.

By Sam Zhong

On October 15, I went to a vLLM meetup in downtown Toronto. The room was packed, and I came away thinking less about a single model benchmark and more about the systems that make agents usable at scale.

A full audience at the Toronto vLLM meetup, with a presentation on serving agentic workloads — The meetup opened with a talk on serving agentic workloads at scale.

Agent traffic changes the serving problem

One slide from the Inferact and vLLM presentation made the workload concrete: an example agent session ran for 43 turns, repeatedly reused a growing prompt, and could fan out into parallel sub-agent sessions after a tool call. In that example, the slide reported a median input length of 142K tokens, a median output length of 444 tokens, more than 95% prefix-cache hits, and 44% bursty sub-agent traffic. Those figures describe the workload shown in the talk, rather than a general benchmark for every agent.

My main takeaway was that an agent request cannot be treated like a single short chat turn. A serving system has to handle long context, reuse work it has already done, and absorb sudden concurrency. Prefix caching and scheduling are therefore part of the user experience: they affect how quickly an agent can resume after each tool result.

Slide diagram showing 43 agent turns, long input context, prefix reuse, and bursty sub-agents — The workload slide connected long conversations, cached prefixes, and bursts of sub-agents.

Memory management has to follow the model

The hybrid KV cache slide showed another layer of the problem. Models may combine full attention, sliding-window attention, and linear attention, each with different cache needs. The vLLM diagram used a uniform page size across those attention types while managing how blocks are allocated. I took this as a reminder that efficient inference depends on understanding a model's attention pattern, not just choosing a larger GPU.

vLLM hybrid KV cache management slide comparing full, sliding-window, and linear attention blocks — A vLLM slide on hybrid KV cache management for different attention types.

Keep the sampler busy

Another presentation described a training setup that separates the sampler from the trainer. Sampling GPUs keep multiple requests in flight and write completed rollouts to a results buffer; the trainer pulls and filters them. The idea I want to remember is simple: orchestration matters as much as raw GPU speed when a pipeline has uneven stages.

Presentation slide describing a decoupled sampler and trainer linked by a results buffer — The sampler and trainer were decoupled so sampling GPUs could maintain parallel work.

Agent fixes still need careful validation

NVIDIA's InferCheck talk asked how well agents can debug inference accuracy bugs. Its conclusion was encouraging but measured: agents can repair real bugs, yet a workaround can leave the underlying issue unresolved; fixing the reported symptom may not restore full quality; and a repair can break another application. That last point stayed with me. An agent-generated patch needs tests that cover the affected application and nearby use cases before it becomes a pull request.

InferCheck conclusions slide warning that agent repairs need review and validation — InferCheck's closing slide: use agents to move faster, then review and validate their fixes.

What I brought home

The meetup gave me a better map of the work behind reliable AI systems. Long agent sessions make caching and scheduling central; mixed attention patterns complicate memory allocation; asynchronous pipelines need deliberate orchestration; and debugging tools need validation beyond a passing symptom check. I left with a clearer question for any future agent system I build: what happens to latency, GPU use, and correctness once the workflow becomes long, parallel, and messy?