Part 4 of a series on data center networking for AI
Many of the distinctive traffic patterns we discussed in Part 1, like elephant flows, synchronized barriers, and incast bursts, come from training. Most enterprise AI infrastructure runs inference rather than training jobs, though. This post makes the case that inference deserves just as much network design attention as training, and walks through why, using examples you’ve likely experienced firsthand as a user of tools like Claude or ChatGPT.
The KPIs at the request level
Training’s KPI is job completion time across one long-running job. Inference has two separate, competing KPIs measured per request:
- Time to first token (TTFT) – the latency from request arrival to the first generated token
- Inter-token latency (ITL), also called time per output token (TPOT) – latency between each subsequent token
These map onto the two phases every inference request goes through, and the phases have almost opposite resource profiles.
Prefill: The entire input prompt is processed in one parallel forward pass. Prefill is compute-bound, scales well across GPUs using the tensor-parallel all-gather pattern from Part 3, and produces both the first output token and the KV cache (the running memory of every prior token’s key/value projections, needed for every subsequent step).
Decode: Tokens are generated one at a time, autoregressively, meaning each step reads the entire, growing KV cache but does comparatively little computation, so the GPU spends most of its time waiting on memory bandwidth rather than compute. From a network perspective, it’s important to know that for a tensor-parallel model, every single decode step re-triggers a collective communication operation, not once per batch like training, but once per generated token. Collective latency stops being an aggregate-throughput concern and becomes directly, visibly user-facing responsiveness. A congested link doesn’t just slow a job down, it stutters the token stream a real person is reading in real time.
As an example, what happens when you send a message to an AI assistant
It’s tempting to assume that starting a new conversation with an AI assistant is “prefill” and continuing that conversation is “decode,” since continuing implies reading cached context. But that’s not true.
Every single response, including the very first one, involves both phases in sequence. Prefill runs first, processing everything accumulated so far, including the system prompt, the entire conversation history, and your newest message, in one parallel pass. Decode then generates the reply, one token at a time, streaming the words you watch appear on screen. This happens on every turn, not just the first.
What does change turn-to-turn is how expensive prefill is. If the serving system caches the KV cache from earlier turns (a technique called prefix caching, or KV cache reuse) a later turn only needs to prefill the new tokens you just typed, while decode still runs in full every time to generate a fresh reply. In essence, decode is the part you watch happen – the streaming reply. Prefill is the part that happens right before each streamed reply begins, and it gets cheaper on later turns if caching is working well.
This gets more interesting once you introduce tool use, which is an increasingly common pattern in agentic AI deployments.
When an assistant calls a tool mid-response, such as a web search, a code execution, a database query, a network command, etc., the tool’s result has to be incorporated before the reply can continue. That triggers another prefill step (processing the new tool output), followed by decode resuming. A single agentic turn, especially one involving several tool calls, can chain multiple prefill/decode cycles together. This is the workload profile driving a wave of new infrastructure investment in KV cache management, which brings us to a new piece of the puzzle.
Where the KV cache lives, and how it moves
Because prefill and decode have opposite resource profiles, an emerging serving architecture called disaggregated prefill/decode serving splits them onto separate, specialized GPU pools rather than running both phases on the same hardware.
The KV cache generated by the prefill pool has to be transferred across the network to the decode pool before decode can begin. Disaggregated prefill/decode serving is a network operation unique to inference, absent from AI training. This network-dependent transfer sits directly on the time-to-first-token critical path.
Where does the KV cache physically live? It’s generated in GPU HBM (high-bandwidth memory) during prefill. As conversations, documents, or agentic sessions grow longer, the KV cache can outgrow HBM entirely. At 1M-token context windows, the KV cache for a single user running a 70-billion-parameter model can exceed 320GB, roughly four times the entire HBM capacity of a single high-end GPU. That gap is what’s driving a new category of infrastructure, which is a tiered storage hierarchy that moves cold KV cache down to cheaper, larger, slower storage, and pre-stages it back up before it’s needed again.
NVIDIA’s recently announced CMX platform is a purpose-built example of this, designed with a dedicated, pod-level, Ethernet-attached flash tier sitting between GPU memory and general-purpose storage, specifically for KV cache. The transfer itself rides the same RDMA fabric concepts covered in Part 2. The low-latency, high-bandwidth Ethernet fabric moves cache data between compute and storage nodes, with a serving framework routing each request to wherever its KV cache already resides, so applications don’t have to manage this placement themselves.
This is not yet a common deployment model today. Plenty of enterprise deployments still run prefill and decode on the same GPUs with no dedicated KV cache tier at all, because it’s operationally simpler and the workload doesn’t yet demand the complexity. Disaggregation and dedicated tiering earn their keep at a specific pain point, usually when serving very long context windows and long-running agentic sessions. Purpose-built, hardware-accelerated platforms like CMX are aimed squarely at large-scale, pod-level infrastructure. Smaller or more modest deployments have a vendor-neutral alternative in open-source projects that provide similar hierarchical caching over standard networking and commodity hardware, without requiring specialized hardware.
At scales beyond the 4 walls of a data center, AI networking providers like Arrcus are solving for the full network path starting at the agent request. The Arrcus Inference Network Fabric (AINF) is intelligently routing each request not just across KV cache tiers in a site, but across multiple data centers, pops, and regions, based on characteristics like latency, cost, kv-cache affinity, and governance, which introduces an entirely new network solution to a real world inference problem that essentially surfaced overnight.
A non-blocking fabric can still get congested
Non-blocking describes design intent, not infinite capacity – it effectively means the fabric has enough bandwidth for the traffic pattern it was engineered around. GPUs on different racks still cross the spine to reach each other, and a large enough all-to-all burst can saturate that spine capacity even in a well-designed fabric.
Remember, training clusters are almost always single-tenant, so a congested link only ever hurts that one customer’s own job. Inference infrastructure is usually multi-tenant, where several models or business units share the same physical fabric simultaneously, because dedicating a full non-blocking pod to every individual workload isn’t economical.
When one tenant’s MoE model fires off an all-to-all burst, that traffic physically shares the same rails and spine links another tenant’s decode traffic is using at that same moment. The fabric isn’t over-provisioned to handle every tenant bursting at once, because doing so isn’t economical. So one tenant’s burst can realistically queue behind or contend with another’s traffic.
This shows up in decode more than prefill. A slow prefill delays time-to-first-token too, so prefill isn’t immune to congestion. The difference is frequency and perceptibility: decode requires a collective operation on every single generated token, so across one response there are dozens or hundreds of opportunities for a shared-link congestion event to land badly, while prefill happens once per request. When congestion hits decode, the user doesn’t experience one delayed measurement. Instead, they watch the reply visibly stutter mid-stream, which is a far more perceptible degradation than a slightly longer initial pause. This is the concrete mechanism behind “noisy neighbor” risk in shared inference infrastructure, and it’s a different problem than anything a dedicated, single-tenant training cluster has to manage.
Some relatable use cases
To help this stick, here are a few use cases, ranging from consumer-familiar to industry-specific:
- RAG applications (retrieval-augmented generation, where a chatbot stuffs retrieved documents into the prompt before answering) are prefill-heavy by nature. Every query re-injects fresh retrieved text, limiting how much KV cache can be reused across turns, so TTFT dominates the experience. This is a common enterprise pattern, seen in internal knowledge-base assistants and support bots that pull from documentation.
- Long agentic coding sessions, where an AI agent works through a codebase across many tool calls, are decode-heavy and cache-reuse-heavy, with context growing across dozens of tool-call cycles. This is the “agentic long-term memory” use case driving investment in KV cache tiering.
- Industry-specific contrast: a legal-tech contract review tool feeding a 400-page document into a single call is almost entirely prefill/TTFT-bound. A healthcare intake chatbot running many short back-and-forth turns is decode/ITL-bound and highly cache-reuse-sensitive. Both are “just chatbots” from the outside, but they’re quite different network design problems.
Takeaways
- TTFT and inter-token latency are the metrics that matter for inference SLAs, not job completion time.
- Decode’s per-token collective frequency means network latency translates directly into perceived responsiveness, which is a much tighter latency budget than training’s per-step collectives.
- If a client is running or considering disaggregated prefill/decode serving, the KV cache transfer link deserves the same design scrutiny as any other elephant flow.
- Multi-tenant inference infrastructure needs explicit isolation and QoS design. Non-blocking fabric design reduces contention risk but doesn’t eliminate it. Noisy-neighbor congestion is a risk in shared GPU fleets in a way it typically isn’t in dedicated training clusters.
Next in this series: fabric topologies for GPU clusters, including Clos and leaf-spine fundamentals, rail-optimized designs, and alternative topologies like Dragonfly.
