a networkers blog
This post emphasizes the significance of network design for inference in AI, highlighting its distinct KPIs: Time to First Token (TTFT) and Inter-Token Latency (ITL). It discusses the prefill and decode phases, their resource profiles, and the challenges of KV cache management in multi-tenant environments, stressing the impact of network congestion on user experience.
This post, part of a series on AI data center networking, clarifies the traffic patterns arising from collective communications during GPU operations. It highlights key operations like broadcast, scatter, gather, and reduce, explaining their significance for both training and inference, especially in Mixture-of-Experts models. Understanding these patterns aids in optimizing network performance.
This article discusses essential aspects of data center networking for AI, emphasizing the unique communication dynamics between GPUs. It highlights the importance of PCIe bus architecture, NUMA locality, and NVLink technology in optimizing performance. The design strategy of one NIC per GPU and the use of SmartNICs in multi-tenant environments are also explored.
This article discusses the necessity for rethinking data center networking in the context of AI. Traditional designs based on traffic averages and oversubscription do not apply, as AI workloads produce massive, synchronized “elephant” flows, necessitating non-blocking architectures and emphasizing metrics like job completion time and GPU utilization over conventional measures.
TLDR; Keep your eye on Meter, a Network-as-a-Service company with a vision so far in the future that it seems nearly impossible, yet they are actively...
In its most rudimentary terms, a digital twin in the realm of networking is the concept of emulating or simulating a like-for-like topology of a production...
The topic of AI in the networking space has quickly moved from “what if” to “how soon”. Every major networking vendor is leveraging some aspect of...
Most network automation journeys start with building custom scripts and integrations, validating use cases, and getting value from low-hanging fruit. This is the fun part! However,...
Network engineers, even those that have adopted a developer mentality, often struggle with getting to that next evolution of self-service in on-premises data centers. We have...
Doing Infrastucture-as-Code (IaC) with Ansible has given me a headache – so I’ve recently been playing around with Terraform as an alternative to Ansible for certain...
