Building an 800G AI back-end fabric on SONiC
How we approach a lossless RoCEv2 fabric for a growing GPU cluster, from hardware selection to the first production training job.
This is an example of how we approach this kind of project, not a report on a named client engagement.
The typical starting point
A team is scaling from a few hundred GPUs to a few thousand. Their current switch vendor’s lead times and licensing make each expansion slow and expensive, and they want a fabric they can grow pod by pod.
How we run it
- Hardware and release selection. We shortlist 400G/800G switches on merchant silicon and qualify a SONiC maintenance release against them in our lab.
- Fabric design. A rail-optimized leaf-spine design with BGP, ECMP and a single lossless priority for RoCEv2.
- Tuning. PFC, ECN thresholds and buffer profiles set for the actual cable lengths and port speeds, then verified with the customer’s own training jobs rather than synthetic tests.
- Operations from day one. Zero-touch provisioning, streaming telemetry and alerts on pause frames and ECN marks.
What we measure
Job completion time, PFC pause rates, ECN-marked packets, and drops on lossless queues under full load. These become the acceptance criteria for go-live.