Question 460
During a GPU datacenter maintenance window, an engineer discovers: NCCL all-reduce performs well within a rack but slows dramatically when ranks span two racks connected by a single overburdened uplink. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Intra-rack success does not prove adequate bisection bandwidth for cluster-wide synchronous communication.) and supports continued assurance through: Inter-rack collective bandwidth meets the approved floor under representative rank placement and link counters show expected distribution.
Incorrect Answers
This prevention control targets another operational risk: An all-reduce job shows periodic bandwidth collapses during heavy traffic, and switch telemetry reveals microburst drops on a small set of fabric ports. The currently reported mechanism remains: Intra-rack success does not prove adequate bisection bandwidth for cluster-wide synchronous communication.
This prevention control targets another operational risk: Network engineers run NCCL tests with rank placement fixed to adjacent hosts and overlook an entire failed fabric rail serving a distant rack. The currently reported mechanism remains: Intra-rack success does not prove adequate bisection bandwidth for cluster-wide synchronous communication.
This prevention control targets another operational risk: A data center validates HPL only for a single peak-score run, although the customer requires continuous model training over long maintenance intervals. The currently reported mechanism remains: Intra-rack success does not prove adequate bisection bandwidth for cluster-wide synchronous communication.