Question 20
During a GPU datacenter maintenance window, an engineer discovers: A training farm connects every GPU node through one oversubscribed top-of-rack switch even though collective jobs exchange large volumes of east-west gradients. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (High-volume east-west collective traffic requires fabric capacity and resilience beyond basic node-to-service reachability.) and supports continued assurance through: A representative all-reduce run across distant racks reaches the approved throughput and remains operational after one specified uplink failure.
Incorrect Answers
This prevention control targets another operational risk: A cluster uses dual network adapters, but both uplinks terminate on one leaf switch and the team claims it can withstand any single switch failure. The currently reported mechanism remains: High-volume east-west collective traffic requires fabric capacity and resilience beyond basic node-to-service reachability.
This prevention control targets another operational risk: A multi-tenant AI facility carries BMC administration and high-throughput GPU training packets on one flat unsegmented fabric. The currently reported mechanism remains: High-volume east-west collective traffic requires fabric capacity and resilience beyond basic node-to-service reachability.
This prevention control targets another operational risk: A monitoring agent reports no GPU errors because its account cannot access the required driver interfaces, while local `nvidia-smi` output reveals active device faults. The currently reported mechanism remains: High-volume east-west collective traffic requires fabric capacity and resilience beyond basic node-to-service reachability.