Question 480
During a GPU datacenter maintenance window, an engineer discovers: A night-long NCCL burn-in crashes on one rank, yet the operator reruns only successful nodes and presents the new partial result as full-cluster acceptance. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Dropping failed participants hides readiness defects and invalidates claims of whole-cluster stability.) and supports continued assurance through: The acceptance manifest shows all required rank IDs finishing one standard test without unacknowledged exclusions or failures.
Incorrect Answers
This prevention control targets another operational risk: Operators increase NCCL stress parallelism until every node becomes thermally unstable, then misclassify the resulting throttling as an InfiniBand cabling fault. The currently reported mechanism remains: Dropping failed participants hides readiness defects and invalidates claims of whole-cluster stability.
This prevention control targets another operational risk: A multi-node cluster passes a ten-second NCCL test but fails after hours of collective communications as one optical link accumulates CRC and FEC errors. The currently reported mechanism remains: Dropping failed participants hides readiness defects and invalidates claims of whole-cluster stability.
This prevention control targets another operational risk: A suspected optical link fault appears only when neighboring rack doors are closed, and receiver power measurements approach the lower supported threshold. The currently reported mechanism remains: Dropping failed participants hides readiness defects and invalidates claims of whole-cluster stability.