Question 160
During a GPU datacenter maintenance window, an engineer discovers: A single-node CUDA test passes, yet a distributed job hangs because the NVIDIA collective communication transport cannot communicate across hosts. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Local GPU success does not demonstrate readiness of the network and software required for distributed collective training.) and supports continued assurance through: A supported multi-node all-reduce exercise completes at the required bandwidth while link and device logs stay clean.
Incorrect Answers
This prevention control targets another operational risk: A node passes an accelerator benchmark only when power caps are disabled, but the approved rack design requires a constrained energy profile. The currently reported mechanism remains: Local GPU success does not demonstrate readiness of the network and software required for distributed collective training.
This prevention control targets another operational risk: After drivers are installed, eight GPUs appear in the host inventory, but a test training program fails when allocating memory on one device. The currently reported mechanism remains: Local GPU success does not demonstrate readiness of the network and software required for distributed collective training.
This prevention control targets another operational risk: A recently delivered accelerator system has inconsistent PSU models across sibling nodes, and unexpected power alarms occur only during high utilization. The currently reported mechanism remains: Local GPU success does not demonstrate readiness of the network and software required for distributed collective training.