Question 260
During a GPU datacenter maintenance window, an engineer discovers: Slurm reports some nodes as allocated indefinitely after a failed distributed job because job cleanup leaves Enroot container artifacts and stale GPU processes. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Stale workloads and container state can make a healthy node appear occupied or leak resource ownership between jobs.) and supports continued assurance through: A deliberately interrupted test job frees its GPU assignments and leaves the host ready for a new tenant after verified cleanup.
Incorrect Answers
This prevention control targets another operational risk: A multi-node NCCL job spans nodes with different fabric interface names, while Slurm launch scripts hard-code the interface present only on the first rack. The currently reported mechanism remains: Stale workloads and container state can make a healthy node appear occupied or leak resource ownership between jobs.
This prevention control targets another operational risk: A new node is added to the wrong Slurm partition and gains access to restricted customer workloads despite being intended for untrusted experimentation. The currently reported mechanism remains: Stale workloads and container state can make a healthy node appear occupied or leak resource ownership between jobs.
This prevention control targets another operational risk: An NGC CLI client is installed on a shared administrative workstation but lacks authenticated registry configuration, so container downloads fail with permissions errors. The currently reported mechanism remains: Stale workloads and container state can make a healthy node appear occupied or leak resource ownership between jobs.