Question 220
During a GPU datacenter maintenance window, an engineer discovers: An AI cluster has a functioning BCM primary and standby, but a split-brain test reveals that both managers can modify node provisioning state after losing their heartbeat. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Redundant managers without proper leadership arbitration can corrupt the provisioning database during network partitions.) and supports continued assurance through: A forced partition leaves a single active management authority and protects configuration state against conflicting writes.
Incorrect Answers
This prevention control targets another operational risk: A BCM provisioning environment discovers every compute node, yet new GPU hosts join the wrong category and receive CPU-only operating system images. The currently reported mechanism remains: Redundant managers without proper leadership arbitration can corrupt the provisioning database during network partitions.
This prevention control targets another operational risk: An operations team upgrades BCM directly in production without testing whether existing Slurm and provisioning integrations are compatible with the new management API. The currently reported mechanism remains: Redundant managers without proper leadership arbitration can corrupt the provisioning database during network partitions.
This prevention control targets another operational risk: Several AI nodes pull a moving NGC container tag at job time, producing different application libraries across otherwise identical distributed training ranks. The currently reported mechanism remains: Redundant managers without proper leadership arbitration can corrupt the provisioning database during network partitions.