Question 180
During a GPU datacenter maintenance window, an engineer discovers: A third-party parallel filesystem is mounted with an inappropriate client timeout, and temporary network congestion causes every training job to abort during checkpoint writes. Which repeatable operational safeguard best prevents recurrence of this failure?
Explanation
Correct Answer
This preventive practice addresses the underlying failure (Distributed AI storage behavior depends on network and client settings as well as backend disk throughput.) and supports continued assurance through: A controlled link disturbance does not cause data corruption or permanent unavailability and job recovery meets the defined service objective.
Incorrect Answers
This prevention control targets another operational risk: A storage array is reachable from AI nodes, but a shared mount used for checkpoints lacks the expected write permissions for the training service identity. The currently reported mechanism remains: Distributed AI storage behavior depends on network and client settings as well as backend disk throughput.
This prevention control targets another operational risk: AI researchers create training directories on a shared NAS but capacity is not monitored; a large dataset import consumes the space reserved for critical model checkpoints. The currently reported mechanism remains: Distributed AI storage behavior depends on network and client settings as well as backend disk throughput.
This prevention control targets another operational risk: An operator initiates simultaneous BMC firmware updates on every host, eliminating the team's ability to recover remote power controls if the update fails. The currently reported mechanism remains: Distributed AI storage behavior depends on network and client settings as well as backend disk throughput.