Question 230
A distributed GPU job is slow even though GPU memory is available and each worker is busy loading CPU-side batches. Which bottleneck should be diagnosed?
Explanation
Correct Answer
Input pipeline starvation can leave accelerators underused; more GPUs alone will not accelerate an inadequate data loader.
Incorrect Answers
The alternative, Coordinate per-worker batch size, effective global batch size, and learning-rate schedule, belongs to a different technical condition: Scaling a model from one to eight workers unexpectedly changes global batch size and model convergence. There the source-specific mechanism or proof is: More workers can alter total batch size and optimization dynamics unless hyperparameters account for the distributed setup. The condition asked about here is: A distributed GPU job is slow even though GPU memory is available and each worker is busy loading CPU-side batches. A suitable correction is Profile preprocessing and data-loader throughput before increasing GPU worker count; the targeted justification or result is: Input pipeline starvation can leave accelerators underused; more GPUs alone will not accelerate an inadequate data loader.
Evaluate inter-node bandwidth and latency for collective gradient communication. That operation would be relevant if: A deep-learning model uses gradient all-reduce, but slow network communication consumes more time than calculation. Its corresponding reason or verification is: Distributed gradient synchronization incurs communication overhead; poor interconnects can erase the benefit of additional workers. However the problem in this question is: A distributed GPU job is slow even though GPU memory is available and each worker is busy loading CPU-side batches. The requirement instead calls for Profile preprocessing and data-loader throughput before increasing GPU worker count; the question-specific check is: Input pipeline starvation can leave accelerators underused; more GPUs alone will not accelerate an inadequate data loader.
Ensure matching world-size, rank, and rendezvous settings on participating workers. A separate source case supports that approach when: A framework's distributed job hangs because workers disagree about world size and rank assignments. The supporting rationale or observable result is: A distributed process group requires consistent membership information to coordinate synchronization across nodes. This question describes a different issue: A distributed GPU job is slow even though GPU memory is available and each worker is busy loading CPU-side batches. Address it through Profile preprocessing and data-loader throughput before increasing GPU worker count, with the required justification or evidence: Input pipeline starvation can leave accelerators underused; more GPUs alone will not accelerate an inadequate data loader.