Question 510
A data engineering team investigates failed Spark transformations. The team implements Review the Spark UI task count and scheduling timeline. The goal was: A job has many tiny tasks and scheduling overhead dominates. Which observed result best verifies success?
Explanation
Correct Answer
The team used Review the Spark UI task count and scheduling timeline to fulfill “A job has many tiny tasks and scheduling overhead dominates.” Observing The stage has excessive short tasks relative to meaningful computation confirms the intended effect in the resulting workload.
Incorrect Answers
That observation is a valid indicator for “Only a few tasks take much longer than others in one shuffle stage” when using Examine Spark task duration distributions and input/shuffle sizes. The scenario instead tests “A job has many tiny tasks and scheduling overhead dominates”; the relevant result is The stage has excessive short tasks relative to meaningful computation.
This signal belongs to the adjacent case “A task fails with a missing dependency after a runtime upgrade,” whose remedy is Inspect driver/executor stack traces for ClassNotFound or ImportError. The observed result for the present requirement is The stage has excessive short tasks relative to meaningful computation.
The suspected stage shows substantial disk spill and associated slowdown would demonstrate the separate action Inspect Spark UI spill metrics and task timelines, which addresses “A stage repeatedly spills large intermediates to disk.” It does not verify the requirement “A job has many tiny tasks and scheduling overhead dominates” following Review the Spark UI task count and scheduling timeline.