Question 640
A Spark batch job has variable source sizes and unexpected shuffle overhead. After configuring Validate output correctness and compare adaptive physical plan independently, the engineer must validate: A batch runtime plan changes but analysts worry results are inconsistent. What evidence is decisive?
Explanation
Correct Answer
Both optimized and baseline execution return equivalent final records directly measures whether Validate output correctness and compare adaptive physical plan independently achieved the requirement “A batch runtime plan changes but analysts worry results are inconsistent.” It checks the operating result rather than just a configured resource name.
Incorrect Answers
That observation is a valid indicator for “The planner initially picks a shuffle join but one input becomes broadcast-eligible after filtering” when using Allow AQE to adapt supported join strategy at runtime. The scenario instead tests “A batch runtime plan changes but analysts worry results are inconsistent”; the relevant result is Both optimized and baseline execution return equivalent final records.
That observation is a valid indicator for “Executors are overwhelmed by excessive task count from tiny shuffle partitions” when using Tune/adopt AQE coalescing and advisory partition settings. The scenario instead tests “A batch runtime plan changes but analysts worry results are inconsistent”; the relevant result is Both optimized and baseline execution return equivalent final records.
That observation is a valid indicator for “Shuffle partitions are badly sized at runtime due to variable daily input” when using Enable supported Adaptive Query Execution partition coalescing. The scenario instead tests “A batch runtime plan changes but analysts worry results are inconsistent”; the relevant result is Both optimized and baseline execution return equivalent final records.