Question 220
A payments ETL service receives changing event payloads and must protect downstream contracts. Engineers have applied Add explicit schema/field validation and route the invalid dataset to a controlled failure. Which result establishes the required behavior: An upstream contract change makes a required field disappear completely?
Explanation
Correct Answer
The job fails with a missing-required-field signal before publishing incorrect output directly measures whether Add explicit schema/field validation and route the invalid dataset to a controlled failure achieved the requirement “An upstream contract change makes a required field disappear completely.” It checks the operating result rather than just a configured resource name.
Incorrect Answers
New unexpected attributes appear in the rescued data payload would demonstrate the separate action Use Auto Loader `rescue` schema evolution mode with a rescued data column, which addresses “Unexpected fields should be retained for inspection but not added to the target schema.” It does not verify the requirement “An upstream contract change makes a required field disappear completely” following Add explicit schema/field validation and route the invalid dataset to a controlled failure.
This signal belongs to the adjacent case “A source numeric type widens within supported compatibility boundaries,” whose remedy is Use supported schema/type widening handling rather than forcing lossy casts. The observed result for the present requirement is The job fails with a missing-required-field signal before publishing incorrect output.
A restart after a new column update recognizes the evolved schema pertains to a different control (Configure appropriate Auto Loader schema evolution and persisted schema location) and a different need (New optional JSON fields should be captured without discarding previously known columns). It cannot establish that Add explicit schema/field validation and route the invalid dataset to a controlled failure met the stated goal.