Question 100
An ETL pipeline sends microbatch results to a transactional downstream Delta table. Following Implement its own idempotency or transactional deduplication at the sink, the incident team needs evidence that An application uses `foreachBatch` but assumes exactly-once by default for every external sink. Which observation supports closure?
Explanation
Correct Answer
The team used Implement its own idempotency or transactional deduplication at the sink to fulfill “An application uses `foreachBatch` but assumes exactly-once by default for every external sink.” Observing An injected retry leaves the sink with one logical result per source key confirms the intended effect in the resulting workload.
Incorrect Answers
That observation is a valid indicator for “A retried microbatch may repeat external side effects” when using Store or use the microbatch ID to make downstream operations idempotent. The scenario instead tests “An application uses `foreachBatch` but assumes exactly-once by default for every external sink”; the relevant result is An injected retry leaves the sink with one logical result per source key.
That observation is a valid indicator for “Microbatch processing unexpectedly skips later records because the consumer ignores part of the batch” when using Ensure the batch handler consumes all input needed by the stateful source. The scenario instead tests “An application uses `foreachBatch` but assumes exactly-once by default for every external sink”; the relevant result is An injected retry leaves the sink with one logical result per source key.
This signal belongs to the adjacent case “A custom MERGE must execute for every streaming microbatch with batch semantics,” whose remedy is Use `foreachBatch` to invoke a batch MERGE for each microbatch DataFrame. The observed result for the present requirement is An injected retry leaves the sink with one logical result per source key.