Question 70
A telemetry team runs stateful Structured Streaming queries in production. After configuring Use `foreachBatch` with idempotent writes keyed by microbatch ID or business key, the engineer must validate: A stream must deliver at-least-once microbatches to an external system safely. What evidence is decisive?
Explanation
Correct Answer
The acceptance criterion for “A stream must deliver at-least-once microbatches to an external system safely” is Retry of one microbatch does not duplicate the externally committed records; this is the expected evidence after Use `foreachBatch` with idempotent writes keyed by microbatch ID or business key.
Incorrect Answers
State store size stabilizes as sufficiently old event-time state expires would demonstrate the separate action Set a suitable event-time watermark for the use case, which addresses “A stateful aggregation grows without bound due to no late-event cutoff.” It does not verify the requirement “A stream must deliver at-least-once microbatches to an external system safely” following Use `foreachBatch` with idempotent writes keyed by microbatch ID or business key.
This signal belongs to the adjacent case “A data engineer changes stateful grouping keys and the existing checkpoint is incompatible,” whose remedy is Plan a new checkpoint and controlled state reinitialization or migration. The observed result for the present requirement is Retry of one microbatch does not duplicate the externally committed records.
That observation is a valid indicator for “Two independent streaming queries were configured to share the same checkpoint path” when using Give each logical streaming query its own checkpoint location. The scenario instead tests “A stream must deliver at-least-once microbatches to an external system safely”; the relevant result is Retry of one microbatch does not duplicate the externally committed records.