Question 180
A product-analytics team writes Spark SQL to compute daily customer metrics. The change record shows Apply a full-record DISTINCT projection to the intended columns. The acceptance test concerns: A raw events dataset must remove exact duplicate records without altering other fields. Which outcome confirms it?
Explanation
Correct Answer
The acceptance criterion for “A raw events dataset must remove exact duplicate records without altering other fields” is Identical duplicate rows collapse to one while distinct rows stay; this is the expected evidence after Apply a full-record DISTINCT projection to the intended columns.
Incorrect Answers
Only the highest-ordered event row per customer is selected would demonstrate the separate action Use `ROW_NUMBER()` with `PARTITION BY customer_id` and deterministic order, which addresses “Per-customer latest event is needed while retaining the full winning row.” It does not verify the requirement “A raw events dataset must remove exact duplicate records without altering other fields” following Apply a full-record DISTINCT projection to the intended columns.
The output includes customers with null transaction values would demonstrate the separate action Use a LEFT JOIN from customers to transaction aggregates, which addresses “Customers with no transactions must remain in a customer enrichment result.” It does not verify the requirement “A raw events dataset must remove exact duplicate records without altering other fields” following Apply a full-record DISTINCT projection to the intended columns.
This signal belongs to the adjacent case “Records need to be ranked within each region without collapsing individual rows,” whose remedy is Use a ranking window with `PARTITION BY region`. The observed result for the present requirement is Identical duplicate rows collapse to one while distinct rows stay.