Question 170
The reliability team at Mercury Travel Tech is closing a corrective-action ticket for this ML failure: A clickstream ingestion job double-counts purchases whenever the publisher retries a delivered event with the same event identifier. Which independently observable result best verifies that the remediation works?
Explanation
Correct Answer
Replayed input produces one canonical purchase event and unchanged feature totals on repeated runs. This controlled observation specifically validates the proposed remedy: Deduplicate by stable event identity and event-version policy before constructing training features. It addresses the documented failure at Mercury Travel Tech: A clickstream ingestion job double-counts purchases whenever the publisher retries a delivered event with the same event identifier.
Incorrect Answers
Training records carry nonreversible working tokens and permission tests prevent analysts from reading raw identifiers. This is evidence for a different corrective change: Tokenize or pseudonymize sensitive identifiers and restrict the reversible mapping to a separately authorized service. That other change addresses: A lending analyst copies raw national identifiers into shared training tables even though the model needs only longitudinal grouping. It does not validate the specific Mercury Travel Tech remediation for: A clickstream ingestion job double-counts purchases whenever the publisher retries a delivered event with the same event identifier. The necessary check instead is: Replayed input produces one canonical purchase event and unchanged feature totals on repeated runs.
Historical replay retrieves status as it was at each event time, rather than the latest account record. This is evidence for a different corrective change: Use event-time or point-in-time feature retrieval against effective-dated feature values. That other change addresses: A recommendation model joins current customer status to old events, creating implausible historical feature values from updated records. It does not validate the specific Mercury Travel Tech remediation for: A clickstream ingestion job double-counts purchases whenever the publisher retries a delivered event with the same event identifier. The necessary check instead is: Replayed input produces one canonical purchase event and unchanged feature totals on repeated runs.
A temporal audit finds zero post-cutoff feature values in training and online replay samples. This is evidence for a different corrective change: Implement point-in-time feature joins that exclude signals produced after each prediction cutoff. That other change addresses: An account fraud model joins labels to activity from after the prediction timestamp, accidentally giving the model future knowledge. It does not validate the specific Mercury Travel Tech remediation for: A clickstream ingestion job double-counts purchases whenever the publisher retries a delivered event with the same event identifier. The necessary check instead is: Replayed input produces one canonical purchase event and unchanged feature totals on repeated runs.