Question 640
For Silverline Insurance, the engineer must verify a remediation with measured evidence, not just a deployment claim. The original failure was: A pipeline deploys any trained candidate even when its held-out performance is worse than the currently approved model. What measured outcome provides the strongest check that this problem is resolved?
Explanation
Correct Answer
An intentionally degraded candidate is blocked while a passing candidate advances with attached metrics. This controlled observation specifically validates the proposed remedy: Add a model evaluation component that compares frozen test metrics to champion and acceptance thresholds. It addresses the documented failure at Silverline Insurance: A pipeline deploys any trained candidate even when its held-out performance is worse than the currently approved model.
Incorrect Answers
A scheduled DAG respects ingestion completion, approval dependencies, and recoverable task retries. This is evidence for a different corrective change: Use Managed Service for Apache Airflow when cross-system DAG orchestration and scheduling requirements justify it. That other change addresses: A data-platform team must coordinate ML jobs with enterprise ingestion schedules and approval tasks spanning several managed services. It does not validate the specific Silverline Insurance remediation for: A pipeline deploys any trained candidate even when its held-out performance is worse than the currently approved model. The necessary check instead is: An intentionally degraded candidate is blocked while a passing candidate advances with attached metrics.
An injected sequence of temporary throttles eventually succeeds, while persistent failure exits within policy bounds. This is evidence for a different corrective change: Apply bounded exponential backoff and retry policies for transient errors while preserving terminal failure detection. That other change addresses: An external training-data API frequently returns temporary rate limits, and the pipeline treats every 429 response as a permanent failure. It does not validate the specific Silverline Insurance remediation for: A pipeline deploys any trained candidate even when its held-out performance is worse than the currently approved model. The necessary check instead is: An intentionally degraded candidate is blocked while a passing candidate advances with attached metrics.
A deliberately altered online scale fails the release gate and the correct transform passes. This is evidence for a different corrective change: Add automated train-serving feature parity checks using a shared golden fixture set as a release gate. That other change addresses: A deployed model's online preprocessing changes independently from training transformations and silently alters numeric feature scaling. It does not validate the specific Silverline Insurance remediation for: A pipeline deploys any trained candidate even when its held-out performance is worse than the currently approved model. The necessary check instead is: An intentionally degraded candidate is blocked while a passing candidate advances with attached metrics.