Question 200
The reliability team at Silverline Insurance is closing a corrective-action ticket for this ML failure: A scikit-learn prototype reports unexpectedly high validation scores because its scaler is fit on the combined training and validation data. What measured outcome provides the strongest check that this problem is resolved?
Explanation
Correct Answer
The experiment trace shows scaler fit only on training folds and validation scores are recomputed without leakage. This controlled observation specifically validates the proposed remedy: Place preprocessing inside a train-only fitted pipeline and transform held-out data using the frozen fitted steps. It addresses the documented failure at Silverline Insurance: A scikit-learn prototype reports unexpectedly high validation scores because its scaler is fit on the combined training and validation data.
Incorrect Answers
A share-and-export inspection finds no embedded secret while the authorized notebook can still access approved resources. This is evidence for a different corrective change: Store secrets in an approved secret-management system, redact outputs, and restrict Colab Enterprise or Workbench sharing. That other change addresses: A collaboration notebook contains raw credentials in its saved cell outputs and is shared beyond the experiment group. It does not validate the specific Silverline Insurance remediation for: A scikit-learn prototype reports unexpectedly high validation scores because its scaler is fit on the combined training and validation data. The necessary check instead is: The experiment trace shows scaler fit only on training folds and validation scores are recomputed without leakage.
A changed input shape fails the local check before a long training job starts and supported shapes complete a training step. This is evidence for a different corrective change: Add explicit sample, batch-shape, and training-step checks to the notebook before comparing model variants. That other change addresses: A PyTorch prototype produces different tensor shapes after a model upgrade, but its notebook has no regression checks around data loaders. It does not validate the specific Silverline Insurance remediation for: A scikit-learn prototype reports unexpectedly high validation scores because its scaler is fit on the combined training and validation data. The necessary check instead is: The experiment trace shows scaler fit only on training folds and validation scores are recomputed without leakage.
Two fresh notebook sessions resolve the same package versions and pass the model's reproducibility tests. This is evidence for a different corrective change: Pin framework and dependency versions in a reproducible environment manifest and test upgrades separately. That other change addresses: A prototype notebook automatically installs the newest PyTorch version, causing inconsistent operators and training results between sessions. It does not validate the specific Silverline Insurance remediation for: A scikit-learn prototype reports unexpectedly high validation scores because its scaler is fit on the combined training and validation data. The necessary check instead is: The experiment trace shows scaler fit only on training folds and validation scores are recomputed without leakage.