Question 740
At Riverton Utilities, the operations analytics engineers maintain smart-meter demand prediction and grid-event models under requirements for stable event-time features and isolated infrastructure. The ML reliability team requests proof of the remediation for: A production A/B test compares models on different customer cohorts without controlling allocation, making observed performance differences misleading. Which observed result BEST demonstrates that the correction works?
Explanation
Correct Answer
The assignment audit shows representative variant groups and the evaluation respects the intended statistical comparison design. This is observable proof for correction of: A production A/B test compares models on different customer cohorts without controlling allocation, making observed performance differences misleading. The approved design change is: Use consistent random traffic allocation and analyze outcomes per experimental assignment and predefined business slices. The Riverton Utilities acceptance test separates measured ML behavior from an unverified implementation claim.
Incorrect Answers
New output artifacts have the documented KMS key metadata and an attempted unencrypted write is rejected by bucket policy. This would verify a different operation: Require appropriate SSE-KMS settings on the output bucket and ensure the training role is authorized to use the chosen key. That separate action addresses: A model training job writes regulated outputs to an unencrypted S3 location despite a documented data-at-rest encryption standard. It cannot establish the required Riverton Utilities correction for: A production A/B test compares models on different customer cohorts without controlling allocation, making observed performance differences misleading. The decisive acceptance check is instead: The assignment audit shows representative variant groups and the evaluation respects the intended statistical comparison design.
A low-traffic replay meets p95 response requirements and documents the cost of maintaining necessary warm capacity. This would verify a different operation: Choose a suitable minimum serving capacity or compatible hosting method and validate the idle-cost versus cold-start tradeoff. That separate action addresses: A model endpoint scales to zero immediately after every request but subsequent cold starts repeatedly violate the low-latency objective. It cannot establish the required Riverton Utilities correction for: A production A/B test compares models on different customer cohorts without controlling allocation, making observed performance differences misleading. The decisive acceptance check is instead: The assignment audit shows representative variant groups and the evaluation respects the intended statistical comparison design.
A production task retrieves its secret with the authorized role while build logs reveal no plaintext credential. This would verify a different operation: Use AWS Secrets Manager or an approved secret provider with a scoped runtime role and suppress secret values from pipeline logs. That separate action addresses: An automated deployment retrieves a production database password from a plaintext environment variable stored in a shared build log. It cannot establish the required Riverton Utilities correction for: A production A/B test compares models on different customer cohorts without controlling allocation, making observed performance differences misleading. The decisive acceptance check is instead: The assignment audit shows representative variant groups and the evaluation respects the intended statistical comparison design.