Exam Bundle

A developer outside the release role cannot replace the artifact, while the expected digest loads successfully. This is evidence for a different corrective change: Enforce least-privilege artifact permissions, immutable versioning, and integrity checks before model deployment. That other change addresses: A trained model is saved in a broadly writable bucket where unrelated developers can replace weights without authorization. It does not validate the specific Silverline Insurance remediation for: A decision model provides probability scores but the governance report offers no reproducible local explanation for an adverse case. The necessary check instead is: An authorized reviewer reproduces a local feature contribution for a frozen case without treating it as causal proof.

A forged policy document is rejected or quarantined and the agent cites only authorized versioned sources. This is evidence for a different corrective change: Validate trusted document provenance, restrict source updates, and record retrieval lineage before ingesting new context. That other change addresses: A question-answering agent indexes mutable policy documents without content verification, allowing poisoned text to contaminate answers. It does not validate the specific Silverline Insurance remediation for: A decision model provides probability scores but the governance report offers no reproducible local explanation for an adverse case. The necessary check instead is: An authorized reviewer reproduces a local feature contribution for a frozen case without treating it as causal proof.

Held-out subgroup results meet the stated fairness guardrails without hiding deterioration in overall utility. This is evidence for a different corrective change: Measure group-sliced error and fairness metrics, investigate feature bias, and enforce an approved remediation process. That other change addresses: A hiring predictor has good total accuracy but systematically rejects qualified applicants from one audited group. It does not validate the specific Silverline Insurance remediation for: A decision model provides probability scores but the governance report offers no reproducible local explanation for an adverse case. The necessary check instead is: An authorized reviewer reproduces a local feature contribution for a frozen case without treating it as causal proof.

Minimize and redact sensitive prompt fields before inference, with access-control and retention policies applied. This action could be suitable for a different engineering incident: A customer assistant includes complete account identifiers in LLM prompts although the response requires only purchase category. It does not directly correct the reported Northgate Clinical Systems failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Treat retrieved content as untrusted and enforce tool authorization, instruction hierarchy, and prompt-injection defenses. This action could be suitable for a different engineering incident: A retrieval-augmented support bot obeys a malicious instruction hidden inside a retrieved webpage to reveal internal prompts. It does not directly correct the reported Northgate Clinical Systems failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Pin reviewed container images and dependencies, verify provenance signatures or digests, and scan for known risks. This action could be suitable for a different engineering incident: A team deploys a model package from an unsigned third-party image with dependencies downloaded at runtime from uncontrolled mirrors. It does not directly correct the reported Northgate Clinical Systems failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Captured test requests contain only approved data fields and a secret-marker test never appears in model inputs. This is evidence for a different corrective change: Minimize and redact sensitive prompt fields before inference, with access-control and retention policies applied. That other change addresses: A customer assistant includes complete account identifiers in LLM prompts although the response requires only purchase category. It does not validate the specific Northgate Clinical Systems remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

A tampered artifact is refused at release and the accepted image matches the approved supply-chain manifest. This is evidence for a different corrective change: Pin reviewed container images and dependencies, verify provenance signatures or digests, and scan for known risks. That other change addresses: A team deploys a model package from an unsigned third-party image with dependencies downloaded at runtime from uncontrolled mirrors. It does not validate the specific Northgate Clinical Systems remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

A red-team retrieved document fails to induce prohibited disclosure or tool use while legitimate questions still work. This is evidence for a different corrective change: Treat retrieved content as untrusted and enforce tool authorization, instruction hierarchy, and prompt-injection defenses. That other change addresses: A retrieval-augmented support bot obeys a malicious instruction hidden inside a retrieved webpage to reveal internal prompts. It does not validate the specific Northgate Clinical Systems remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Use Secret Manager or a secured secret-injection workflow and revoke exposed credentials. This action could be suitable for a different engineering incident: A pipeline embeds production database passwords in plain text inside model-training container environment files committed to source control. It does not directly correct the reported Stonewell Payments failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Set data-minimizing storage and deletion policies across prompt, retrieval, logging, and model telemetry systems. This action could be suitable for a different engineering incident: A legal-assistance assistant stores personal prompts indefinitely although contractual retention limits require rapid deletion. It does not directly correct the reported Stonewell Payments failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Document feature attributions as model-behavior explanations and require separate causal evidence for intervention claims. This action could be suitable for a different engineering incident: A business team presents feature attribution values from an opaque model as mathematical proof that changing the top-ranked attribute will change an outcome. It does not directly correct the reported Stonewell Payments failure: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The discriminating acceptance check here is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

The model card labels explanatory associations accurately and rejects unsupported causal conclusions. This is evidence for a different corrective change: Document feature attributions as model-behavior explanations and require separate causal evidence for intervention claims. That other change addresses: A business team presents feature attribution values from an opaque model as mathematical proof that changing the top-ranked attribute will change an outcome. It does not validate the specific Stonewell Payments remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

A retention test confirms expired personal prompts are deleted while required non-sensitive audit markers remain. This is evidence for a different corrective change: Set data-minimizing storage and deletion policies across prompt, retrieval, logging, and model telemetry systems. That other change addresses: A legal-assistance assistant stores personal prompts indefinitely although contractual retention limits require rapid deletion. It does not validate the specific Stonewell Payments remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

A repository scan finds no active plaintext secret and the authorized training task obtains it only at execution time. This is evidence for a different corrective change: Use Secret Manager or a secured secret-injection workflow and revoke exposed credentials. That other change addresses: A pipeline embeds production database passwords in plain text inside model-training container environment files committed to source control. It does not validate the specific Stonewell Payments remediation for: An autonomous workflow agent can invoke billing exports using the same highly privileged identity it uses for ordinary summarization. The necessary check instead is: A harmless summary succeeds but a disallowed export tool call is rejected and logged.

Version the evaluator model, rubric, and prompt and recalibrate against fixed human-rated anchor examples. This action could be suitable for a different engineering incident: An automated LLM-as-a-judge metric suddenly improves after a prompt template change, making historical quality trends incomparable. It does not directly correct the reported Harbor Rail Logistics failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Compare representative training and serving feature statistics and preprocessing parity for training-serving skew. This action could be suitable for a different engineering incident: A production model receives amounts normalized differently than the training pipeline, but monitoring checks only prediction latency. It does not directly correct the reported Harbor Rail Logistics failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Monitor input/output token use, price assumptions, and per-task spend by model and prompt-template version. This action could be suitable for a different engineering incident: A foundation-model API becomes expensive after a prompt template adds large redundant context, but no cost metric is tied to release versions. It does not directly correct the reported Harbor Rail Logistics failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

The amended template produces an attributable cost alert while answer quality remains separately tracked. This is evidence for a different corrective change: Monitor input/output token use, price assumptions, and per-task spend by model and prompt-template version. That other change addresses: A foundation-model API becomes expensive after a prompt template adds large redundant context, but no cost metric is tied to release versions. It does not validate the specific Harbor Rail Logistics remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

A deliberately changed normalization factor is detected by the skew check before widespread rollout. This is evidence for a different corrective change: Compare representative training and serving feature statistics and preprocessing parity for training-serving skew. That other change addresses: A production model receives amounts normalized differently than the training pipeline, but monitoring checks only prediction latency. It does not validate the specific Harbor Rail Logistics remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Anchor scores reveal and correct judge-template drift before quality dashboards compare periods. This is evidence for a different corrective change: Version the evaluator model, rubric, and prompt and recalibrate against fixed human-rated anchor examples. That other change addresses: An automated LLM-as-a-judge metric suddenly improves after a prompt template change, making historical quality trends incomparable. It does not validate the specific Harbor Rail Logistics remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Track feature attribution distributions and investigate large attribution drift alongside data and quality metrics. This action could be suitable for a different engineering incident: A classifier's top explanatory features change sharply after an upstream data-source update although overall model input averages look similar. It does not directly correct the reported Aster Pharma failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Collect per-version request latency percentiles and alert on the defined p99 SLO at peak throughput. This action could be suitable for a different engineering incident: A recommendation endpoint meets uptime checks but p99 latency exceeds its service objective during nightly peak demand. It does not directly correct the reported Aster Pharma failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Compare representative training and serving feature statistics and preprocessing parity for training-serving skew. This action could be suitable for a different engineering incident: A production model receives amounts normalized differently than the training pipeline, but monitoring checks only prediction latency. It does not directly correct the reported Aster Pharma failure: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The discriminating acceptance check here is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

An injected upstream mapping change raises an attribution-drift signal tied to the affected model version. This is evidence for a different corrective change: Track feature attribution distributions and investigate large attribution drift alongside data and quality metrics. That other change addresses: A classifier's top explanatory features change sharply after an upstream data-source update although overall model input averages look similar. It does not validate the specific Aster Pharma remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

A replayed peak raises the expected latency alert and isolates the serving-version bottleneck. This is evidence for a different corrective change: Collect per-version request latency percentiles and alert on the defined p99 SLO at peak throughput. That other change addresses: A recommendation endpoint meets uptime checks but p99 latency exceeds its service objective during nightly peak demand. It does not validate the specific Aster Pharma remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

A deliberately changed normalization factor is detected by the skew check before widespread rollout. This is evidence for a different corrective change: Compare representative training and serving feature statistics and preprocessing parity for training-serving skew. That other change addresses: A production model receives amounts normalized differently than the training pipeline, but monitoring checks only prediction latency. It does not validate the specific Aster Pharma remediation for: A vision classifier's incoming product photos have shifted to a new lighting distribution but the monitoring dashboard tracks only endpoint uptime. The necessary check instead is: A synthetic lighting shift raises a drift alert while normal image mix stays below the configured threshold.

Add answer factuality and abstention tests with reference-grounded human or calibrated judge review. This action could be suitable for a different engineering incident: A Gemini-powered agent produces confident invented facts for questions absent from the internal knowledge corpus and is evaluated only for grammar. It does not directly correct the reported Brookfield Credit Union failure: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The discriminating acceptance check here is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.

Monitor subgroup-specific model quality and fairness thresholds with adequate sample-size disclosure. This action could be suitable for a different engineering incident: A medical triage model has stable overall AUROC while its recall for an underrepresented patient subgroup deteriorates. It does not directly correct the reported Brookfield Credit Union failure: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The discriminating acceptance check here is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.

Track safety violations and policy-required refusal rates on tagged production samples and adversarial regression suites. This action could be suitable for a different engineering incident: An AI chat service begins emitting unsafe responses after a rollout but its monitoring aggregates refusals with all normal completions. It does not directly correct the reported Brookfield Credit Union failure: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The discriminating acceptance check here is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.

Unknown-answer prompts produce acceptable abstentions and grounded questions are scored for citation fidelity. This is evidence for a different corrective change: Add answer factuality and abstention tests with reference-grounded human or calibrated judge review. That other change addresses: A Gemini-powered agent produces confident invented facts for questions absent from the internal knowledge corpus and is evaluated only for grammar. It does not validate the specific Brookfield Credit Union remediation for: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The necessary check instead is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.

A seeded unsafe-output regression crosses the safety threshold and halts further release traffic expansion. This is evidence for a different corrective change: Track safety violations and policy-required refusal rates on tagged production samples and adversarial regression suites. That other change addresses: An AI chat service begins emitting unsafe responses after a rollout but its monitoring aggregates refusals with all normal completions. It does not validate the specific Brookfield Credit Union remediation for: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The necessary check instead is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.

A controlled subgroup degradation triggers review despite unchanged aggregate AUROC. This is evidence for a different corrective change: Monitor subgroup-specific model quality and fairness thresholds with adequate sample-size disclosure. That other change addresses: A medical triage model has stable overall AUROC while its recall for an underrepresented patient subgroup deteriorates. It does not validate the specific Brookfield Credit Union remediation for: A fraud predictor's feature distribution remains broadly stable yet actual fraud patterns change and detection precision falls. The necessary check instead is: The monitored precision drop appears after labels mature even without a significant covariate-drift alert.