Question 620
A release pipeline reports overall evaluator averages but ignores a must-pass safety criterion. What should the gate require?
Explanation
Correct Answer
A passing average cannot compensate for a mandatory high-severity policy failure in a sensitive application.
Incorrect Answers
Define application-specific evaluation policies and thresholds alongside shared model checks. A separate source case supports that approach when: Two applications use the same Foundry model but different policy requirements, and one global evaluation threshold causes false approvals. The supporting rationale or observable result is: A model can be appropriate for one workflow but unacceptable for another with stricter requirements. This question describes a different issue: A release pipeline reports overall evaluator averages but ignores a must-pass safety criterion. Address it through Block promotion whenever any critical safety metric fails its approved threshold, with the required justification or evidence: A passing average cannot compensate for a mandatory high-severity policy failure in a sensitive application.
Run approved evaluation jobs as a required CI or release-workflow stage. This is useful for the distinct issue: A team manually runs Foundry quality evaluations before some releases but forgets after small prompt changes. The relevant technical rationale or evidence for that issue is: Automated gates apply the same checks to each relevant model or prompt version without relying on memory. Here the reported issue is: A release pipeline reports overall evaluator averages but ignores a must-pass safety criterion. The relevant response is: Block promotion whenever any critical safety metric fails its approved threshold. The first result cannot verify this different correction: A passing average cannot compensate for a mandatory high-severity policy failure in a sensitive application.
Make the release gate explicitly inspect evaluator results and fail when thresholds are breached. It addresses another case: A release pipeline runs model quality tests but automatically ignores every failed case because the evaluation command exits zero. For that case, the distinguishing justification or acceptance check is: Successful evaluator execution is not the same as acceptable evaluated model behavior. The present case instead states: A release pipeline reports overall evaluator averages but ignores a must-pass safety criterion. It calls for Block promotion whenever any critical safety metric fails its approved threshold. Showing success for the other case would not establish: A passing average cannot compensate for a mandatory high-severity policy failure in a sensitive application.