Question 580
An evaluation dataset mixes tool-call traces, user messages, and generated answers into a single untyped column. What should be changed?
Explanation
Correct Answer
Evaluators need consistent typed inputs; ambiguous columns make scores misleading or cause validation failures.
Incorrect Answers
Build a representative evaluation dataset including ordinary and ambiguous high-risk user intents. A separate source case supports that approach when: A RAG assistant is evaluated on only easy factual queries even though it also handles ambiguous refund requests. The supporting rationale or observable result is: Test cases must reflect the variety of real requests; easy-only evaluation masks failures on important edge cases. This question describes a different issue: An evaluation dataset mixes tool-call traces, user messages, and generated answers into a single untyped column. Address it through Map each field to its documented query, response, context, and tool metadata role, with the required justification or evidence: Evaluators need consistent typed inputs; ambiguous columns make scores misleading or cause validation failures.
Version the evaluation documents and record the exact policy snapshot used for scoring. This is useful for the distinct issue: A Foundry evaluation repeatedly uses stale customer policies despite new production document versions. The relevant technical rationale or evidence for that issue is: Document provenance makes results comparable and prevents outdated reference content from masquerading as model failure. Here the reported issue is: An evaluation dataset mixes tool-call traces, user messages, and generated answers into a single untyped column. The relevant response is: Map each field to its documented query, response, context, and tool metadata role. The first result cannot verify this different correction: Evaluators need consistent typed inputs; ambiguous columns make scores misleading or cause validation failures.
Use a separate held-out test set not consulted during prompt optimization. It addresses another case: A release candidate is judged on the same examples used repeatedly to tune its prompt. For that case, the distinguishing justification or acceptance check is: An untouched holdout reduces the chance that a prompt is approved merely for memorizing known evaluation examples. The present case instead states: An evaluation dataset mixes tool-call traces, user messages, and generated answers into a single untyped column. It calls for Map each field to its documented query, response, context, and tool metadata role. Showing success for the other case would not establish: Evaluators need consistent typed inputs; ambiguous columns make scores misleading or cause validation failures.