Question 500
Following a proposed change in a decision-support initiative at Eastshore Agriculture, the commercial systems team reports that it addressed the following need under restricted source access: A chatbot ignores tone and output requirements despite the task being simple and well documented. Which observation provides the STRONGEST evidence that the change worked?
Explanation
Correct Answer
A test suite shows more consistent style and format using the revised prompt alone. This is the direct observable test of the intended response: Improve prompt instructions and output constraints before pursuing expensive retraining. In the context of Eastshore Agriculture's a decision-support initiative, it checks an actual outcome instead of treating completion of a project ticket or an untested assertion as evidence.
Incorrect Answers
Security scans and deployment records demonstrate fixes under the defined maintenance window. This would instead validate the different situation: A deployed AI application accumulates unpatched dependencies and a known exposed endpoint. It would ordinarily follow the action: Maintain a patch and vulnerability process across the application and model-serving stack. It does not establish that Eastshore Agriculture solved the need specified in this scenario; the relevant check is: A test suite shows more consistent style and format using the revised prompt alone.
A held-out factual test shows unsupported citations are rejected and valid claims link to sources. This would instead validate the different situation: A policy assistant confidently invents regulation sections when its model lacks the facts. It would ordinarily follow the action: Ground responses in approved regulatory sources and require traceable evidence or abstention. It does not establish that Eastshore Agriculture solved the need specified in this scenario; the relevant check is: A test suite shows more consistent style and format using the revised prompt alone.
The attack suite records blocked data exfiltration and unauthorized tool attempts. This would instead validate the different situation: A red-team exercise finds prompts that request secrets or unsafe internal actions. It would ordinarily follow the action: Test adversarial prompts and enforce access controls, tool permissions, and output safety rules. It does not establish that Eastshore Agriculture solved the need specified in this scenario; the relevant check is: A test suite shows more consistent style and format using the revised prompt alone.