Question 550
An evaluation selects the best prompt after repeatedly tuning against the same 20 examples. What guards against overfitting?
Explanation
Correct Answer
Repeated tuning can memorize evaluation examples; an independent holdout provides more realistic generalization evidence.
Incorrect Answers
Include prompt-injection and untrusted-context cases in the variant test set. It addresses another case: A new prompt's evaluation dataset omits adversarial instructions embedded in retrieved documents. For that case, the distinguishing justification or acceptance check is: Security-relevant prompt changes need adversarial coverage, not only ordinary benign user questions. The present case instead states: An evaluation selects the best prompt after repeatedly tuning against the same 20 examples. It calls for Validate the selected prompt on a separate untouched test set. Showing success for the other case would not establish: Repeated tuning can memorize evaluation examples; an independent holdout provides more realistic generalization evidence.
The alternative, Use a broad regression dataset weighted to real workload distribution and critical edge cases, belongs to a different technical condition: A prompt candidate fixes one high-profile complaint but degrades several routine intents. There the source-specific mechanism or proof is: An isolated success case is insufficient evidence that overall prompt behavior remains acceptable. The condition asked about here is: An evaluation selects the best prompt after repeatedly tuning against the same 20 examples. A suitable correction is Validate the selected prompt on a separate untouched test set; the targeted justification or result is: Repeated tuning can memorize evaluation examples; an independent holdout provides more realistic generalization evidence.
Evaluate both variants against the same representative labeled evaluation cases. That operation would be relevant if: Two prompt candidates are tested on unrelated user questions, and the team compares average accuracy. Its corresponding reason or verification is: Paired inputs make outcome differences attributable to the prompt rather than changing test-set difficulty. However the problem in this question is: An evaluation selects the best prompt after repeatedly tuning against the same 20 examples. The requirement instead calls for Validate the selected prompt on a separate untouched test set; the question-specific check is: Repeated tuning can memorize evaluation examples; an independent holdout provides more realistic generalization evidence.