Question 650
A deployment handles more queries per second after scaling but response time also rises beyond policy. How should it be evaluated?
Explanation
Correct Answer
Higher throughput is not sufficient when the service violates its tail response-time requirements.
Incorrect Answers
The alternative, Track incoming request rate and completed token throughput together, belongs to a different technical condition: An operator wants to know whether a model throughput decline reflects low traffic or reduced processing capacity. There the source-specific mechanism or proof is: Distinguishing demand from completed work helps identify saturation rather than assume lower activity means a healthy service. The condition asked about here is: A deployment handles more queries per second after scaling but response time also rises beyond policy. A suitable correction is Compare throughput and latency together against the declared SLA; the targeted justification or result is: Higher throughput is not sufficient when the service violates its tail response-time requirements.
Measure agent tool, retrieval, orchestration, and model-call stages separately. That operation would be relevant if: An agent takes 10 seconds to answer while the foundation-model call itself lasts only one second. Its corresponding reason or verification is: End-to-end agent latency can be dominated by external operations that model-only timing does not include. However the problem in this question is: A deployment handles more queries per second after scaling but response time also rises beyond policy. The requirement instead calls for Compare throughput and latency together against the declared SLA; the question-specific check is: Higher throughput is not sufficient when the service violates its tail response-time requirements.
Measure end-to-end latency from callers in relevant regions to approved model locations. A separate source case supports that approach when: A model deployment in a distant region adds network delay for a local application. The supporting rationale or observable result is: Network round-trip time contributes to response speed independently of model computation. This question describes a different issue: A deployment handles more queries per second after scaling but response time also rises beyond policy. Address it through Compare throughput and latency together against the declared SLA, with the required justification or evidence: Higher throughput is not sufficient when the service violates its tail response-time requirements.