Exam Bundle

The test matrix isolates the marginal impact of the feature and optimizer changes without conflating their effects. This would verify a different operation: Run controlled ablations or reproducible one-factor comparisons while holding other settings fixed. That separate action addresses: A model accuracy experiment introduces a new feature and several new hyperparameters at once, preventing attribution of the observed gain. It cannot establish the required Granite SaaS correction for: A computer-vision model fails on slightly rotated images even though real deployment images naturally vary in orientation within valid limits. The decisive acceptance check is instead: The robustness benchmark improves on legitimate rotated examples while clean-image accuracy remains above its acceptance floor.

The selected approach meets the required recall and precision tradeoff on the untouched production-like class distribution. This would verify a different operation: Select a classifier and decision threshold based on false-negative cost and class-aware metrics rather than majority-class accuracy. That separate action addresses: A defect prediction project has very few confirmed positive labels and proposes to optimize raw accuracy on an unbalanced evaluation dataset. It cannot establish the required Granite SaaS correction for: A computer-vision model fails on slightly rotated images even though real deployment images naturally vary in orientation within valid limits. The decisive acceptance check is instead: The robustness benchmark improves on legitimate rotated examples while clean-image accuracy remains above its acceptance floor.

An auditor can reproduce the compared configurations and identify the changed input or parameter causing the metric difference. This would verify a different operation: Track training jobs, hyperparameters, dataset manifests and metrics in a reproducible SageMaker experiment catalog. That separate action addresses: A model's validation score degrades between two SageMaker runs but the experiment records lack parameter and dataset lineage. It cannot establish the required Granite SaaS correction for: A computer-vision model fails on slightly rotated images even though real deployment images naturally vary in orientation within valid limits. The decisive acceptance check is instead: The robustness benchmark improves on legitimate rotated examples while clean-image accuracy remains above its acceptance floor.

Add macro-averaged and per-class precision-recall reporting alongside micro metrics for the model acceptance decision. It does not directly address the Bluehaven Transit case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Evaluate an unsupervised clustering approach and select a suitable distance representation before proposing any business segment labels. It does not directly address the Bluehaven Transit case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Include inference latency, compute cost and model size in a feasibility benchmark against smaller compatible candidates. It does not directly address the Bluehaven Transit case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

The experiment log proves whether predictions were merely observed or whether an authorized portion of traffic received candidate outputs. This would verify a different operation: Differentiate shadow validation from controlled live traffic allocation and select the appropriate comparison procedure. That separate action addresses: An ML team labels a production shadow experiment as an A/B test even though the shadow predictions never influence user-facing decisions. It cannot establish the required Bluehaven Transit correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

The candidate evaluation demonstrates compliant request routing and authorized data handling as well as acceptable answer quality. This would verify a different operation: Compare model and deployment options against data residency, access policy and cost constraints before deciding on Bedrock or a supported hosted alternative. That separate action addresses: An inference team chooses a managed language model with the largest context window even though the protected request content cannot leave its designated processing environment. It cannot establish the required Bluehaven Transit correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

All training timestamps precede each evaluation window and the aggregated backtest reflects the real forecast schedule. This would verify a different operation: Use rolling-origin or other chronological backtesting splits appropriate to the prediction horizon. That separate action addresses: A time-series forecast is assessed using random record-level cross-validation that trains on observations later than the validation period. It cannot establish the required Bluehaven Transit correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Evaluate both candidate models on the same versioned untouched holdout dataset with identical metric definitions. It does not directly address the Brookside Manufacturing case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Compute a class confusion matrix and inspect high-confusion pairs with representative labeled examples. It does not directly address the Brookside Manufacturing case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Select an appropriate regression or time-series modeling approach aligned to the continuous demand target and its evaluation horizon. It does not directly address the Brookside Manufacturing case: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The relevant acceptance criterion for this case is: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

The candidate evaluation demonstrates compliant request routing and authorized data handling as well as acceptable answer quality. This would verify a different operation: Compare model and deployment options against data residency, access policy and cost constraints before deciding on Bedrock or a supported hosted alternative. That separate action addresses: An inference team chooses a managed language model with the largest context window even though the protected request content cannot leave its designated processing environment. It cannot establish the required Brookside Manufacturing correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

The held-out chronological evaluation shows whether the model benefits from the actual interaction order. This would verify a different operation: Evaluate a sequence-aware recommendation or ranking approach and compare it with a simpler nonsequential baseline. That separate action addresses: A recommendation model treats independently ordered shopping events as a flat feature bag even though the prediction depends on recent interaction sequence. It cannot establish the required Brookside Manufacturing correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

The chosen clustering method forms stable explainable groups and holds out representative customers for business validation. This would verify a different operation: Evaluate an unsupervised clustering approach and select a suitable distance representation before proposing any business segment labels. That separate action addresses: A retailer wants segments of similar shoppers without an existing definition of customer groups but configures an ordinary labeled classifier. It cannot establish the required Brookside Manufacturing correction for: A model training run terminates because it assumes access to external Python package indexes blocked by the organization's isolated VPC. The decisive acceptance check is instead: A clean isolated training job starts without public internet egress and logs the expected dependency versions.

Include interpretability and governance constraints in model selection and compare explainable models or validated explanation techniques. It does not directly address the Harborline Logistics case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Evaluate reliability diagrams and use an appropriate validation-fitted probability calibration method if needed. It does not directly address the Harborline Logistics case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Use suitable regression metrics such as MAE and RMSE with error units and compare against a simple business baseline. It does not directly address the Harborline Logistics case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

The experiment log proves whether predictions were merely observed or whether an authorized portion of traffic received candidate outputs. This would verify a different operation: Differentiate shadow validation from controlled live traffic allocation and select the appropriate comparison procedure. That separate action addresses: An ML team labels a production shadow experiment as an A/B test even though the shadow predictions never influence user-facing decisions. It cannot establish the required Harborline Logistics correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

A realistic concurrency test meets the latency SLA and the final choice documents the measured accuracy-cost tradeoff. This would verify a different operation: Include inference latency, compute cost and model size in a feasibility benchmark against smaller compatible candidates. That separate action addresses: A fraud detection service requires millisecond decisions but a very large transformer is selected solely because its offline F1 score is highest. It cannot establish the required Harborline Logistics correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

The held-out chronological evaluation shows whether the model benefits from the actual interaction order. This would verify a different operation: Evaluate a sequence-aware recommendation or ranking approach and compare it with a simpler nonsequential baseline. That separate action addresses: A recommendation model treats independently ordered shopping events as a flat feature bag even though the prediction depends on recent interaction sequence. It cannot establish the required Harborline Logistics correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Evaluate a sequence-aware recommendation or ranking approach and compare it with a simpler nonsequential baseline. It does not directly address the Northgate Telecom case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Interpret the train-validation generalization gap as likely overfitting and compare regularization or data improvements first. It does not directly address the Northgate Telecom case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Evaluate an appropriate generative foundation model or managed AI capability while constraining input length, retrieval and output validation. It does not directly address the Northgate Telecom case: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The relevant acceptance criterion for this case is: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

The final report identifies the confusing class pairs and tracks their individual error rates after model changes. This would verify a different operation: Compute a class confusion matrix and inspect high-confusion pairs with representative labeled examples. That separate action addresses: An image classifier's evaluation logs only overall accuracy while it frequently misclassifies visually similar protected categories. It cannot establish the required Northgate Telecom correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

The report shows every priority class's performance and a rare-category failure cannot be masked by majority volume. This would verify a different operation: Add macro-averaged and per-class precision-recall reporting alongside micro metrics for the model acceptance decision. That separate action addresses: A product classifier reports a high micro-averaged F1 but hides very poor performance on a critical rare product category. It cannot establish the required Northgate Telecom correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Holdout predictions include usable value estimates and documented coverage behavior for uncertainty intervals. This would verify a different operation: Choose a calibrated regression or probabilistic forecasting method compatible with the numeric target and uncertainty reporting requirement. That separate action addresses: A pricing model must output predicted revenue with confidence bounds, but the team selects an algorithm intended solely to rank products. It cannot establish the required Northgate Telecom correction for: A fine-tuning experiment silently changes the pretrained tokenizer but retains the old embedding layer and produces inconsistent token indices. The decisive acceptance check is instead: A fixed sample tokenizes and passes through the fine-tuned model without index errors or mismatched vocabulary metadata.

Compare Amazon Rekognition's supported capabilities against the need for custom features before committing to training. It does not directly address the Pinecrest Hotels case: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The relevant acceptance criterion for this case is: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.

Choose an unsupervised anomaly-detection approach suited to the available feature data and explicitly define anomaly review outcomes. It does not directly address the Pinecrest Hotels case: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The relevant acceptance criterion for this case is: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.

Use rolling-origin or other chronological backtesting splits appropriate to the prediction horizon. It does not directly address the Pinecrest Hotels case: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The relevant acceptance criterion for this case is: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.

An audited holdout set shows meaningful semantic grouping and the chosen model's distance assumptions are documented. This would verify a different operation: Compare a suitable text vector representation and model family using relevant similarity measures and unsupervised evaluation criteria. That separate action addresses: A classification dataset contains many thousands of sparse token features, and teams choose k-means with an inappropriate raw Euclidean representation. It cannot establish the required Pinecrest Hotels correction for: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The decisive acceptance check is instead: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.

The candidate evaluation demonstrates compliant request routing and authorized data handling as well as acceptable answer quality. This would verify a different operation: Compare model and deployment options against data residency, access policy and cost constraints before deciding on Bedrock or a supported hosted alternative. That separate action addresses: An inference team chooses a managed language model with the largest context window even though the protected request content cannot leave its designated processing environment. It cannot establish the required Pinecrest Hotels correction for: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The decisive acceptance check is instead: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.

The paired model comparison uses matching sample IDs and reproducible evaluation settings for both candidates. This would verify a different operation: Evaluate both candidate models on the same versioned untouched holdout dataset with identical metric definitions. That separate action addresses: An ML review compares two model variants using different holdout sets and claims the one with higher score is superior. It cannot establish the required Pinecrest Hotels correction for: A regression model uses squared-error loss despite severe outlier noise dominating its updates and producing unstable fits. The decisive acceptance check is instead: Holdout errors on normal and exceptional records are compared explicitly and the selected loss meets the business tolerance.