Exam Bundle

An audited holdout set shows meaningful semantic grouping and the chosen model's distance assumptions are documented. This would verify a different operation: Compare a suitable text vector representation and model family using relevant similarity measures and unsupervised evaluation criteria. That separate action addresses: A classification dataset contains many thousands of sparse token features, and teams choose k-means with an inappropriate raw Euclidean representation. It cannot establish the required Oakridge Public Services correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

The training trace identifies the instability window and a corrected run converges with finite monitored values. This would verify a different operation: Review optimizer, gradients and intermediate tensors using supported SageMaker Debugger or framework diagnostics. That separate action addresses: Training logs show unstable loss values and stalled convergence but reviewers look only at the final validation score rather than intermediate diagnostics. It cannot establish the required Oakridge Public Services correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

The report shows every priority class's performance and a rare-category failure cannot be masked by majority volume. This would verify a different operation: Add macro-averaged and per-class precision-recall reporting alongside micro metrics for the model acceptance decision. That separate action addresses: A product classifier reports a high micro-averaged F1 but hides very poor performance on a critical rare product category. It cannot establish the required Oakridge Public Services correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

Measure p95 and p99 latency with representative concurrency rather than relying on average inference time alone. It does not directly address the Harborview Insurance case: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The relevant acceptance criterion for this case is: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

Treat attributions as model-specific explanations and validate findings with controlled experiments or domain review before causal claims. It does not directly address the Harborview Insurance case: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The relevant acceptance criterion for this case is: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

Evaluate reliability diagrams and use an appropriate validation-fitted probability calibration method if needed. It does not directly address the Harborview Insurance case: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The relevant acceptance criterion for this case is: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

The candidate evaluation demonstrates compliant request routing and authorized data handling as well as acceptable answer quality. This would verify a different operation: Compare model and deployment options against data residency, access policy and cost constraints before deciding on Bedrock or a supported hosted alternative. That separate action addresses: An inference team chooses a managed language model with the largest context window even though the protected request content cannot leave its designated processing environment. It cannot establish the required Harborview Insurance correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

A realistic concurrency test meets the latency SLA and the final choice documents the measured accuracy-cost tradeoff. This would verify a different operation: Include inference latency, compute cost and model size in a feasibility benchmark against smaller compatible candidates. That separate action addresses: A fraud detection service requires millisecond decisions but a very large transformer is selected solely because its offline F1 score is highest. It cannot establish the required Harborview Insurance correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

The evaluation reveals positive-class misses and explicitly reports the accepted recall and false-positive tradeoff. This would verify a different operation: Use confusion-matrix precision, recall and F1 metrics with a business-defined threshold on a representative holdout set. That separate action addresses: A high-risk fraud classifier shows 99 percent accuracy by predicting the majority class and is approved without examining missed fraud cases. It cannot establish the required Harborview Insurance correction for: An ML engineer runs random search without bounding costly hyperparameters and repeatedly launches training jobs beyond the approved experiment budget. The decisive acceptance check is instead: The tuning experiment completes inside its resource cap and produces an auditable best-candidate comparison.

Treat attributions as model-specific explanations and validate findings with controlled experiments or domain review before causal claims. It does not directly address the Redstone Aviation case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

Use a separate validation set or cross-validation for tuning and reserve the final holdout for one fixed-model assessment. It does not directly address the Redstone Aviation case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

Include inference latency, compute cost and model size in a feasibility benchmark against smaller compatible candidates. It does not directly address the Redstone Aviation case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

A realistic concurrency test meets the latency SLA and the final choice documents the measured accuracy-cost tradeoff. This would verify a different operation: Include inference latency, compute cost and model size in a feasibility benchmark against smaller compatible candidates. That separate action addresses: A fraud detection service requires millisecond decisions but a very large transformer is selected solely because its offline F1 score is highest. It cannot establish the required Redstone Aviation correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

The selection report compares test metrics, training cost and operational complexity on a common fixed dataset. This would verify a different operation: Benchmark suitable tabular algorithms such as SageMaker XGBoost against the complex model using the same split and budget. That separate action addresses: An analyst trains a high-capacity neural network on a small, mostly numeric transaction table without first benchmarking simpler tabular methods. It cannot establish the required Redstone Aviation correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

A controlled reference request produces equivalent features in both pipelines and the corrected performance gap is reassessed. This would verify a different operation: Compare sampled training and serving feature transformations and distributions before concluding the model weights are defective. That separate action addresses: A model with good offline precision unexpectedly underperforms after deployment because the serving feature values are generated by a different pipeline. It cannot establish the required Redstone Aviation correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

Use suitable regression metrics such as MAE and RMSE with error units and compare against a simple business baseline. It does not directly address the Meadowbrook Finance case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

Compare Amazon Rekognition's supported capabilities against the need for custom features before committing to training. It does not directly address the Meadowbrook Finance case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

Evaluate reliability diagrams and use an appropriate validation-fitted probability calibration method if needed. It does not directly address the Meadowbrook Finance case: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The relevant acceptance criterion for this case is: A deployed model version resolves to its exact training job, data manifest and approved model package.

The final report identifies the confusing class pairs and tracks their individual error rates after model changes. This would verify a different operation: Compute a class confusion matrix and inspect high-confusion pairs with representative labeled examples. That separate action addresses: An image classifier's evaluation logs only overall accuracy while it frequently misclassifies visually similar protected categories. It cannot establish the required Meadowbrook Finance correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

Holdout predictions include usable value estimates and documented coverage behavior for uncertainty intervals. This would verify a different operation: Choose a calibrated regression or probabilistic forecasting method compatible with the numeric target and uncertainty reporting requirement. That separate action addresses: A pricing model must output predicted revenue with confidence bounds, but the team selects an algorithm intended solely to rank products. It cannot establish the required Meadowbrook Finance correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

The held-out chronological evaluation shows whether the model benefits from the actual interaction order. This would verify a different operation: Evaluate a sequence-aware recommendation or ranking approach and compare it with a simpler nonsequential baseline. That separate action addresses: A recommendation model treats independently ordered shopping events as a flat feature bag even though the prediction depends on recent interaction sequence. It cannot establish the required Meadowbrook Finance correction for: An organization selects a model solely from the latest notebook file and cannot identify which SageMaker training run produced the released weights. The decisive acceptance check is instead: A deployed model version resolves to its exact training job, data manifest and approved model package.

Choose a multi-label output design with independent label scores and threshold selection instead of single-class competition. It does not directly address the Maplewood Schools case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

Treat attributions as model-specific explanations and validate findings with controlled experiments or domain review before causal claims. It does not directly address the Maplewood Schools case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

Use confusion-matrix precision, recall and F1 metrics with a business-defined threshold on a representative holdout set. It does not directly address the Maplewood Schools case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

The candidate evaluation demonstrates compliant request routing and authorized data handling as well as acceptable answer quality. This would verify a different operation: Compare model and deployment options against data residency, access policy and cost constraints before deciding on Bedrock or a supported hosted alternative. That separate action addresses: An inference team chooses a managed language model with the largest context window even though the protected request content cannot leave its designated processing environment. It cannot establish the required Maplewood Schools correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

A diagnostic plot shows reduced validation loss without relying on increased training-only accuracy. This would verify a different operation: Interpret the train-validation generalization gap as likely overfitting and compare regularization or data improvements first. That separate action addresses: A model has strong training results but weak validation performance, yet the team calls this underfitting and increases its capacity. It cannot establish the required Maplewood Schools correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

The selection report compares test metrics, training cost and operational complexity on a common fixed dataset. This would verify a different operation: Benchmark suitable tabular algorithms such as SageMaker XGBoost against the complex model using the same split and budget. That separate action addresses: An analyst trains a high-capacity neural network on a small, mostly numeric transaction table without first benchmarking simpler tabular methods. It cannot establish the required Maplewood Schools correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

Treat attributions as model-specific explanations and validate findings with controlled experiments or domain review before causal claims. It does not directly address the Highland Education case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

Differentiate shadow validation from controlled live traffic allocation and select the appropriate comparison procedure. It does not directly address the Highland Education case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

Use a separate validation set or cross-validation for tuning and reserve the final holdout for one fixed-model assessment. It does not directly address the Highland Education case: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The relevant acceptance criterion for this case is: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

A threshold sweep documents the target false-positive rate and final locked threshold performs as expected on test data. This would verify a different operation: Tune the operating threshold against the business cost curve on validation data while keeping the final test set untouched. That separate action addresses: A binary classifier produces ranking scores with useful ROC AUC but a default 0.5 threshold yields too many false positives for an operational queue. It cannot establish the required Highland Education correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

A representative image set demonstrates whether the managed API meets accuracy and supported-feature requirements within cost bounds. This would verify a different operation: Compare Amazon Rekognition's supported capabilities against the need for custom features before committing to training. That separate action addresses: An image-processing system needs managed detection of standard objects and text in product photos, but a project proposal requires a custom vision model for each image type. It cannot establish the required Highland Education correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.

An approved sample recording yields usable time-aligned text and an operating cost comparison documents the managed-service choice. This would verify a different operation: Evaluate Amazon Transcribe as the managed speech-to-text capability before building and maintaining a custom recognizer. That separate action addresses: A customer support application needs to convert recorded calls into searchable text but developers plan to train speech recognition from scratch for a small dataset. It cannot establish the required Highland Education correction for: A multi-class classifier trained with a single dominant class consistently ignores minority groups, yet training loss alone appears healthy. The decisive acceptance check is instead: Per-class evaluation improves without using synthetic training records inside validation or test partitions.