SageMaker inference experiments should have instance storage volume encrypted with customer managed AWS KMS keys

How to fix it

Set at creation: fixing it means replacing the resource.

Name a KMS key you control for the storage volume on the compute instance hosting the endpoint under test, in the inference experiment's top-level KmsKey. It is set when the experiment is created and UpdateInferenceExperiment has no place for it, so a failing experiment is replaced: stop it, create a new one with the key named, and let it run for the sampling window you need. The comparison so far is lost with the experiment you stopped. The SageMaker execution role needs kms:CreateGrant on the key. SageMaker.24 reads a different key on the same experiment, over the payloads captured in Amazon S3, and that one needs no replacement.

In the console

SageMaker AI → Inference → Shadow tests → Create shadow test → Enter shadow test settings → Encryption key

With the AWS CLI

aws sagemaker create-inference-experiment --name <experiment-name> --type ShadowMode --role-arn <execution-role-arn> --endpoint-name <endpoint-name> --model-variants '[{"ModelName":"<model-name>","VariantName":"<variant-name>","InfrastructureConfig":{"InfrastructureType":"RealTimeInference","RealTimeInferenceConfig":{"InstanceType":"<instance-type>","InstanceCount":1}}}]' --shadow-mode-config '{"SourceModelVariantName":"<variant-name>","ShadowModelVariants":[{"ShadowModelVariantName":"<shadow-variant-name>","SamplingPercentage":10}]}' --kms-key <kms-key-arn> --region <region>