Skip to content

create_online_evaluation_config

Operation

create_online_evaluation_config

create_online_evaluation_config(request: CreateOnlineEvaluationConfigInput | CreateOnlineEvaluationConfigInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[CreateOnlineEvaluationConfigInputDict]) -> CreateOnlineEvaluationConfigOutput

Creates an online evaluation configuration for continuous monitoring of agent performance. Online evaluation automatically samples live traffic from CloudWatch logs at specified rates and applies evaluators to assess agent quality in production.

Parameters:

Name Type Description Default
request CreateOnlineEvaluationConfigInput | CreateOnlineEvaluationConfigInputDict | None

The input as a model or dictionary. Pass this or keyword arguments, not both.

None
plugins list[Plugin] | None

A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations.

None
**kwargs Unpack[CreateOnlineEvaluationConfigInputDict]

The input fields as keyword arguments.

{}

Returns:

Type Description
CreateOnlineEvaluationConfigOutput

An instance of CreateOnlineEvaluationConfigOutput.

create_online_evaluation_config async

create_online_evaluation_config(request: CreateOnlineEvaluationConfigInput | CreateOnlineEvaluationConfigInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[CreateOnlineEvaluationConfigInputDict]) -> CreateOnlineEvaluationConfigOutput

Creates an online evaluation configuration for continuous monitoring of agent performance. Online evaluation automatically samples live traffic from CloudWatch logs at specified rates and applies evaluators to assess agent quality in production.

Parameters:

Name Type Description Default
request CreateOnlineEvaluationConfigInput | CreateOnlineEvaluationConfigInputDict | None

The input as a model or dictionary. Pass this or keyword arguments, not both.

None
plugins list[Plugin] | None

A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations.

None
**kwargs Unpack[CreateOnlineEvaluationConfigInputDict]

The input fields as keyword arguments.

{}

Returns:

Type Description
CreateOnlineEvaluationConfigOutput

An instance of CreateOnlineEvaluationConfigOutput.

Input

Pass the input as keyword arguments, a positional dictionary, or a positional model object. Pass plugins separately as a keyword argument.

CreateOnlineEvaluationConfigInputDict

Bases: TypedDict

Dictionary form of CreateOnlineEvaluationConfigInput.

Attributes

client_token instance-attribute
client_token: str | None

A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.

This field is autopopulated if not provided.

clustering_config instance-attribute
clustering_config: ClusteringConfig | ClusteringConfigDict | None

Configuration for periodic batch evaluation clustering of insight results.

data_source_config instance-attribute

The data source configuration that specifies CloudWatch log groups and service names to monitor for agent traces.

description instance-attribute
description: str | None

The description of the online evaluation configuration that explains its monitoring purpose and scope.

enable_on_create instance-attribute
enable_on_create: Required[bool]

Whether to enable the online evaluation configuration immediately upon creation. If true, evaluation begins automatically.

evaluation_execution_role_arn instance-attribute
evaluation_execution_role_arn: Required[str]

The Amazon Resource Name (ARN) of the IAM role that grants permissions to read from CloudWatch logs, write evaluation results, and invoke Amazon Bedrock models for evaluation. If the configuration references evaluators encrypted with a customer managed KMS key, this role must also have kms:Decrypt permission on the KMS key. The service validates this permission at configuration creation time. For more information, see Encryption at rest for AgentCore Evaluations.

evaluators instance-attribute

The list of evaluators to apply during online evaluation. Can include both built-in evaluators and custom evaluators created with CreateEvaluator.

insights instance-attribute
insights: Sequence[Insight | InsightDict] | None

The list of insight types to run against agent sessions.

online_evaluation_config_name instance-attribute
online_evaluation_config_name: Required[str]

The name of the online evaluation configuration. Must be unique within your account.

output_config instance-attribute
output_config: OutputConfig | OutputConfigDict | None

The configuration that specifies where evaluation results should be written for monitoring and analysis.

rule instance-attribute

The evaluation rule that defines sampling configuration, filters, and session detection settings for the online evaluation.

tags instance-attribute
tags: Mapping[str, str] | None

A map of tag keys and values to assign to an AgentCore Online Evaluation Config. Tags enable you to categorize your resources in different ways, for example, by purpose, owner, or environment.

CreateOnlineEvaluationConfigInput dataclass

Bases: _SerdeShape

Dataclass for CreateOnlineEvaluationConfigInput structure.

Attributes

client_token class-attribute instance-attribute
client_token: str | None = ...

A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.

This field is autopopulated if not provided.

clustering_config class-attribute instance-attribute
clustering_config: ClusteringConfig | None = ...

Configuration for periodic batch evaluation clustering of insight results.

data_source_config class-attribute instance-attribute
data_source_config: DataSourceConfig | None = ...

The data source configuration that specifies CloudWatch log groups and service names to monitor for agent traces.

description class-attribute instance-attribute
description: str | None = ...

The description of the online evaluation configuration that explains its monitoring purpose and scope.

enable_on_create class-attribute instance-attribute
enable_on_create: bool | None = ...

Whether to enable the online evaluation configuration immediately upon creation. If true, evaluation begins automatically.

evaluation_execution_role_arn class-attribute instance-attribute
evaluation_execution_role_arn: str | None = ...

The Amazon Resource Name (ARN) of the IAM role that grants permissions to read from CloudWatch logs, write evaluation results, and invoke Amazon Bedrock models for evaluation. If the configuration references evaluators encrypted with a customer managed KMS key, this role must also have kms:Decrypt permission on the KMS key. The service validates this permission at configuration creation time. For more information, see Encryption at rest for AgentCore Evaluations.

evaluators class-attribute instance-attribute
evaluators: list[EvaluatorReference] | None = ...

The list of evaluators to apply during online evaluation. Can include both built-in evaluators and custom evaluators created with CreateEvaluator.

insights class-attribute instance-attribute
insights: list[Insight] | None = ...

The list of insight types to run against agent sessions.

online_evaluation_config_name class-attribute instance-attribute
online_evaluation_config_name: str | None = ...

The name of the online evaluation configuration. Must be unique within your account.

output_config class-attribute instance-attribute
output_config: OutputConfig | None = ...

The configuration that specifies where evaluation results should be written for monitoring and analysis.

rule class-attribute instance-attribute
rule: Rule | None = ...

The evaluation rule that defines sampling configuration, filters, and session detection settings for the online evaluation.

tags class-attribute instance-attribute
tags: dict[str, str] | None = ...

A map of tag keys and values to assign to an AgentCore Online Evaluation Config. Tags enable you to categorize your resources in different ways, for example, by purpose, owner, or environment.

Output

CreateOnlineEvaluationConfigOutput dataclass

Bases: _SerdeShape

Dataclass for CreateOnlineEvaluationConfigOutput structure.

Attributes

created_at instance-attribute
created_at: datetime

The timestamp when the online evaluation configuration was created.

execution_status instance-attribute
execution_status: str

The execution status indicating whether the online evaluation is currently running.

failure_reason class-attribute instance-attribute
failure_reason: str | None = ...

The reason for failure if the online evaluation configuration creation or execution failed.

online_evaluation_config_arn instance-attribute
online_evaluation_config_arn: str

The Amazon Resource Name (ARN) of the created online evaluation configuration.

online_evaluation_config_id instance-attribute
online_evaluation_config_id: str

The unique identifier of the created online evaluation configuration.

output_config class-attribute instance-attribute
output_config: OutputConfig | None = ...

The configuration that specifies where evaluation results should be written for monitoring and analysis.

response_metadata class-attribute instance-attribute
response_metadata: ResponseMetadata = ...

Metadata about the response that produced this output. Use this to recover the request identifiers a service's support team needs in order to investigate a call. Members of the metadata are individually optional.

status instance-attribute
status: str

The status of the online evaluation configuration.