create_evaluator¶
Operation¶
create_evaluator
¶
create_evaluator(request: CreateEvaluatorInput | CreateEvaluatorInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[CreateEvaluatorInputDict]) -> CreateEvaluatorOutput
Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
CreateEvaluatorInput | CreateEvaluatorInputDict | None
|
The input as a model or dictionary. Pass this or keyword arguments, not both. |
None
|
plugins
|
list[Plugin] | None
|
A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations. |
None
|
**kwargs
|
Unpack[CreateEvaluatorInputDict]
|
The input fields as keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
CreateEvaluatorOutput
|
An instance of |
create_evaluator
async
¶
create_evaluator(request: CreateEvaluatorInput | CreateEvaluatorInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[CreateEvaluatorInputDict]) -> CreateEvaluatorOutput
Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
CreateEvaluatorInput | CreateEvaluatorInputDict | None
|
The input as a model or dictionary. Pass this or keyword arguments, not both. |
None
|
plugins
|
list[Plugin] | None
|
A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations. |
None
|
**kwargs
|
Unpack[CreateEvaluatorInputDict]
|
The input fields as keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
CreateEvaluatorOutput
|
An instance of |
Input¶
Pass the input as keyword arguments, a positional dictionary, or a positional model object. Pass plugins separately as a keyword argument.
CreateEvaluatorInputDict
¶
Bases: TypedDict
Dictionary form of CreateEvaluatorInput.
Attributes¶
client_token
instance-attribute
¶
client_token: str | None
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
This field is autopopulated if not provided.
description
instance-attribute
¶
description: str | None
The description of the evaluator that explains its purpose and evaluation criteria.
evaluator_config
instance-attribute
¶
evaluator_config: Required[EvaluatorConfig | EvaluatorConfigDict]
The configuration for the evaluator. Specify either LLM-as-a-Judge settings with instructions, rating scale, and model configuration, or code-based settings with a customer-managed Lambda function.
evaluator_name
instance-attribute
¶
The name of the evaluator. Must be unique within your account.
kms_key_arn
instance-attribute
¶
kms_key_arn: str | None
The Amazon Resource Name (ARN) of a customer managed KMS key to use for encrypting sensitive evaluator data, including instructions and rating scale. If you don't specify a KMS key, the evaluator data is encrypted with an Amazon Web Services owned key. Only symmetric encryption KMS keys are supported. For more information, see Encryption at rest for AgentCore Evaluations.
level
instance-attribute
¶
The evaluation level that determines the scope of evaluation. Valid
values are TOOL_CALL for individual tool invocations, TRACE for
single request-response interactions, or SESSION for entire
conversation sessions.
CreateEvaluatorInput
dataclass
¶
Bases: _SerdeShape
Dataclass for CreateEvaluatorInput structure.
Attributes¶
client_token
class-attribute
instance-attribute
¶
client_token: str | None = ...
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
This field is autopopulated if not provided.
description
class-attribute
instance-attribute
¶
description: str | None = ...
The description of the evaluator that explains its purpose and evaluation criteria.
evaluator_config
class-attribute
instance-attribute
¶
evaluator_config: EvaluatorConfig | None = ...
The configuration for the evaluator. Specify either LLM-as-a-Judge settings with instructions, rating scale, and model configuration, or code-based settings with a customer-managed Lambda function.
evaluator_name
class-attribute
instance-attribute
¶
evaluator_name: str | None = ...
The name of the evaluator. Must be unique within your account.
kms_key_arn
class-attribute
instance-attribute
¶
kms_key_arn: str | None = ...
The Amazon Resource Name (ARN) of a customer managed KMS key to use for encrypting sensitive evaluator data, including instructions and rating scale. If you don't specify a KMS key, the evaluator data is encrypted with an Amazon Web Services owned key. Only symmetric encryption KMS keys are supported. For more information, see Encryption at rest for AgentCore Evaluations.
Output¶
CreateEvaluatorOutput
dataclass
¶
Bases: _SerdeShape
Dataclass for CreateEvaluatorOutput structure.
Attributes¶
evaluator_arn
instance-attribute
¶
evaluator_arn: str
The Amazon Resource Name (ARN) of the created evaluator.
response_metadata
class-attribute
instance-attribute
¶
Metadata about the response that produced this output. Use this to recover the request identifiers a service's support team needs in order to investigate a call. Members of the metadata are individually optional.