evaluate¶
Operation¶
evaluate
¶
evaluate(request: EvaluateInput | EvaluateInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[EvaluateInputDict]) -> EvaluateOutput
Performs on-demand evaluation of agent traces using a specified evaluator. This synchronous API accepts traces in OpenTelemetry format and returns immediate scoring results with detailed explanations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
EvaluateInput | EvaluateInputDict | None
|
The input as a model or dictionary. Pass this or keyword arguments, not both. |
None
|
plugins
|
list[Plugin] | None
|
A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations. |
None
|
**kwargs
|
Unpack[EvaluateInputDict]
|
The input fields as keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
EvaluateOutput
|
An instance of |
evaluate
async
¶
evaluate(request: EvaluateInput | EvaluateInputDict | None = None, /, *, plugins: list[Plugin] | None = None, **kwargs: Unpack[EvaluateInputDict]) -> EvaluateOutput
Performs on-demand evaluation of agent traces using a specified evaluator. This synchronous API accepts traces in OpenTelemetry format and returns immediate scoring results with detailed explanations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
request
|
EvaluateInput | EvaluateInputDict | None
|
The input as a model or dictionary. Pass this or keyword arguments, not both. |
None
|
plugins
|
list[Plugin] | None
|
A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations. |
None
|
**kwargs
|
Unpack[EvaluateInputDict]
|
The input fields as keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
EvaluateOutput
|
An instance of |
Input¶
Pass the input as keyword arguments, a positional dictionary, or a positional model object. Pass plugins separately as a keyword argument.
EvaluateInputDict
¶
Bases: TypedDict
Dictionary form of EvaluateInput.
Attributes¶
evaluation_input
instance-attribute
¶
evaluation_input: Required[EvaluationInput | EvaluationInputDict]
The input data containing agent session spans to be evaluated. Includes a list of spans in OpenTelemetry format from supported frameworks like Strands (AgentCore Runtime) or LangGraph with OpenInference instrumentation.
evaluation_reference_inputs
instance-attribute
¶
evaluation_reference_inputs: Sequence[EvaluationReferenceInput | EvaluationReferenceInputDict] | None
Ground truth data to compare against agent responses during evaluation. Allows to provide expected responses, assertions, and expected tool trajectories at different evaluation levels. Session-level reference inputs apply to the entire conversation, while trace-level reference inputs target specific request-response interactions identified by trace ID.
evaluation_target
instance-attribute
¶
evaluation_target: EvaluationTarget | EvaluationTargetDict | None
The specific trace or span IDs to evaluate within the provided input. Allows targeting evaluation at different levels: individual tool calls, single request-response interactions (traces), or entire conversation sessions.
EvaluateInput
dataclass
¶
Bases: _SerdeShape
Dataclass for EvaluateInput structure.
Attributes¶
evaluation_input
class-attribute
instance-attribute
¶
evaluation_input: EvaluationInput | None = ...
The input data containing agent session spans to be evaluated. Includes a list of spans in OpenTelemetry format from supported frameworks like Strands (AgentCore Runtime) or LangGraph with OpenInference instrumentation.
evaluation_reference_inputs
class-attribute
instance-attribute
¶
evaluation_reference_inputs: list[EvaluationReferenceInput] | None = ...
Ground truth data to compare against agent responses during evaluation. Allows to provide expected responses, assertions, and expected tool trajectories at different evaluation levels. Session-level reference inputs apply to the entire conversation, while trace-level reference inputs target specific request-response interactions identified by trace ID.
evaluation_target
class-attribute
instance-attribute
¶
evaluation_target: EvaluationTarget | None = ...
The specific trace or span IDs to evaluate within the provided input. Allows targeting evaluation at different levels: individual tool calls, single request-response interactions (traces), or entire conversation sessions.
Output¶
EvaluateOutput
dataclass
¶
Bases: _SerdeShape
Dataclass for EvaluateOutput structure.
Attributes¶
evaluation_results
instance-attribute
¶
evaluation_results: list[EvaluationResultContent]
The detailed evaluation results containing scores, explanations, and metadata. Includes the evaluator information, numerical or categorical ratings based on the evaluator's rating scale, and token usage statistics for the evaluation process.
response_metadata
class-attribute
instance-attribute
¶
Metadata about the response that produced this output. Use this to recover the request identifiers a service's support team needs in order to investigate a call. Members of the metadata are individually optional.