Skip to content

AutomatedEvaluationConfig

Structure Class

AutomatedEvaluationConfig dataclass

Bases: _SerdeShape

The configuration details of an automated evaluation job. The EvaluationDatasetMetricConfig object is used to specify the prompt datasets, task type, and metric names.

Attributes

custom_metric_config class-attribute instance-attribute
custom_metric_config: AutomatedEvaluationCustomMetricConfig | None = ...

Defines the configuration of custom metrics to be used in an evaluation job.

dataset_metric_configs instance-attribute
dataset_metric_configs: list[EvaluationDatasetMetricConfig]

Configuration details of the prompt datasets and metrics you want to use for your evaluation job.

evaluator_model_config class-attribute instance-attribute
evaluator_model_config: EvaluatorModelConfig | None = ...

Contains the evaluator model configuration details. EvaluatorModelConfig is required for evaluation jobs that use a knowledge base or in model evaluation job that use a model as judge. This model computes all evaluation related metrics.