Skip to content

InferenceConfiguration

Structure Class

InferenceConfiguration dataclass

Bases: _SerdeShape

The configuration parameters that control how the foundation model behaves during evaluation, including response generation settings.

Attributes

max_tokens class-attribute instance-attribute
max_tokens: int | None = ...

The maximum number of tokens to generate in the model response during evaluation.

stop_sequences class-attribute instance-attribute
stop_sequences: list[str] | None = ...

The list of sequences that will cause the model to stop generating tokens when encountered.

temperature class-attribute instance-attribute
temperature: float | None = ...

The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.

top_p class-attribute instance-attribute
top_p: float | None = ...

The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.