Skip to content

InferenceConfigurationDict

Dictionary Type

InferenceConfigurationDict

Bases: TypedDict

Dictionary form of InferenceConfiguration.

Attributes

max_tokens instance-attribute
max_tokens: int | None

The maximum number of tokens to generate in the model response during evaluation.

stop_sequences instance-attribute
stop_sequences: Sequence[str] | None

The list of sequences that will cause the model to stop generating tokens when encountered.

temperature instance-attribute
temperature: float | None

The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.

top_p instance-attribute
top_p: float | None

The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.