Skip to content

PromptModelInferenceConfigurationDict

Dictionary Type

PromptModelInferenceConfigurationDict

Bases: TypedDict

Dictionary form of PromptModelInferenceConfiguration.

Attributes

max_tokens instance-attribute
max_tokens: int | None

The maximum number of tokens to return in the response.

stop_sequences instance-attribute
stop_sequences: Sequence[str] | None

A list of strings that define sequences after which the model will stop generating.

temperature instance-attribute
temperature: float | None

Controls the randomness of the response. Choose a lower value for more predictable outputs and a higher value for more surprising outputs.

top_p instance-attribute
top_p: float | None

The percentage of most-likely candidates that the model considers for the next token.