Skip to content

InferenceConfigurationDict

Dictionary Type

InferenceConfigurationDict

Bases: TypedDict

Dictionary form of InferenceConfiguration.

Attributes

max_tokens instance-attribute
max_tokens: int | None

The maximum number of tokens to allow in the generated response. The default value is the maximum allowed value for the model that you are using.

stop_sequences instance-attribute
stop_sequences: Sequence[str] | None

A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response.

temperature instance-attribute
temperature: float | None

The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options, while a higher value makes the model more likely to choose lower-probability options.

top_p instance-attribute
top_p: float | None

The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of 0.8 for topP, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence.