101 lines
3.4 KiB
Markdown
101 lines
3.4 KiB
Markdown
|
# Temperature Parameter Fix for GPT-5 Mini
|
||
|
|
|
||
|
|
## Problem
|
||
|
|
|
||
|
|
GPT-5 mini model does not support the `temperature` parameter when set to 0.0 or any non-default value. It only supports the default temperature value (1). This caused the following error:
|
||
|
|
|
||
|
|
```
|
||
|
|
Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.0 with this model. Only the default (1) value is supported.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_value'}}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Root Cause
|
||
|
|
|
||
|
|
The system was always passing a `temperature` parameter to the LLM, even when it was commented out in the configuration file. This happened because:
|
||
|
|
|
||
|
|
1. `LLMParametersConfig` had a default value of `temperature: float = 0`
|
||
|
|
2. `LLMRagConfig` had a default value of `temperature: float = 0.2`
|
||
|
|
3. The LLM client always passed temperature to the model constructor
|
||
|
|
|
||
|
|
## Solution
|
||
|
|
|
||
|
|
Modified the code to only pass the `temperature` parameter when it's explicitly set in the configuration:
|
||
|
|
|
||
|
|
### 1. Changed Configuration Classes
|
||
|
|
|
||
|
|
**File: `service/config.py`**
|
||
|
|
|
||
|
|
- `LLMParametersConfig.temperature`: Changed from `float = 0` to `Optional[float] = None`
|
||
|
|
- `LLMRagConfig.temperature`: Changed from `float = 0.2` to `Optional[float] = None`
|
||
|
|
|
||
|
|
### 2. Updated Configuration Loading
|
||
|
|
|
||
|
|
**File: `service/config.py` - `get_llm_config()` method**
|
||
|
|
|
||
|
|
- Only include `temperature` in the config dict when it's explicitly set (not None)
|
||
|
|
- Added proper null checks for both new and legacy configuration formats
|
||
|
|
|
||
|
|
### 3. Modified LLM Client Construction
|
||
|
|
|
||
|
|
**File: `service/llm_client.py` - `_create_llm()` method**
|
||
|
|
|
||
|
|
- Changed to only pass `temperature` parameter when it exists in the config
|
||
|
|
- Removed hardcoded fallback temperature values
|
||
|
|
- Works for both OpenAI and Azure OpenAI providers
|
||
|
|
|
||
|
|
## Behavior
|
||
|
|
|
||
|
|
### Before Fix
|
||
|
|
- Temperature was always passed to the model (either 0, 0.2, or configured value)
|
||
|
|
- GPT-5 mini would reject requests with temperature != 1
|
||
|
|
|
||
|
|
### After Fix
|
||
|
|
- When `temperature` is commented out or not set: Parameter is not passed to model (uses model default)
|
||
|
|
- When `temperature` is explicitly set: Parameter is passed with the configured value
|
||
|
|
- GPT-5 mini works correctly as it uses its default temperature when none is specified
|
||
|
|
|
||
|
|
## Testing
|
||
|
|
|
||
|
|
Created comprehensive test script: `scripts/test_temperature_fix.py`
|
||
|
|
|
||
|
|
Test results show:
|
||
|
|
- ✅ When temperature not set: No temperature passed to model, API calls succeed
|
||
|
|
- ✅ When temperature set: Correct value passed to model
|
||
|
|
- ✅ API stability: Multiple consecutive calls work correctly
|
||
|
|
|
||
|
|
## Configuration Examples
|
||
|
|
|
||
|
|
### No Temperature (Uses Model Default)
|
||
|
|
```yaml
|
||
|
|
# llm_prompt.yaml
|
||
|
|
parameters:
|
||
|
|
# temperature: 0 # Commented out
|
||
|
|
max_context_length: 100000
|
||
|
|
```
|
||
|
|
|
||
|
|
### Explicit Temperature
|
||
|
|
```yaml
|
||
|
|
# llm_prompt.yaml
|
||
|
|
parameters:
|
||
|
|
temperature: 0.7 # Will be passed to model
|
||
|
|
max_context_length: 100000
|
||
|
|
```
|
||
|
|
|
||
|
|
## Backward Compatibility
|
||
|
|
|
||
|
|
- ✅ Existing configurations continue to work
|
||
|
|
- ✅ Legacy `config.yaml` LLM configurations still supported
|
||
|
|
- ✅ No breaking changes to API or behavior when temperature is explicitly set
|
||
|
|
|
||
|
|
## Files Modified
|
||
|
|
|
||
|
|
1. `service/config.py`
|
||
|
|
- `LLMParametersConfig.temperature` → `Optional[float] = None`
|
||
|
|
- `LLMRagConfig.temperature` → `Optional[float] = None`
|
||
|
|
- `get_llm_config()` → Only include temperature when set
|
||
|
|
|
||
|
|
2. `service/llm_client.py`
|
||
|
|
- `_create_llm()` → Only pass temperature when in config
|
||
|
|
|
||
|
|
3. `scripts/test_temperature_fix.py` (New)
|
||
|
|
- Comprehensive test suite for temperature handling
|