263 lines
6.9 KiB
Markdown
263 lines
6.9 KiB
Markdown
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
# Doris MCP Server Testing System
|
|
|
|
## Overview
|
|
|
|
This testing system adopts a layered architecture, including unit tests, integration tests, and client-server tests. The testing system assumes the server is already properly started and focuses on testing functionality rather than startup configuration.
|
|
|
|
## Testing Architecture
|
|
|
|
### 1. Unit Tests
|
|
- **Location**: `test/security/`, `test/utils/`, `test/tools/`
|
|
- **Purpose**: Test individual module functionality
|
|
- **Features**: Uses Mock objects, no dependency on external services
|
|
|
|
### 2. Integration Tests
|
|
- **Location**: `test/integration/`
|
|
- **Purpose**: Test collaboration between modules
|
|
- **Features**: Test complete workflows
|
|
|
|
### 3. Client-Server Tests
|
|
- **Location**: `test/tools/test_tools_client_server.py`, `test/utils/test_query_executor_client_server.py`
|
|
- **Purpose**: Test actual server functionality through MCP client
|
|
- **Features**: Assumes server is running, skips tests if server is not available
|
|
|
|
## Configuration Files
|
|
|
|
### test_config.json
|
|
Test configuration file defines how to connect to the running server:
|
|
|
|
```json
|
|
{
|
|
"server_endpoints": {
|
|
"http": {
|
|
"url": "http://localhost:3000/mcp",
|
|
"timeout": 30
|
|
},
|
|
"stdio": {
|
|
"command": "uv",
|
|
"args": ["run", "python", "-m", "doris_mcp_server.main", "--transport", "stdio"],
|
|
"timeout": 30
|
|
}
|
|
},
|
|
"test_settings": {
|
|
"default_transport": "http",
|
|
"retry_attempts": 3,
|
|
"retry_delay": 1.0,
|
|
"test_timeout": 60,
|
|
"enable_performance_tests": true,
|
|
"enable_security_tests": true
|
|
}
|
|
}
|
|
```
|
|
|
|
## Usage
|
|
|
|
### 1. Start the Server
|
|
|
|
Before running client-server tests, you need to start the server first:
|
|
|
|
#### HTTP Mode (Recommended)
|
|
```bash
|
|
# Start HTTP server
|
|
./start_server.sh
|
|
# or
|
|
uv run python -m doris_mcp_server.main --transport http --port 3000
|
|
```
|
|
|
|
#### Stdio Mode
|
|
```bash
|
|
# Stdio mode is started directly by the client, no need to pre-start
|
|
```
|
|
|
|
### 2. Run Tests
|
|
|
|
#### Run All Tests
|
|
```bash
|
|
python -m pytest test/ -v
|
|
```
|
|
|
|
#### Run Unit Tests
|
|
```bash
|
|
# Security module tests
|
|
python -m pytest test/security/ -v
|
|
|
|
# Tools module tests
|
|
python -m pytest test/tools/test_tools_manager.py -v
|
|
|
|
# Query executor tests
|
|
python -m pytest test/utils/test_query_executor.py -v
|
|
```
|
|
|
|
#### Run Integration Tests
|
|
```bash
|
|
python -m pytest test/integration/ -v
|
|
```
|
|
|
|
#### Run Client-Server Tests
|
|
```bash
|
|
# Tools Client-Server tests
|
|
python -m pytest test/tools/test_tools_client_server.py -v
|
|
|
|
# QueryExecutor Client-Server tests
|
|
python -m pytest test/utils/test_query_executor_client_server.py -v
|
|
```
|
|
|
|
### 3. Test Configuration
|
|
|
|
#### Modify Server Endpoints
|
|
Edit the `test/test_config.json` file:
|
|
|
|
```json
|
|
{
|
|
"server_endpoints": {
|
|
"http": {
|
|
"url": "http://your-server:port/mcp"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Enable/Disable Specific Tests
|
|
```json
|
|
{
|
|
"test_settings": {
|
|
"enable_performance_tests": false, // Disable performance tests
|
|
"enable_security_tests": true // Enable security tests
|
|
}
|
|
}
|
|
```
|
|
|
|
## Test Status
|
|
|
|
### ✅ Completed Test Modules
|
|
|
|
1. **Security Module** (100% Pass)
|
|
- Authentication tests: 5/5 passed
|
|
- Authorization tests: 7/7 passed
|
|
- Data masking tests: 13/13 passed
|
|
- SQL validation tests: 10/10 passed
|
|
- Security manager tests: 7/7 passed
|
|
- Coverage: 88%
|
|
|
|
2. **Client-Server Test Architecture** (Implemented)
|
|
- Automatic server connection status detection
|
|
- Automatically skip tests when server is not running
|
|
- Support for both HTTP and Stdio transport modes
|
|
|
|
### 🔄 Tests Requiring Server Running
|
|
|
|
1. **Tools Client-Server Tests**
|
|
- Tool list retrieval
|
|
- SQL query execution
|
|
- Database list retrieval
|
|
- Table schema queries
|
|
- Performance statistics
|
|
- Error handling
|
|
- Security authentication
|
|
|
|
2. **QueryExecutor Client-Server Tests**
|
|
- Simple query execution
|
|
- Database queries
|
|
- Information schema queries
|
|
- Parameterized queries
|
|
- Error handling
|
|
- Security authentication
|
|
|
|
## Testing Best Practices
|
|
|
|
### 1. Server Startup Check
|
|
All client-server tests automatically check server connection status:
|
|
- If server is running normally, execute actual tests
|
|
- If server is not running, skip tests and display appropriate message
|
|
|
|
### 2. Test Isolation
|
|
- Unit tests use Mock objects, no dependency on external services
|
|
- Integration tests use controlled test environments
|
|
- Client-server tests connect to actually running servers
|
|
|
|
### 3. Error Handling
|
|
- Tests don't assume specific success/failure results
|
|
- Verify response structure rather than specific content
|
|
- Gracefully handle connection failures and timeouts
|
|
|
|
### 4. Configuration Management
|
|
- Use configuration files to manage test parameters
|
|
- Support configuration switching for different environments
|
|
- Provide reasonable default values
|
|
|
|
## Troubleshooting
|
|
|
|
### 1. Server Connection Failure
|
|
```
|
|
ERROR: Server is not running or not accessible
|
|
```
|
|
**Solution**: Ensure the server is started and listening on the correct port
|
|
|
|
### 2. Import Errors
|
|
```
|
|
ImportError: cannot import name 'DorisUnifiedClient'
|
|
```
|
|
**Solution**: Check Python path and dependency installation
|
|
|
|
### 3. Test Timeouts
|
|
```
|
|
TimeoutError: Test execution timeout
|
|
```
|
|
**Solution**: Increase timeout settings in `test_config.json`
|
|
|
|
## Development Guide
|
|
|
|
### Adding New Client-Server Tests
|
|
|
|
1. Add test methods in the appropriate test file
|
|
2. Use `@pytest.mark.asyncio` decorator
|
|
3. Get test client through `client` fixture
|
|
4. Implement test callback function
|
|
5. Verify response structure
|
|
|
|
Example:
|
|
```python
|
|
@pytest.mark.asyncio
|
|
async def test_new_feature_via_client(self, client, test_config):
|
|
"""Test new feature through client"""
|
|
async def test_callback(client_instance):
|
|
result = await client_instance.call_tool("new_tool", {
|
|
"param": "value"
|
|
})
|
|
|
|
assert "success" in result
|
|
return result
|
|
|
|
result = await client.connect_and_run(test_callback)
|
|
assert "success" in result
|
|
```
|
|
|
|
### Modifying Test Configuration
|
|
|
|
Edit the `test/test_config.json` file to adjust:
|
|
- Server endpoints
|
|
- Timeout settings
|
|
- Test data
|
|
- Feature switches
|
|
|
|
## Summary
|
|
|
|
This testing system provides complete test coverage, from unit tests to end-to-end client-server tests. Through reasonable configuration and automated connection detection, it ensures tests can run stably in different environments. |