AIRegulation/AIRegulation-DocAnalysis
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor code structure for improved readability and maintainability

Browse Source
This commit is contained in:
ash66
2026-05-14 18:09:15 +08:00
parent 10d04c4083
commit 35cd927d02
105 changed files with 9043 additions and 7720 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
.gitignore vendored Normal file
View File

@@ -0,0 +1,59 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# Distribution / packaging
build/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
*.egg
MANIFEST
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
.coverage
.coverage.*
.pytest_cache/
.mypy_cache/
.ruff_cache/
.pyre/
.hypothesis/
htmlcov/
coverage.xml
nosetests.xml
*.cover
*.py,cover
# Environments
.env
.env.*
.venv/
venv/
env/
ENV/
# Type checker / notebook / tool state
.python-version
.ipynb_checkpoints/
# IDEs and editors
.idea/
.vscode/
# OS files
.DS_Store
Thumbs.db

39
AGENTS.md Normal file
View File

@@ -0,0 +1,39 @@
# AGENTS.md
## Scope
- This repo uses `backend/app/` for the backend and `frontend/` for the Vite React app.
## Real Entrypoints
- Current backend app entrypoint is `backend/app/main.py`, exporting `app` from `app.api.main`.
- Current backend dev start command is `python -m uvicorn app.main:app --reload` with `PYTHONPATH=backend`.
- `dev.sh start api --foreground` is the repo script flow that encodes the expected backend startup behavior.
- Frontend dev server is `frontend` Vite on port `5173`, proxying `/api` to `http://localhost:8000`.
## Commands
- Backend install: `pip install -r backend/requirements.txt`
- Backend run from repo root: `PYTHONPATH=backend uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload`
- Frontend install: `cd frontend && npm install`
- Frontend dev: `cd frontend && npm run dev`
- Frontend build: `cd frontend && npm run build`
- Frontend lint: `cd frontend && npm run lint`
## Infra And Env
- Backend settings load from root `.env`, not `backend/.env`, because `backend/app/config/settings.py` uses `env_file = ".env"`.
- Docker infra is defined in `docker/docker-compose.yml`; it starts Milvus, MinIO, Redis, and PostgreSQL.
- Default local service ports: Milvus `19530`, MinIO `9000/9001`, Redis `6379`, PostgreSQL `5432`, backend `8000`, frontend `5173`.
- LLM base URLs in `.env.example` point at a shared remote gateway (`http://6.86.80.4:30080/v1`); do not assume offline/local-only LLM execution.
## Verification
- Root pytest config in `pyproject.toml` points at root `tests/`, and those tests import from `backend/app` via `PYTHONPATH` setup.
- For frontend-only changes, run `npm run lint` and `npm run build` in `frontend`.
- For backend changes, prefer focused import/startup verification against `backend/app`, and run root `tests/` when the environment supports it.
## Gotchas
- Backend settings load from root `.env`, not `backend/.env`, because `backend/app/config/settings.py` uses `env_file = ".env"`.
- The root `pyproject.toml` is the active Python package manifest for the repo.

124
QUICK_DEPLOY.md
View File

@@ -71,11 +71,11 @@ docker ps
## 三、安装Python依赖
### 方式A：使用快速启动脚本（推荐）
### 方式A：使用统一初始化脚本（推荐）
```bash
chmod +x quick_start.sh
./quick_start.sh
chmod +x dev.sh
./dev.sh setup
```
脚本自动完成：
@@ -91,7 +91,7 @@ python3 -m venv .venv
source .venv/bin/activate
# 安装依赖
pip install -r requirements.txt
pip install -r backend/requirements.txt
```
---
@@ -115,39 +115,42 @@ python -c "from modelscope import snapshot_download; snapshot_download('Xorbits/
## 五、启动服务
### 整合启动脚本（推荐）
### 统一命令入口（推荐）
```bash
# 赋予脚本执行权限
chmod +x start_all.sh stop_all.sh restart_all.sh status.sh
chmod +x dev.sh
# 启动所有服务（API + 前端）
./start_all.sh
./dev.sh start
# 查看服务状态
./status.sh
./dev.sh status
# 重启所有服务
./restart_all.sh
./dev.sh restart
# 停止所有服务
./stop_all.sh
./dev.sh stop
```
### 单独启动（可选）
```bash
# 仅启动API服务（前台运行，可调试）
./start_api.sh
./dev.sh start api --foreground
# 仅启动API服务（后台运行）
./start_api_background.sh
./dev.sh start api
# 仅停止API服务
./stop_api.sh
./dev.sh stop api
# 仅启动前端服务
./start_frontend.sh
# 仅启动前端服务（Vite 开发模式）
./dev.sh start frontend --mode dev
# 仅启动前端服务（静态构建模式）
./dev.sh start frontend --mode static
```
---
@@ -161,9 +164,9 @@ chmod +x start_all.sh stop_all.sh restart_all.sh status.sh
| **API服务** | http://localhost:8000 |
| **API文档** | http://localhost:8000/docs |
| **健康检查** | http://localhost:8000/health |
| **前端测试页面** | http://localhost:3000 |
| **前端测试页面** | http://localhost:5173 |
> 注意：前端测试页面通过 `http://localhost:3000` 访问，自动连接到API服务。
> 注意：前端默认通过 `http://localhost:5173` 访问，自动代理到 API 服务。
---
@@ -221,24 +224,46 @@ curl -X POST http://localhost:8000/api/v1/agent/chat \
| 操作 | 命令 |
|------|------|
| **启动所有服务** | `./start_all.sh` |
| **停止所有服务** | `./stop_all.sh` |
| **重启所有服务** | `./restart_all.sh` |
| **查看服务状态** | `./status.sh` |
| **启动所有服务** | `./dev.sh start` |
| **停止所有服务** | `./dev.sh stop` |
| **重启所有服务** | `./dev.sh restart` |
| **查看服务状态** | `./dev.sh status` |
| 查看API日志 | `tail -f logs/api.log` |
| 查看前端日志 | `tail -f logs/frontend.log` |
| 环境初始化 | `./quick_start.sh` |
| 环境初始化 | `./dev.sh setup` |
| 前台调试 API | `./dev.sh start api --foreground` |
| 静态模式启动前端 | `./dev.sh start frontend --mode static` |
| 重启Docker | `cd docker && docker-compose restart` |
| 下载嵌入模型 | `./download_model.sh` |
---
## 九、服务状态检查
## 九、Linux / Windows 命令对照表
| 操作 | Linux / macOS | Windows |
|------|---------------|---------|
| 环境初始化 | `./dev.sh setup` | `dev.bat setup` |
| 启动所有服务 | `./dev.sh start` | `dev.bat start` |
| 停止所有服务 | `./dev.sh stop` | `dev.bat stop` |
| 重启所有服务 | `./dev.sh restart` | `dev.bat restart` |
| 查看服务状态 | `./dev.sh status` | `dev.bat status` |
| 前台调试 API | `./dev.sh start api --foreground` | `dev.bat start api --foreground` |
| 后台启动 API | `./dev.sh start api` | `dev.bat start api` |
| 启动前端开发模式 | `./dev.sh start frontend --mode dev` | `dev.bat start frontend --mode dev` |
| 启动前端静态模式 | `./dev.sh start frontend --mode static` | `dev.bat start frontend --mode static` |
| 查看 API 日志 | `./dev.sh logs api --follow` | `dev.bat logs api --follow` |
| 查看前端日志 | `./dev.sh logs frontend --follow` | `dev.bat logs frontend --follow` |
> Linux / macOS 首次使用前建议先执行：`chmod +x dev.sh`
---
## 十、服务状态检查
运行状态检查脚本：
```bash
./status.sh
./dev.sh status
```
输出示例：
@@ -256,7 +281,7 @@ API服务:
前端服务:
状态: 运行中 ✓
PID: 12346
地址: http://localhost:3000
地址: http://localhost:5173
Docker服务:
milvus: 运行中 ✓
@@ -271,7 +296,7 @@ Docker服务:
---
## 十、常见问题
## 十一、常见问题
### Q1: Milvus连接失败
@@ -334,49 +359,30 @@ rm -f logs/*.pid
---
## 十一、目录结构
## 十二、目录结构
```
```text
Demo-glm/
├── src/
│ ├── api/ # FastAPI接口
│ │ ├── main.py # API入口
│ │ └── routes/
│ │ ├── documents.py # 文档上传
│ │ ├── knowledge.py # 知识检索
│ │ └── agent.py # 智能问答
│ ├── services/ # 核心服务
│ │ ├── llm/ # LLM调用（Qwen/DeepSeek）
│ │ ├── rag/ # RAG检索
│ │ ├── agent/ # 问答Agent
│ │ ├── parser/ # 文档解析
│ │ ├── embedding/ # 向量嵌入（BGE-M3）
│ │ └── storage/ # Milvus存储
│ └── config/ # 配置管理
├── frontend/ # 前端测试页面
│ └── index.html # 测试界面
├── docker/ # Docker配置
├── backend/
│ ├── app/ # FastAPI 后端代码
│ ├── requirements.txt # Python 依赖
│ └── main.py # 后端启动入口
├── frontend/ # Vite React 前端
├── docker/ # Docker 配置
│ └── docker-compose.yml
├── logs/ # 运行日志
│ ├── api.log
│ └── frontend.log
├── tests/ # 测试脚本
├── tests/ # 根级测试脚本
├── .env # 环境配置
├── .env.example # 配置模板
├── requirements.txt # Python依赖
├── quick_start.sh # 环境初始化脚本
├── start_all.sh # 整合启动脚本
├── stop_all.sh # 整合停止脚本
├── restart_all.sh # 重启脚本
├── status.sh # 状态检查脚本
├── start_api.sh # 单独启动API
├── start_frontend.sh # 单独启动前端
├── pyproject.toml # 根级 Python 项目配置
├── dev.sh # Linux/macOS 统一入口
├── dev.bat # Windows 统一入口
└── QUICK_DEPLOY.md # 本文档
```
---
## 十二、API接口清单
## 十三、API接口清单
| 接口 | 路径 | 方法 | 功能 |
|------|------|------|------|
@@ -419,4 +425,4 @@ Demo-glm/
## 技术支持
- API文档：http://localhost:8000/docs
- 问题反馈：提交Issue到项目仓库
- 问题反馈：提交Issue到项目仓库

52
README.md
View File

@@ -14,38 +14,20 @@
## 项目结构
```
Demo-glm/
├── src/
│ ├── api/ # FastAPI接口层
│ │ ├── main.py # API入口
│ │ ├── routes/
│ │ │ ├── documents.py # 文档上传接口
│ │ │ └── knowledge.py # 知识库检索接口
│ │ └── models/
│ │ └── document.py # Pydantic数据模型
│ ├── services/
│ │ ├── parser/ # 文档解析服务
│ │ │ ├── pdf_parser.py # PDF解析（PyMuPDF）
│ │ │ ├── docx_parser.py # Word解析
│ │ │ └── mineru_parser.py # MinerU多模态解析
│ │ ├── embedding/ # 嵌入服务
│ │ │ ├── text_chunker.py # 智能分块器
│ │ │ └── bge_m3_embedder.py # BGE-M3嵌入
│ │ ├── storage/
│ │ │ └── milvus_client.py # Milvus客户端
│ │ └── document_processor.py # 文档处理主流程
│ └── config/
│ │ ├── settings.py # 配置管理
│ │ └── logging.py # 日志配置
├── tests/
│ ├── test_parser.py # 解析测试
│ ├── test_embedding.py # 嵌入测试
│ ├── test_milvus.py # Milvus测试
│ └── verify_mvp.py # MVP验证脚本
```text
AIRegulation-DocAnalysis-Demo/
├── backend/
│ ├── app/
│ │ ├── api/ # FastAPI 接口层
│ │ ├── config/ # 配置与日志
│ │ ├── services/ # 解析、分块、嵌入、存储、Agent
│ │ └── workers/
│ ├── requirements.txt
│ └── main.py
├── frontend/ # Vite React 前端
├── tests/ # 根级测试，导入 backend/app
├── docker/
│ └── docker-compose.yml # Milvus/MinIO部署
├── requirements.txt
│ └── docker-compose.yml
├── pyproject.toml
└── .env.example
```
@@ -55,7 +37,7 @@ Demo-glm/
### 1. 安装依赖
```bash
pip install -r requirements.txt
pip install -r backend/requirements.txt
```
### 2. 启动Milvus向量数据库
@@ -76,10 +58,12 @@ docker-compose logs -f milvus
python tests/verify_mvp.py
```
根级测试脚本会自动把 `backend/` 加入导入路径，并从 `app.*` 加载当前后端代码。
### 4. 启动API服务
```bash
uvicorn src.api.main:app --reload --port 8000
PYTHONPATH=backend uvicorn app.main:app --reload --port 8000
```
访问API文档：http://localhost:8000/docs
@@ -140,4 +124,4 @@ CHUNK_SIZE=512
## 许可证
MIT License
MIT License

7
backend/README.md
View File

@@ -1,6 +1,6 @@
# AI+合规智能中枢后端
`backend` 已承接原 `src` 的完整 FastAPI 后端能力，当前正式入口为 `app.main:app`。
`backend` 是当前正式使用的 FastAPI 后端目录，入口为 `app.main:app`。
## 启动
@@ -9,10 +9,10 @@ pip install -r backend/requirements.txt
PYTHONPATH=backend uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
```
也可以直接使用根目录脚本：
也可以直接使用根目录统一脚本：
```bash
./start_api.sh
./dev.sh start api --foreground
```
## 主要接口
@@ -45,6 +45,5 @@ backend/
## 说明
- `backend/app/api/main.py` 来自原 `src/api/main.py`，已切换为 `app.*` 导入。
- 路由前缀保持为 `/api/v1`，以兼容当前前端。
- 原 `backend/app/api/routes/docs.py`、`rag.py`、`compliance.py`、`status.py` 仍保留在仓库中，但不再作为主路由入口。

3
backend/app/api/__init__.py
View File

@@ -1,2 +1 @@
# src/api/__init__.py
"""API接口模块"""
"""API接口模块"""

3
backend/app/api/models/__init__.py
View File

@@ -1,4 +1,3 @@
# src/api/models/__init__.py
"""API数据模型"""
from .document import (
@@ -19,4 +18,4 @@ __all__ = [
"SearchResponse",
"DocumentStatusResponse",
"ErrorResponse"
]
]

3
backend/app/api/models/document.py
View File

@@ -1,4 +1,3 @@
# src/api/models/document.py
"""文档相关Pydantic数据模型"""
from pydantic import BaseModel, Field
@@ -60,4 +59,4 @@ class ErrorResponse(BaseModel):
"""错误响应"""
error: str = Field(..., description="错误类型")
message: str = Field(..., description="错误消息")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")

3
backend/app/api/routes/__init__.py
View File

@@ -1,4 +1,3 @@
# src/api/routes/__init__.py
"""API路由模块"""
from fastapi import APIRouter
@@ -14,4 +13,4 @@ api_router.include_router(documents_router)
api_router.include_router(knowledge_router)
api_router.include_router(agent_router)
__all__ = ["api_router", "documents_router", "knowledge_router", "agent_router"]
__all__ = ["api_router", "documents_router", "knowledge_router", "agent_router"]

1
backend/app/api/routes/agent.py
View File

@@ -1,4 +1,3 @@
# src/api/routes/agent.py
"""Agent API接口 - 问答对话接口"""
from fastapi import APIRouter, HTTPException, Depends

1
backend/app/api/routes/documents.py
View File

@@ -1,4 +1,3 @@
# src/api/routes/documents.py
"""文档上传与处理接口"""
from fastapi import APIRouter, UploadFile, File, Form, HTTPException

1
backend/app/api/routes/knowledge.py
View File

@@ -1,4 +1,3 @@
# src/api/routes/knowledge.py
"""知识库检索接口"""
from fastapi import APIRouter, HTTPException

1
backend/app/config/__init__.py
View File

@@ -1,4 +1,3 @@
# src/config/__init__.py
"""配置模块"""
from .settings import Settings, get_settings, settings

3
backend/app/config/logging.py
View File

@@ -1,4 +1,3 @@
# src/config/logging.py
"""日志配置"""
from loguru import logger
@@ -29,4 +28,4 @@ def setup_logging(level: str = "INFO"):
compression="zip"
)
return logger
return logger

1
backend/app/config/settings.py
View File

@@ -1,4 +1,3 @@
# src/config/settings.py
"""配置管理 - 环境变量和默认配置"""
from pydantic_settings import BaseSettings

3
backend/app/services/agent/__init__.py
View File

@@ -1,7 +1,6 @@
# src/services/agent/__init__.py
"""Agent服务模块"""
from .qa_agent import QAAgent, ask_compliance_question
from .session_manager import SessionManager, ChatSession
__all__ = ["QAAgent", "ask_compliance_question", "SessionManager", "ChatSession"]
__all__ = ["QAAgent", "ask_compliance_question", "SessionManager", "ChatSession"]

1
backend/app/services/agent/qa_agent.py
View File

@@ -1,4 +1,3 @@
# src/services/agent/qa_agent.py
"""RAG问答Agent - 合规智能问答核心实现"""
import time

3
backend/app/services/agent/session_manager.py
View File

@@ -1,4 +1,3 @@
# src/services/agent/session_manager.py
"""多轮对话会话管理"""
import time
@@ -244,4 +243,4 @@ class SessionManager:
def clear_all_sessions(self):
"""清空所有会话"""
self._sessions.clear()
logger.info("所有会话已清空")
logger.info("所有会话已清空")

1
backend/app/services/document_processor.py
View File

@@ -1,4 +1,3 @@
# src/services/document_processor.py
"""文档处理主流程 - 解析→摘要→分块→嵌入→入库"""
import os

3
backend/app/services/embedding/__init__.py
View File

@@ -1,7 +1,6 @@
# src/services/embedding/__init__.py
"""嵌入和分块服务"""
from .text_chunker import RegulationChunker
from .bge_m3_embedder import BGEM3Embedder
__all__ = ["RegulationChunker", "BGEM3Embedder"]
__all__ = ["RegulationChunker", "BGEM3Embedder"]

3
backend/app/services/embedding/bge_m3_embedder.py
View File

@@ -1,4 +1,3 @@
# src/services/embedding/bge_m3_embedder.py
"""BGE-M3嵌入服务 - Dense+Sparse双路向量生成"""
import numpy as np
@@ -293,4 +292,4 @@ def embed_single_text(
) -> Dict:
"""便捷函数：对单个文本生成嵌入"""
embedder = BGEM3Embedder(model_name=model_name, **kwargs)
return embedder.embed_single(text)
return embedder.embed_single(text)

3
backend/app/services/embedding/text_chunker.py
View File

@@ -1,4 +1,3 @@
# src/services/embedding/text_chunker.py
"""智能分块器 - 章节级+条款级双粒度切割"""
import re
@@ -446,4 +445,4 @@ def chunk_regulation_document(
doc_name,
regulation_type,
version
)
)

3
backend/app/services/llm/__init__.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/__init__.py
"""LLM服务模块"""
from .llm_factory import LLMFactory, get_llm_client
@@ -12,4 +11,4 @@ __all__ = [
"BaseLLMClient", "LLMResponse", "LLMConfig", "LLMProvider",
"DeepSeekClient", "QwenClient", "QwenVLClient",
"DocumentSummarizer", "summarize_document", "DocumentSummary"
]
]

3
backend/app/services/llm/base_client.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/base_client.py
"""LLM客户端基类 - 统一接口定义"""
from abc import ABC, abstractmethod
@@ -113,4 +112,4 @@ class BaseLLMClient(ABC):
# 中文字符约1.5 token，英文约0.25 token
chinese_chars = sum(1 for c in text if '一' <= c <= '鿿')
other_chars = len(text) - chinese_chars
return int(chinese_chars * 1.5 + other_chars * 0.25)
return int(chinese_chars * 1.5 + other_chars * 0.25)

1
backend/app/services/llm/deepseek_client.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/deepseek_client.py
"""DeepSeek LLM客户端 - OpenAI兼容API"""
import time

1
backend/app/services/llm/document_summarizer.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/document_summarizer.py
"""文档摘要生成服务 - LLM生成法规文档摘要"""
from typing import Dict, Optional

1
backend/app/services/llm/llm_factory.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/llm_factory.py
"""LLM工厂 - 统一创建和管理LLM客户端"""
from typing import Optional, Dict, Any

1
backend/app/services/llm/qwen_client.py
View File

@@ -1,4 +1,3 @@
# src/services/llm/qwen_client.py
"""Qwen LLM客户端 - 支持OpenAI兼容API格式"""
import time

3
backend/app/services/parser/__init__.py
View File

@@ -1,7 +1,6 @@
# src/services/parser/__init__.py
"""文档解析服务"""
from .pdf_parser import PDFParser
from .docx_parser import DocxParser
__all__ = ["PDFParser", "DocxParser"]
__all__ = ["PDFParser", "DocxParser"]

3
backend/app/services/parser/docx_parser.py
View File

@@ -1,4 +1,3 @@
# src/services/parser/docx_parser.py
"""Word文档解析 - 使用python-docx"""
from docx import Document
@@ -284,4 +283,4 @@ def parse_docx(file_path: str) -> DocxDocumentContent:
def parse_docx_to_markdown(file_path: str) -> str:
"""便捷函数：解析Word并返回Markdown"""
parser = DocxParser()
return parser.parse_to_markdown(file_path)
return parser.parse_to_markdown(file_path)

3
backend/app/services/parser/mineru_parser.py
View File

@@ -1,4 +1,3 @@
# src/services/parser/mineru_parser.py
"""MinerU多模态PDF解析 - 版面感知解析"""
from typing import Optional, Dict
@@ -201,4 +200,4 @@ def parse_with_mineru(file_path: str) -> MinerUResult:
def parse_pdf_smart(file_path: str) -> str:
"""便捷函数：智能解析PDF（自动选择最佳解析器）"""
orchestrator = ParserOrchestrator()
return orchestrator.parse_pdf(file_path)
return orchestrator.parse_pdf(file_path)

3
backend/app/services/parser/pdf_parser.py
View File

@@ -1,4 +1,3 @@
# src/services/parser/pdf_parser.py
"""PDF文档解析 - 使用PyMuPDF基础解析"""
import fitz # PyMuPDF
@@ -265,4 +264,4 @@ def parse_pdf(file_path: str, **kwargs) -> PDFDocumentContent:
def parse_pdf_to_markdown(file_path: str) -> str:
"""便捷函数：解析PDF并返回Markdown"""
parser = PDFParser()
return parser.parse_to_markdown(file_path)
return parser.parse_to_markdown(file_path)

3
backend/app/services/rag/__init__.py
View File

@@ -1,4 +1,3 @@
# src/services/rag/__init__.py
"""RAG服务模块"""
from .retriever import Retriever, retrieve_regulations
@@ -9,4 +8,4 @@ __all__ = [
"Retriever", "retrieve_regulations",
"ContextBuilder", "build_rag_context",
"PromptTemplates", "get_prompt_template"
]
]

1
backend/app/services/rag/context_builder.py
View File

@@ -1,4 +1,3 @@
# src/services/rag/context_builder.py
"""RAG上下文构建服务 - 构建LLM输入上下文"""
from typing import List, Dict, Optional

3
backend/app/services/rag/prompt_templates.py
View File

@@ -1,4 +1,3 @@
# src/services/rag/prompt_templates.py
"""RAG Prompt模板 - 合规问答专用Prompt"""
from typing import Dict, Optional
@@ -293,4 +292,4 @@ def get_prompt_template(name: str) -> PromptTemplate:
template = PromptTemplates.get_template(name)
if not template:
raise ValueError(f"不存在的模板: {name}")
return template
return template

1
backend/app/services/rag/retriever.py
View File

@@ -1,4 +1,3 @@
# src/services/rag/retriever.py
"""RAG检索服务 - 封装Milvus检索"""
from typing import List, Dict, Optional, Any

3
backend/app/services/storage/__init__.py
View File

@@ -1,7 +1,6 @@
# src/services/storage/__init__.py
"""存储服务"""
from .milvus_client import MilvusClient
from .minio_client import MinIOClient
__all__ = ["MilvusClient", "MinIOClient"]
__all__ = ["MilvusClient", "MinIOClient"]

1
backend/app/services/storage/milvus_client.py
View File

@@ -1,4 +1,3 @@
# src/services/storage/milvus_client.py
"""Milvus向量数据库客户端 - 存储与检索服务"""
from pymilvus import (

1
backend/app/services/storage/minio_client.py
View File

@@ -1,4 +1,3 @@
# src/services/storage/minio_client.py
"""MinIO对象存储客户端 - 文档文件存储"""
from minio import Minio

3
backend/app/workers/__init__.py
View File

@@ -1,2 +1 @@
# src/workers/__init__.py
"""异步任务Worker模块"""
"""异步任务Worker模块"""

35
backend/pyproject.toml
View File

@@ -1,35 +0,0 @@
[project]
name = "ai-regulations-backend"
version = "0.1.0"
description = "Migrated FastAPI backend for AI regulations demo"
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"fastapi>=0.110.0",
"uvicorn[standard]>=0.27.0",
"python-multipart>=0.0.9",
"pydantic>=2.0.0",
"pydantic-settings>=2.0.0",
"python-dotenv>=1.0.0",
"loguru>=0.7.0",
"httpx>=0.25.0",
"tiktoken>=0.5.0",
"tenacity>=8.2.0",
"pymilvus>=2.4.0",
"minio>=7.1.0",
"pymupdf>=1.24.0",
"python-docx>=1.1.0",
"FlagEmbedding>=1.2.0",
"sentence-transformers>=2.2.0",
"torch>=2.0.0",
"numpy>=1.24.0",
"langchain>=0.1.0",
"langchain-milvus>=0.1.0",
]
[project.scripts]
backend = "main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

630
dev.bat Normal file
View File

@@ -0,0 +1,630 @@
@echo off
setlocal EnableExtensions EnableDelayedExpansion
set "ROOT_DIR=%~dp0"
pushd "%ROOT_DIR%" >nul
set "VENV_DIR=.venv"
set "VENV_PYTHON=%VENV_DIR%\Scripts\python.exe"
set "LOG_DIR=logs"
set "API_PID_FILE=%LOG_DIR%\api.pid"
set "FRONTEND_PID_FILE=%LOG_DIR%\frontend.pid"
set "API_LOG_FILE=%CD%\%LOG_DIR%\api.log"
set "FRONTEND_LOG_FILE=%CD%\%LOG_DIR%\frontend.log"
call :load_env
if not defined API_HOST set "API_HOST=0.0.0.0"
if not defined API_PORT set "API_PORT=8000"
if not defined FRONTEND_PORT set "FRONTEND_PORT=5173"
if not defined FRONTEND_MODE set "FRONTEND_MODE=dev"
if "%~1"=="" goto help
if /I "%~1"=="help" goto help
if /I "%~1"=="-h" goto help
if /I "%~1"=="--help" goto help
if /I "%~1"=="setup" goto setup
if /I "%~1"=="start" goto start
if /I "%~1"=="stop" goto stop
if /I "%~1"=="restart" goto restart
if /I "%~1"=="status" goto status
if /I "%~1"=="logs" goto logs
echo Unknown command: %~1
echo Use dev.bat help to see available commands.
exit /b 1
:help
echo.
echo ========================================
echo AI+合规智能中枢统一脚本（Windows）
echo ========================================
echo.
echo Usage:
echo dev.bat help
echo dev.bat setup
echo dev.bat start [all^|api^|frontend] [--foreground] [--mode dev^|static]
echo dev.bat stop [all^|api^|frontend]
echo dev.bat restart [all^|api^|frontend] [--mode dev^|static]
echo dev.bat status
echo dev.bat logs ^<api^|frontend^> [--follow]
echo.
echo Command details:
echo help
echo Show the full command list, default ports, log paths and examples.
echo.
echo setup
echo Create .venv if missing, install backend dependencies from backend\requirements.txt,
echo run npm install in frontend, then report Docker base service status.
echo.
echo start
echo Start services. Default target is all.
echo start api --foreground Run uvicorn in the current window with --reload for debugging.
echo start frontend --mode static Build the frontend and serve frontend\dist as static files.
echo.
echo stop
echo Stop services. It first uses logs\*.pid, then falls back to listeners on the configured ports.
echo.
echo restart
echo Stop and then start the selected target.
echo.
echo status
echo Show API, frontend and Docker service status. API status also checks /health.
echo.
echo logs
echo Show the last 50 lines by default. Add --follow to stream updates.
echo.
echo Defaults:
echo API_HOST = %API_HOST%
echo API_PORT = %API_PORT%
echo FRONTEND_PORT = %FRONTEND_PORT%
echo FRONTEND_MODE = %FRONTEND_MODE%
echo Log files = logs\api.log, logs\frontend.log
echo.
echo Examples:
echo dev.bat setup
echo dev.bat start
echo dev.bat start api --foreground
echo dev.bat start frontend --mode static
echo dev.bat restart frontend --mode dev
echo dev.bat status
echo dev.bat logs api --follow
echo.
exit /b 0
:setup
call :ensure_log_dir
echo.
echo ========================================
echo AI+合规智能中枢 - 环境初始化
echo ========================================
echo.
call :resolve_python_bootstrap || exit /b 1
call %PYTHON_BOOTSTRAP% -c "import sys; raise SystemExit(0 if sys.version_info ^>= (3, 10) else 1)" || (
echo Python 3.10+ is required.
exit /b 1
)
if not exist "%VENV_PYTHON%" (
call %PYTHON_BOOTSTRAP% -m venv "%VENV_DIR%" || exit /b 1
echo Created virtual environment: %VENV_DIR%
) else (
echo Virtual environment already exists: %VENV_DIR%
)
"%VENV_PYTHON%" -m pip install --upgrade pip || exit /b 1
"%VENV_PYTHON%" -m pip install -r backend\requirements.txt || exit /b 1
echo Backend dependencies installed.
if not exist "frontend\package.json" (
echo Frontend package.json not found.
exit /b 1
)
where npm >nul 2>nul || (
echo npm was not found. Install Node.js 20+ first.
exit /b 1
)
pushd frontend >nul
call npm install || (
popd >nul
exit /b 1
)
popd >nul
echo Frontend dependencies installed.
echo.
echo Docker base services:
for %%C in (milvus minio redis postgres) do call :print_docker_status %%C
echo.
echo Setup complete.
echo dev.bat start
echo dev.bat status
echo dev.bat logs api --follow
exit /b 0
:start
set "TARGET=all"
set "MODE=%FRONTEND_MODE%"
set "FOREGROUND=0"
if /I "%~2"=="all" set "TARGET=all"
if /I "%~2"=="api" set "TARGET=api"
if /I "%~2"=="frontend" set "TARGET=frontend"
call :parse_start_options %* || exit /b 1
if /I "%TARGET%"=="all" (
if "%FOREGROUND%"=="1" (
echo start all does not support --foreground.
exit /b 1
)
call :start_api_background || exit /b 1
call :start_frontend "%MODE%" || exit /b 1
exit /b 0
)
if /I "%TARGET%"=="api" (
if "%FOREGROUND%"=="1" (
call :start_api_foreground
exit /b %errorlevel%
)
call :start_api_background
exit /b %errorlevel%
)
if /I "%TARGET%"=="frontend" (
call :start_frontend "%MODE%"
exit /b %errorlevel%
)
echo Invalid start target.
exit /b 1
:parse_start_options
set "MODE=%FRONTEND_MODE%"
set "FOREGROUND=0"
:parse_start_options_loop
if "%~1"=="" exit /b 0
if /I "%~1"=="start" (
shift
goto parse_start_options_loop
)
if /I "%~1"=="all" (
shift
goto parse_start_options_loop
)
if /I "%~1"=="api" (
shift
goto parse_start_options_loop
)
if /I "%~1"=="frontend" (
shift
goto parse_start_options_loop
)
if /I "%~1"=="--foreground" (
set "FOREGROUND=1"
shift
goto parse_start_options_loop
)
if /I "%~1"=="--mode" (
if "%~2"=="" (
echo --mode requires dev or static.
exit /b 1
)
set "MODE=%~2"
call :validate_mode "%MODE%" || exit /b 1
shift
shift
goto parse_start_options_loop
)
echo Unknown argument: %~1
exit /b 1
:stop
set "TARGET=all"
if /I "%~2"=="api" set "TARGET=api"
if /I "%~2"=="frontend" set "TARGET=frontend"
if /I "%TARGET%"=="all" (
call :stop_frontend
call :stop_api
exit /b 0
)
if /I "%TARGET%"=="api" (
call :stop_api
exit /b 0
)
if /I "%TARGET%"=="frontend" (
call :stop_frontend
exit /b 0
)
echo Invalid stop target.
exit /b 1
:restart
set "TARGET=all"
set "MODE=%FRONTEND_MODE%"
if /I "%~2"=="api" set "TARGET=api"
if /I "%~2"=="frontend" set "TARGET=frontend"
call :parse_restart_options %* || exit /b 1
if /I "%TARGET%"=="all" (
call :stop_frontend
call :stop_api
call :start_api_background || exit /b 1
call :start_frontend "%MODE%" || exit /b 1
exit /b 0
)
if /I "%TARGET%"=="api" (
call :stop_api
call :start_api_background
exit /b %errorlevel%
)
if /I "%TARGET%"=="frontend" (
call :stop_frontend
call :start_frontend "%MODE%"
exit /b %errorlevel%
)
echo Invalid restart target.
exit /b 1
:parse_restart_options
set "MODE=%FRONTEND_MODE%"
:parse_restart_options_loop
if "%~1"=="" exit /b 0
if /I "%~1"=="restart" (
shift
goto parse_restart_options_loop
)
if /I "%~1"=="all" (
shift
goto parse_restart_options_loop
)
if /I "%~1"=="api" (
shift
goto parse_restart_options_loop
)
if /I "%~1"=="frontend" (
shift
goto parse_restart_options_loop
)
if /I "%~1"=="--mode" (
if "%~2"=="" (
echo --mode requires dev or static.
exit /b 1
)
set "MODE=%~2"
call :validate_mode "%MODE%" || exit /b 1
shift
shift
goto parse_restart_options_loop
)
echo Unknown argument: %~1
exit /b 1
:status
call :ensure_log_dir
echo.
echo ========================================
echo AI+合规智能中枢 - 服务状态
echo ========================================
echo.
echo API service:
set "API_PID="
set "API_RUNNING=0"
if exist "%API_PID_FILE%" set /p API_PID=<"%API_PID_FILE%"
if defined API_PID (
call :pid_exists %API_PID%
if not errorlevel 1 (
set "API_RUNNING=1"
echo Status: running
echo PID: %API_PID%
goto api_health
) else (
del /q "%API_PID_FILE%" >nul 2>nul
set "API_PID="
)
)
call :get_listener_pid %API_PORT% API_LISTENER
if defined API_LISTENER (
set "API_RUNNING=1"
echo Status: running (no PID file)
echo PID: %API_LISTENER%
) else (
echo Status: stopped
goto api_done
)
:api_health
if "%API_RUNNING%"=="1" (
call :check_api_health
if not errorlevel 1 (
echo Health: ok
) else (
echo Health: failed
)
)
:api_done
echo URL: http://localhost:%API_PORT%
echo Docs: http://localhost:%API_PORT%/docs
echo.
echo Frontend service:
set "FRONTEND_PID="
if exist "%FRONTEND_PID_FILE%" set /p FRONTEND_PID=<"%FRONTEND_PID_FILE%"
if defined FRONTEND_PID (
call :pid_exists %FRONTEND_PID%
if not errorlevel 1 (
echo Status: running
echo PID: %FRONTEND_PID%
goto frontend_done
) else (
del /q "%FRONTEND_PID_FILE%" >nul 2>nul
set "FRONTEND_PID="
)
)
call :get_listener_pid %FRONTEND_PORT% FRONTEND_LISTENER
if defined FRONTEND_LISTENER (
echo Status: running (no PID file)
echo PID: %FRONTEND_LISTENER%
) else (
echo Status: stopped
)
:frontend_done
echo Mode: %FRONTEND_MODE%
echo URL: http://localhost:%FRONTEND_PORT%
echo.
echo Docker services:
where docker >nul 2>nul || (
echo Docker not installed.
exit /b 0
)
for %%C in (milvus minio redis postgres) do call :print_docker_status %%C
exit /b 0
:logs
if "%~2"=="" (
echo Specify api or frontend.
exit /b 1
)
if /I "%~2"=="api" (
set "LOG_FILE=%API_LOG_FILE%"
)
if /I "%~2"=="frontend" (
set "LOG_FILE=%FRONTEND_LOG_FILE%"
)
if not defined LOG_FILE (
echo Unknown log target: %~2
exit /b 1
)
if not exist "%LOG_FILE%" (
echo Log file not found: %LOG_FILE%
exit /b 1
)
if /I "%~3"=="--follow" (
pwsh -NoProfile -Command "Get-Content -LiteralPath '%LOG_FILE%' -Wait"
) else (
pwsh -NoProfile -Command "Get-Content -LiteralPath '%LOG_FILE%' -Tail 50"
)
exit /b %errorlevel%
:start_api_background
call :ensure_log_dir
if not exist "%VENV_PYTHON%" (
echo Virtual environment not found. Run dev.bat setup first.
exit /b 1
)
set "EXISTING_PID="
if exist "%API_PID_FILE%" set /p EXISTING_PID=<"%API_PID_FILE%"
if defined EXISTING_PID (
call :pid_exists %EXISTING_PID%
if not errorlevel 1 (
echo API is already running. PID: %EXISTING_PID%
exit /b 0
) else del /q "%API_PID_FILE%" >nul 2>nul
)
for /f %%P in ('pwsh -NoProfile -Command "$p = Start-Process -FilePath 'pwsh' -ArgumentList '-NoProfile','-Command',('Set-Location ''%CD%''; $env:PYTHONPATH=''backend''; & ''.\%VENV_PYTHON%'' -m uvicorn app.main:app --host ''%API_HOST%'' --port ''%API_PORT%'' *^> ''%API_LOG_FILE%''') -WindowStyle Hidden -PassThru; $p.Id"') do set "API_PID=%%P"
if not defined API_PID (
echo Failed to start API.
exit /b 1
)
> "%API_PID_FILE%" echo %API_PID%
timeout /t 3 /nobreak >nul
call :pid_exists %API_PID%
if errorlevel 1 (
del /q "%API_PID_FILE%" >nul 2>nul
echo API failed to start. Check logs\api.log
exit /b 1
)
echo API started. PID: %API_PID%
echo URL: http://localhost:%API_PORT%
echo Docs: http://localhost:%API_PORT%/docs
echo Log: logs\api.log
exit /b 0
:start_api_foreground
if not exist "%VENV_PYTHON%" (
echo Virtual environment not found. Run dev.bat setup first.
exit /b 1
)
set "PYTHONPATH=backend;%PYTHONPATH%"
echo API running in foreground with reload enabled.
echo URL: http://localhost:%API_PORT%
echo Docs: http://localhost:%API_PORT%/docs
"%VENV_PYTHON%" -m uvicorn app.main:app --host "%API_HOST%" --port "%API_PORT%" --reload
exit /b %errorlevel%
:start_frontend
set "MODE=%~1"
if "%MODE%"=="" set "MODE=%FRONTEND_MODE%"
call :ensure_log_dir
where npm >nul 2>nul || (
echo npm was not found. Install Node.js 20+ first.
exit /b 1
)
if not exist "frontend\node_modules\vite" (
echo Frontend dependencies are missing. Running npm install...
pushd frontend >nul
call npm install || (
popd >nul
exit /b 1
)
popd >nul
)
set "EXISTING_PID="
if exist "%FRONTEND_PID_FILE%" set /p EXISTING_PID=<"%FRONTEND_PID_FILE%"
if defined EXISTING_PID (
call :pid_exists %EXISTING_PID%
if not errorlevel 1 (
echo Frontend is already running. PID: %EXISTING_PID%
exit /b 0
) else del /q "%FRONTEND_PID_FILE%" >nul 2>nul
)
if /I "%MODE%"=="static" (
pushd frontend >nul
call npm run build || (
popd >nul
exit /b 1
)
popd >nul
call :resolve_python_bootstrap || exit /b 1
for /f %%P in ('pwsh -NoProfile -Command "$p = Start-Process -FilePath 'pwsh' -ArgumentList '-NoProfile','-Command',('Set-Location ''%CD%''; & %PYTHON_BOOTSTRAP% -m http.server %FRONTEND_PORT% --bind 0.0.0.0 --directory frontend/dist *^> ''%FRONTEND_LOG_FILE%''') -WindowStyle Hidden -PassThru; $p.Id"') do set "FRONTEND_PID=%%P"
) else (
for /f %%P in ('pwsh -NoProfile -Command "$p = Start-Process -FilePath 'pwsh' -ArgumentList '-NoProfile','-Command',('Set-Location ''%CD%''; & npm --prefix frontend run dev -- --host 0.0.0.0 --port %FRONTEND_PORT% *^> ''%FRONTEND_LOG_FILE%''') -WindowStyle Hidden -PassThru; $p.Id"') do set "FRONTEND_PID=%%P"
)
if not defined FRONTEND_PID (
echo Failed to start frontend.
exit /b 1
)
> "%FRONTEND_PID_FILE%" echo %FRONTEND_PID%
timeout /t 4 /nobreak >nul
call :pid_exists %FRONTEND_PID%
if errorlevel 1 (
del /q "%FRONTEND_PID_FILE%" >nul 2>nul
echo Frontend failed to start. Check logs\frontend.log
exit /b 1
)
echo Frontend started. PID: %FRONTEND_PID%
echo URL: http://localhost:%FRONTEND_PORT%
echo Mode: %MODE%
echo Log: logs\frontend.log
exit /b 0
:stop_api
set "API_PID="
if exist "%API_PID_FILE%" set /p API_PID=<"%API_PID_FILE%"
if defined API_PID (
call :stop_pid %API_PID%
del /q "%API_PID_FILE%" >nul 2>nul
echo API stopped.
exit /b 0
)
call :get_listener_pid %API_PORT% API_PORT_PID
if defined API_PORT_PID (
call :stop_pid %API_PORT_PID%
echo Stopped process listening on API port %API_PORT%.
exit /b 0
)
echo API is not running.
exit /b 0
:stop_frontend
set "FRONTEND_PID="
if exist "%FRONTEND_PID_FILE%" set /p FRONTEND_PID=<"%FRONTEND_PID_FILE%"
if defined FRONTEND_PID (
call :stop_pid %FRONTEND_PID%
del /q "%FRONTEND_PID_FILE%" >nul 2>nul
echo Frontend stopped.
exit /b 0
)
call :get_listener_pid %FRONTEND_PORT% FRONTEND_PORT_PID
if defined FRONTEND_PORT_PID (
call :stop_pid %FRONTEND_PORT_PID%
echo Stopped process listening on frontend port %FRONTEND_PORT%.
exit /b 0
)
echo Frontend is not running.
exit /b 0
:ensure_log_dir
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
exit /b 0
:validate_mode
if /I "%~1"=="dev" exit /b 0
if /I "%~1"=="static" exit /b 0
echo Invalid frontend mode: %~1
exit /b 1
:resolve_python_bootstrap
where python >nul 2>nul && (
set "PYTHON_BOOTSTRAP=python"
exit /b 0
)
where py >nul 2>nul && (
set "PYTHON_BOOTSTRAP=py -3"
exit /b 0
)
echo Python was not found.
exit /b 1
:load_env
for %%K in (API_HOST API_PORT FRONTEND_PORT FRONTEND_MODE) do call :read_env %%K
exit /b 0
:read_env
if not exist ".env" exit /b 0
for /f "usebackq tokens=1,* delims==" %%A in (`findstr /r /b /c:"%~1=" ".env" 2^>nul`) do set "%~1=%%B"
exit /b 0
:pid_exists
pwsh -NoProfile -Command "exit([int](-not [bool](Get-Process -Id %~1 -ErrorAction SilentlyContinue)))"
exit /b %errorlevel%
:stop_pid
pwsh -NoProfile -Command "$p = Get-Process -Id %~1 -ErrorAction SilentlyContinue; if ($p) { Stop-Process -Id %~1 -Force }"
exit /b 0
:get_listener_pid
set "%~2="
for /f %%P in ('pwsh -NoProfile -Command "(Get-NetTCPConnection -LocalPort %~1 -State Listen -ErrorAction SilentlyContinue ^| Select-Object -ExpandProperty OwningProcess -Unique ^| Select-Object -First 1)"') do set "%~2=%%P"
exit /b 0
:check_api_health
pwsh -NoProfile -Command "try { $r = Invoke-WebRequest -Uri 'http://localhost:%API_PORT%/health' -TimeoutSec 3 -UseBasicParsing; exit([int](-not ($r.Content -match 'healthy'))) } catch { exit 1 }"
exit /b %errorlevel%
:print_docker_status
docker ps --format "{{.Names}}" | findstr /x /c:"%~1" >nul && (
echo %~1: running
exit /b 0
)
docker ps -a --format "{{.Names}}" | findstr /x /c:"%~1" >nul && (
echo %~1: stopped
exit /b 0
)
echo %~1: not created
exit /b 0

710
dev.sh Normal file
View File

@@ -0,0 +1,710 @@
#!/bin/bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$ROOT_DIR"
VENV_DIR=".venv"
VENV_PYTHON="$VENV_DIR/bin/python"
LOG_DIR="logs"
API_PID_FILE="$LOG_DIR/api.pid"
FRONTEND_PID_FILE="$LOG_DIR/frontend.pid"
API_LOG_FILE="$LOG_DIR/api.log"
FRONTEND_LOG_FILE="$LOG_DIR/frontend.log"
DOCKER_CONTAINERS="milvus minio redis postgres"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m'
load_env() {
if [ -f ".env" ]; then
while IFS='=' read -r key value; do
key="${key%$'\r'}"
value="${value%$'\r'}"
case "$key" in
""|\#*)
continue
;;
API_HOST|API_PORT|FRONTEND_PORT|FRONTEND_MODE)
export "$key=$value"
;;
esac
done < ".env"
fi
API_HOST="${API_HOST:-0.0.0.0}"
API_PORT="${API_PORT:-8000}"
FRONTEND_PORT="${FRONTEND_PORT:-5173}"
FRONTEND_MODE="${FRONTEND_MODE:-dev}"
}
ensure_log_dir() {
mkdir -p "$LOG_DIR"
}
print_header() {
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} $1${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
}
info() {
echo -e "${CYAN}$1${NC}"
}
success() {
echo -e "${GREEN}$1${NC}"
}
warn() {
echo -e "${YELLOW}$1${NC}"
}
error() {
echo -e "${RED}$1${NC}" >&2
}
die() {
error "$1"
exit 1
}
is_pid_running() {
local pid="$1"
[ -n "$pid" ] && ps -p "$pid" > /dev/null 2>&1
}
read_pid() {
local pid_file="$1"
if [ -f "$pid_file" ]; then
cat "$pid_file"
fi
}
cleanup_stale_pid() {
local pid_file="$1"
local pid
pid="$(read_pid "$pid_file")"
if [ -n "$pid" ] && ! is_pid_running "$pid"; then
rm -f "$pid_file"
fi
}
port_pid() {
local port="$1"
if command -v lsof > /dev/null 2>&1; then
lsof -ti tcp:"$port" 2>/dev/null | head -n 1 || true
return 0
fi
if command -v ss > /dev/null 2>&1; then
ss -lptn "sport = :$port" 2>/dev/null | awk -F 'pid=' 'NR>1 && NF>1 {split($2,a,/,/); print a[1]; exit}' || true
return 0
fi
if command -v netstat > /dev/null 2>&1; then
netstat -lntp 2>/dev/null | awk -v port=":$port" '$4 ~ port {split($7,a,"/"); if (a[1] != "-") {print a[1]; exit}}' || true
fi
return 0
}
require_python_bootstrap() {
if command -v python3 > /dev/null 2>&1; then
PYTHON_BOOTSTRAP="python3"
elif command -v python > /dev/null 2>&1; then
PYTHON_BOOTSTRAP="python"
else
die "未找到 Python，请先安装 Python 3.10+。"
fi
}
require_venv() {
[ -x "$VENV_PYTHON" ] || die "虚拟环境不存在，请先运行 ./dev.sh setup"
}
ensure_frontend_deps() {
if [ ! -d "frontend" ]; then
die "前端目录不存在: frontend"
fi
if [ ! -d "frontend/node_modules" ] || [ ! -d "frontend/node_modules/vite" ]; then
warn "前端依赖不存在或不完整，正在执行 npm install..."
npm --prefix frontend install
fi
}
validate_frontend_mode() {
case "$1" in
dev|static)
;;
*)
die "前端模式仅支持 dev 或 static，当前值: $1"
;;
esac
}
check_python_version() {
local python_cmd="$1"
"$python_cmd" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 10) else 1)'
}
run_setup() {
load_env
ensure_log_dir
print_header "AI+合规智能中枢 - 环境初始化"
require_python_bootstrap
info "[1/4] 检查 Python 版本"
check_python_version "$PYTHON_BOOTSTRAP" || die "需要 Python 3.10+"
success "Python 版本检查通过"
echo ""
info "[2/4] 准备虚拟环境"
if [ ! -d "$VENV_DIR" ]; then
"$PYTHON_BOOTSTRAP" -m venv "$VENV_DIR"
success "已创建虚拟环境: $VENV_DIR"
else
success "虚拟环境已存在: $VENV_DIR"
fi
"$VENV_PYTHON" -m pip install --upgrade pip
"$VENV_PYTHON" -m pip install -r backend/requirements.txt
success "后端依赖安装完成"
echo ""
info "[3/4] 准备前端依赖"
if ! command -v npm > /dev/null 2>&1; then
die "未找到 npm，请先安装 Node.js 20+。"
fi
npm --prefix frontend install
success "前端依赖安装完成"
echo ""
info "[4/4] 检查 Docker 基础服务"
if command -v docker > /dev/null 2>&1; then
local container
for container in $DOCKER_CONTAINERS; do
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
success "${container}: 运行中"
elif docker ps -a --format '{{.Names}}' | grep -q "^${container}$"; then
warn "${container}: 已创建但未运行"
else
warn "${container}: 未找到容器"
fi
done
else
warn "未检测到 Docker，已跳过容器检查"
fi
echo ""
success "环境初始化完成"
echo "后续常用命令:"
echo " ./dev.sh start"
echo " ./dev.sh status"
echo " ./dev.sh logs api --follow"
}
api_health_ok() {
if command -v curl > /dev/null 2>&1; then
curl -fsS "http://localhost:$API_PORT/health" > /dev/null 2>&1
return
fi
require_venv
"$VENV_PYTHON" - <<PY > /dev/null 2>&1
import sys
from urllib.request import urlopen
try:
with urlopen("http://localhost:${API_PORT}/health", timeout=3) as response:
body = response.read().decode("utf-8", errors="ignore")
sys.exit(0 if "healthy" in body.lower() else 1)
except Exception:
sys.exit(1)
PY
}
start_api() {
local mode="${1:-background}"
load_env
ensure_log_dir
require_venv
cleanup_stale_pid "$API_PID_FILE"
local pid
pid="$(read_pid "$API_PID_FILE")"
if is_pid_running "$pid"; then
warn "API 已在运行 (PID: $pid)"
return
fi
export PYTHONPATH="backend${PYTHONPATH:+:$PYTHONPATH}"
if [ "$mode" = "foreground" ]; then
print_header "AI+合规智能中枢 - 启动 API"
echo "运行模式: 前台调试（带 --reload）"
echo "服务地址: http://localhost:$API_PORT"
echo "文档地址: http://localhost:$API_PORT/docs"
echo "健康检查: http://localhost:$API_PORT/health"
echo ""
exec "$VENV_PYTHON" -m uvicorn app.main:app --host "$API_HOST" --port "$API_PORT" --reload
fi
nohup "$VENV_PYTHON" -m uvicorn app.main:app --host "$API_HOST" --port "$API_PORT" > "$API_LOG_FILE" 2>&1 &
pid=$!
echo "$pid" > "$API_PID_FILE"
sleep 3
if is_pid_running "$pid"; then
success "API 启动成功 (PID: $pid)"
echo " 地址: http://localhost:$API_PORT"
echo " 文档: http://localhost:$API_PORT/docs"
echo " 日志: $API_LOG_FILE"
else
rm -f "$API_PID_FILE"
die "API 启动失败，请查看日志: $API_LOG_FILE"
fi
}
start_frontend() {
local mode="${1:-$FRONTEND_MODE}"
load_env
ensure_log_dir
cleanup_stale_pid "$FRONTEND_PID_FILE"
local pid
pid="$(read_pid "$FRONTEND_PID_FILE")"
if is_pid_running "$pid"; then
warn "前端已在运行 (PID: $pid)"
return
fi
if ! command -v npm > /dev/null 2>&1; then
die "未找到 npm，请先安装 Node.js 20+。"
fi
ensure_frontend_deps
if [ "$mode" = "static" ]; then
require_python_bootstrap
npm --prefix frontend run build
nohup "$PYTHON_BOOTSTRAP" -m http.server "$FRONTEND_PORT" --bind 0.0.0.0 --directory frontend/dist > "$FRONTEND_LOG_FILE" 2>&1 &
else
nohup npm --prefix frontend run dev -- --host 0.0.0.0 --port "$FRONTEND_PORT" > "$FRONTEND_LOG_FILE" 2>&1 &
fi
pid=$!
echo "$pid" > "$FRONTEND_PID_FILE"
sleep 4
if is_pid_running "$pid"; then
success "前端启动成功 (PID: $pid)"
echo " 地址: http://localhost:$FRONTEND_PORT"
echo " 模式: $mode"
echo " 日志: $FRONTEND_LOG_FILE"
else
rm -f "$FRONTEND_PID_FILE"
die "前端启动失败，请查看日志: $FRONTEND_LOG_FILE"
fi
}
kill_pid_file_process() {
local pid_file="$1"
local label="$2"
local pid
pid="$(read_pid "$pid_file")"
if ! is_pid_running "$pid"; then
rm -f "$pid_file"
warn "$label 未通过 PID 文件发现运行中的进程"
return 1
fi
kill "$pid" 2>/dev/null || true
sleep 2
if is_pid_running "$pid"; then
kill -9 "$pid" 2>/dev/null || true
fi
rm -f "$pid_file"
success "$label 已停止"
return 0
}
stop_api() {
load_env
ensure_log_dir
if kill_pid_file_process "$API_PID_FILE" "API"; then
return
fi
local port_listener
port_listener="$(port_pid "$API_PORT")"
if [ -n "$port_listener" ]; then
kill "$port_listener" 2>/dev/null || true
sleep 1
if is_pid_running "$port_listener"; then
kill -9 "$port_listener" 2>/dev/null || true
fi
success "已停止监听 API 端口 $API_PORT 的进程 (PID: $port_listener)"
else
warn "未发现运行中的 API 服务"
fi
rm -f "$API_PID_FILE"
}
stop_frontend() {
load_env
ensure_log_dir
if kill_pid_file_process "$FRONTEND_PID_FILE" "前端"; then
return
fi
local port_listener
port_listener="$(port_pid "$FRONTEND_PORT")"
if [ -n "$port_listener" ]; then
kill "$port_listener" 2>/dev/null || true
sleep 1
if is_pid_running "$port_listener"; then
kill -9 "$port_listener" 2>/dev/null || true
fi
success "已停止监听前端端口 $FRONTEND_PORT 的进程 (PID: $port_listener)"
else
warn "未发现运行中的前端服务"
fi
rm -f "$FRONTEND_PID_FILE"
}
run_status() {
load_env
ensure_log_dir
print_header "AI+合规智能中枢 - 服务状态"
cleanup_stale_pid "$API_PID_FILE"
cleanup_stale_pid "$FRONTEND_PID_FILE"
local api_running=false
local frontend_running=false
local pid
local port_listener
echo -e "${YELLOW}API 服务:${NC}"
pid="$(read_pid "$API_PID_FILE")"
if is_pid_running "$pid"; then
api_running=true
success " 状态: 运行中"
echo " PID: $pid"
else
port_listener="$(port_pid "$API_PORT")"
if [ -n "$port_listener" ]; then
api_running=true
success " 状态: 运行中 (无 PID 文件)"
echo " PID: $port_listener"
else
error " 状态: 已停止"
fi
fi
if [ "$api_running" = true ]; then
if api_health_ok; then
success " 健康检查: 正常"
else
warn " 健康检查: 未通过"
fi
fi
echo " 地址: http://localhost:$API_PORT"
echo " 文档: http://localhost:${API_PORT}/docs"
echo ""
echo -e "${YELLOW}前端服务:${NC}"
pid="$(read_pid "$FRONTEND_PID_FILE")"
if is_pid_running "$pid"; then
frontend_running=true
success " 状态: 运行中"
echo " PID: $pid"
else
port_listener="$(port_pid "$FRONTEND_PORT")"
if [ -n "$port_listener" ]; then
frontend_running=true
success " 状态: 运行中 (无 PID 文件)"
echo " PID: $port_listener"
else
error " 状态: 已停止"
fi
fi
echo " 模式: $FRONTEND_MODE"
echo " 地址: http://localhost:$FRONTEND_PORT"
echo ""
echo -e "${YELLOW}Docker 服务:${NC}"
if command -v docker > /dev/null 2>&1; then
local container
for container in $DOCKER_CONTAINERS; do
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
success " ${container}: 运行中"
elif docker ps -a --format '{{.Names}}' | grep -q "^${container}$"; then
warn " ${container}: 已停止"
else
warn " ${container}: 未创建"
fi
done
else
warn " Docker 未安装，已跳过"
fi
echo ""
if [ "$api_running" = true ] && [ "$frontend_running" = true ]; then
success "所有核心服务均在运行"
else
warn "存在未运行的服务，可使用 ./dev.sh start 启动"
fi
}
run_logs() {
local target="${1:-}"
local follow="${2:-}"
local log_file
case "$target" in
api)
log_file="$API_LOG_FILE"
;;
frontend)
log_file="$FRONTEND_LOG_FILE"
;;
*)
die "请指定日志类型: api 或 frontend"
;;
esac
[ -f "$log_file" ] || die "日志文件不存在: $log_file"
if [ "$follow" = "--follow" ]; then
tail -f "$log_file"
else
tail -n 50 "$log_file"
fi
}
show_help() {
cat <<'EOF'
AI+合规智能中枢统一脚本
用法:
./dev.sh help
./dev.sh setup
./dev.sh start [all|api|frontend] [--foreground] [--mode dev|static]
./dev.sh stop [all|api|frontend]
./dev.sh restart [all|api|frontend] [--mode dev|static]
./dev.sh status
./dev.sh logs <api|frontend> [--follow]
命令说明:
help
输出完整帮助信息、默认端口、日志目录和常见示例。
setup
进行一次性的本地初始化。
包含 Python 版本检查、.venv 虚拟环境创建、后端依赖安装、前端 npm install、
以及 Docker 基础容器状态检查。
start
启动服务。默认行为等同于 ./dev.sh start all。
可选目标:
all 同时启动 API 和前端。
api 只启动后端 API。
frontend 只启动前端。
可选参数:
--foreground 仅对 start api 生效，前台运行并开启 --reload，便于调试。
--mode dev 前端使用 Vite 开发服务器，默认端口 5173。
--mode static 前端先执行 npm run build，再以静态文件服务器方式启动。
stop
停止服务。默认行为等同于 ./dev.sh stop all。
会优先读取 logs/*.pid，PID 文件失效时会回退到端口探测。
restart
先停止再启动，支持 all/api/frontend。
restart frontend --mode static 可直接切换前端启动模式。
status
查看 API、前端、Docker 基础容器的状态。
API 状态包含健康检查；前端状态包含当前模式和访问地址。
logs
查看日志。默认输出最后 50 行。
追加 --follow 后会持续跟踪日志输出。
默认约定:
API_HOST 默认 0.0.0.0
API_PORT 默认 8000
FRONTEND_PORT 默认 5173
FRONTEND_MODE 默认 dev
日志目录 logs/
PID 文件 logs/api.pid, logs/frontend.pid
常用示例:
./dev.sh setup
./dev.sh start
./dev.sh start api --foreground
./dev.sh start frontend --mode static
./dev.sh restart frontend --mode dev
./dev.sh status
./dev.sh logs api --follow
./dev.sh logs frontend
EOF
}
parse_target() {
local default_target="$1"
local candidate="${2:-}"
case "$candidate" in
all|api|frontend)
echo "$candidate"
;;
*)
echo "$default_target"
;;
esac
}
main() {
local command="${1:-help}"
local target="all"
local mode=""
local foreground=false
local log_target=""
shift || true
load_env
case "$command" in
help|-h|--help)
show_help
;;
setup)
run_setup
;;
start)
target="$(parse_target all "${1:-}")"
if [ "$target" != "all" ]; then
shift || true
fi
while [ $# -gt 0 ]; do
case "$1" in
--foreground)
foreground=true
;;
--mode)
shift || die "--mode 需要指定 dev 或 static"
mode="$1"
validate_frontend_mode "$mode"
;;
*)
die "未知参数: $1"
;;
esac
shift || true
done
case "$target" in
all)
[ "$foreground" = false ] || die "start all 不支持 --foreground，请使用 start api --foreground"
print_header "AI+合规智能中枢 - 启动服务"
start_api background
start_frontend "${mode:-$FRONTEND_MODE}"
;;
api)
if [ "$foreground" = true ]; then
start_api foreground
else
print_header "AI+合规智能中枢 - 启动 API"
start_api background
fi
;;
frontend)
print_header "AI+合规智能中枢 - 启动前端"
start_frontend "${mode:-$FRONTEND_MODE}"
;;
esac
;;
stop)
target="$(parse_target all "${1:-}")"
print_header "AI+合规智能中枢 - 停止服务"
case "$target" in
all)
stop_frontend
stop_api
;;
api)
stop_api
;;
frontend)
stop_frontend
;;
esac
;;
restart)
target="$(parse_target all "${1:-}")"
if [ "$target" != "all" ]; then
shift || true
fi
while [ $# -gt 0 ]; do
case "$1" in
--mode)
shift || die "--mode 需要指定 dev 或 static"
mode="$1"
validate_frontend_mode "$mode"
;;
*)
die "未知参数: $1"
;;
esac
shift || true
done
print_header "AI+合规智能中枢 - 重启服务"
case "$target" in
all)
stop_frontend
stop_api
start_api background
start_frontend "${mode:-$FRONTEND_MODE}"
;;
api)
stop_api
start_api background
;;
frontend)
stop_frontend
start_frontend "${mode:-$FRONTEND_MODE}"
;;
esac
;;
status)
run_status
;;
logs)
log_target="${1:-}"
run_logs "$log_target" "${2:-}"
;;
*)
die "未知命令: $command。可使用 ./dev.sh help 查看帮助。"
;;
esac
}
main "$@"

2
frontend/README.md
View File

@@ -64,7 +64,7 @@ npm run preview
## 项目结构
```
src/
frontend/src/
├── components/
│ ├── common/ # 通用组件 (Logo, Pattern, ThemeToggle)
│ ├── layout/ # 布局组件 (Header, Tabs, Content)

1968
frontend/pnpm-lock.yaml generated Normal file
View File

File diff suppressed because it is too large Load Diff

1
logs/api.pid
View File

@@ -1 +0,0 @@
1894

119
logs/app_2026-05-14.log Normal file
View File

@@ -0,0 +1,119 @@
2026-05-14 16:41:52 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:41:52 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:41:52 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:41:54 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:41:54 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:41:54 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:41:55 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:41:55 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:41:55 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:42:22 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:42:22 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:42:28 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:42:28 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:42:28 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:42:28 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:42:28 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:42:28 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:42:29 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:42:29 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:42:29 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:42:31 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:42:31 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:42:37 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:42:37 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:42:37 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:42:37 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:42:37 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:42:37 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:42:38 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:42:38 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:42:38 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:43:28 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:43:28 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:43:34 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:43:34 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:43:34 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:43:34 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:43:34 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:43:34 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:43:34 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:43:34 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:43:34 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:43:35 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:43:35 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:46:25 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:46:25 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:46:25 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:46:26 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:46:26 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:46:26 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:46:27 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:46:27 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:46:27 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:46:40 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:46:40 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:47:08 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:47:08 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:47:08 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:47:08 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:47:08 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:47:08 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:47:08 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:47:08 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:47:08 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:57:16 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 16:57:16 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 16:57:36 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 16:57:36 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:57:36 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:57:36 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:57:36 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 16:57:36 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 16:57:36 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 16:57:37 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:57:37 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:57:37 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 16:57:37 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 16:57:37 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 16:57:37 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 17:14:37 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 17:14:37 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 17:14:37 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 17:14:37 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 17:14:53 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 17:14:53 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 17:14:53 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 17:14:54 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 17:14:54 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 17:14:54 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 17:14:54 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 17:14:54 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 17:14:54 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 17:16:10 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 17:16:10 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 17:16:21 | INFO | src.api.main:lifespan:27 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 17:16:21 | INFO | src.api.main:lifespan:28 - 调试模式: False
2026-05-14 17:16:21 | INFO | src.api.main:lifespan:31 - 预加载LLM客户端...
2026-05-14 17:16:22 | INFO | src.services.llm.qwen_client:_init_client:59 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 17:16:22 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 17:16:22 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: qwen
2026-05-14 17:16:22 | INFO | src.services.llm.deepseek_client:_init_client:50 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 17:16:22 | INFO | src.services.llm.llm_factory:create:113 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 17:16:22 | SUCCESS | src.services.llm.llm_factory:preload_clients:201 - 预加载LLM客户端成功: deepseek
2026-05-14 17:17:07 | INFO | src.api.main:lifespan:37 - 应用关闭，执行清理...
2026-05-14 17:17:07 | INFO | src.services.llm.llm_factory:cleanup:226 - 所有LLM客户端已清理
2026-05-14 17:19:47 | INFO | app.api.main:lifespan:22 - 启动 AI+合规智能中枢 v0.1.0
2026-05-14 17:19:47 | INFO | app.api.main:lifespan:23 - 调试模式: False
2026-05-14 17:19:47 | INFO | app.api.main:lifespan:24 - 预加载LLM客户端...
2026-05-14 17:19:48 | INFO | app.services.llm.qwen_client:_init_client:58 - Qwen客户端初始化完成: http://6.86.80.4:30080/v1 - qwen3.5-flash
2026-05-14 17:19:48 | INFO | app.services.llm.llm_factory:create:112 - LLM客户端创建成功并缓存: qwen - qwen3.5-flash
2026-05-14 17:19:48 | SUCCESS | app.services.llm.llm_factory:preload_clients:200 - 预加载LLM客户端成功: qwen
2026-05-14 17:19:49 | INFO | app.services.llm.deepseek_client:_init_client:49 - DeepSeek客户端初始化完成: http://6.86.80.4:30080/v1 - deepseek-v4-flash
2026-05-14 17:19:49 | INFO | app.services.llm.llm_factory:create:112 - LLM客户端创建成功并缓存: deepseek - deepseek-v4-flash
2026-05-14 17:19:49 | SUCCESS | app.services.llm.llm_factory:preload_clients:200 - 预加载LLM客户端成功: deepseek

16
pyproject.toml
View File

@@ -26,22 +26,26 @@ dependencies = [
"loguru>=0.7.0",
"tenacity>=8.2.0",
"httpx>=0.24.0",
"celery>=5.3.0",
"redis>=4.5.0",
"minio>=7.1.0",
"psycopg2-binary>=2.9.0"
]
[project.optional-dependencies]
mineru = ["magic-pdf[full]>=0.6.0"]
queue = ["celery>=5.3.0", "redis>=4.5.0"]
storage = ["minio>=7.1.0"]
db = ["psycopg2-binary>=2.9.0"]
dev = ["pytest>=7.0.0", "pytest-asyncio>=0.21.0"]
[dependency-groups]
dev = ["pytest>=7.0.0", "pytest-asyncio>=0.21.0", "isort>=8.0.1"]
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
where = ["src"]
where = ["backend"]
include = ["app*"]
[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
asyncio_mode = "auto"

250
quick_start.sh
View File

@@ -1,250 +0,0 @@
#!/bin/bash
# quick_start.sh - 快速启动脚本
# 适配Docker部署的数据库环境
set -e
echo "========================================"
echo "AI+合规智能中枢 - 快速启动脚本"
echo "========================================"
echo ""
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
VENV_DIR=".venv"
# 检查Python版本
echo -e "${YELLOW}[1/8] 检查Python环境...${NC}"
if command -v python3 &> /dev/null; then
PYTHON_CMD=python3
else
PYTHON_CMD=python
fi
PYTHON_VERSION=$($PYTHON_CMD --version 2>&1 | awk '{print $2}')
echo "Python版本: $PYTHON_VERSION"
if [[ ! "$PYTHON_VERSION" =~ ^3\.1[0-9] ]]; then
echo -e "${RED}需要Python 3.10+，当前版本: $PYTHON_VERSION${NC}"
exit 1
fi
echo -e "${GREEN}Python环境检查通过${NC}"
echo ""
# 创建虚拟环境
echo -e "${YELLOW}[2/8] 创建虚拟环境...${NC}"
if [ -d "$VENV_DIR" ]; then
echo "虚拟环境已存在: $VENV_DIR"
else
echo "正在创建虚拟环境..."
$PYTHON_CMD -m venv $VENV_DIR
echo -e "${GREEN}虚拟环境创建成功${NC}"
fi
# 激活虚拟环境
source $VENV_DIR/bin/activate
echo "已激活虚拟环境: $VENV_DIR"
echo ""
# 安装依赖（使用国内镜像源）
echo -e "${YELLOW}[3/8] 安装Python依赖...${NC}"
# 配置pip使用清华镜像源（国内加速）
PIP_MIRROR="https://mirrors.aliyun.com/pypi/simple"
echo "使用镜像源: $PIP_MIRROR"
pip config set global.index-url $PIP_MIRROR -q
pip install --upgrade pip -q
pip install -r requirements.txt -q
if [ $? -eq 0 ]; then
echo -e "${GREEN}依赖安装完成${NC}"
else
echo -e "${RED}依赖安装失败，请检查requirements.txt${NC}"
exit 1
fi
echo ""
# 检查Docker容器状态
echo -e "${YELLOW}[4/8] 检查Docker容器状态...${NC}"
REQUIRED_CONTAINERS="milvus minio redis postgres"
for container in $REQUIRED_CONTAINERS; do
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
echo -e "${GREEN}✓ ${container} 容器运行正常${NC}"
elif docker ps -a --format '{{.Names}}' | grep -q "^${container}$"; then
echo -e "${YELLOW}⚠ ${container} 容器已停止，尝试启动...${NC}"
docker start ${container}
sleep 2
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
echo -e "${GREEN}✓ ${container} 已启动${NC}"
else
echo -e "${RED}✗ ${container} 启动失败${NC}"
fi
else
echo -e "${RED}✗ ${container} 容器不存在${NC}"
fi
done
echo ""
# 检查PostgreSQL连接（通过Python）
echo -e "${YELLOW}[5/8] 检查PostgreSQL连接...${NC}"
python << 'EOF'
import sys
try:
import psycopg2
conn = psycopg2.connect(
host="localhost",
port=5432,
user="postgresql",
password="postgresql123456",
database="postgres"
)
cur = conn.cursor()
# 检查compliance_db是否存在
cur.execute("SELECT 1 FROM pg_database WHERE datname='compliance_db'")
exists = cur.fetchone()
if not exists:
cur.execute("CREATE DATABASE compliance_db")
print("数据库 compliance_db 创建成功")
else:
print("数据库 compliance_db 已存在")
conn.commit()
cur.close()
conn.close()
print("PostgreSQL连接成功")
except Exception as e:
print(f"PostgreSQL连接失败: {e}")
sys.exit(1)
EOF
if [ $? -eq 0 ]; then
echo -e "${GREEN}PostgreSQL服务运行正常${NC}"
else
echo -e "${RED}PostgreSQL连接失败${NC}"
exit 1
fi
echo ""
# 检查Redis连接（通过Python）
echo -e "${YELLOW}[6/8] 检查Redis连接...${NC}"
python << 'EOF'
import sys
try:
import redis
r = redis.Redis(
host="localhost",
port=6379,
password="redis@123",
decode_responses=True
)
result = r.ping()
if result:
print("Redis连接成功")
else:
sys.exit(1)
except Exception as e:
print(f"Redis连接失败: {e}")
sys.exit(1)
EOF
if [ $? -eq 0 ]; then
echo -e "${GREEN}Redis服务运行正常${NC}"
else
echo -e "${RED}Redis连接失败${NC}"
exit 1
fi
echo ""
# 检查Milvus连接（通过Python）
echo -e "${YELLOW}[7/8] 检查Milvus连接...${NC}"
python << 'EOF'
import sys
try:
from pymilvus import connections, utility
connections.connect(host="localhost", port=19530)
print("Milvus连接成功")
connections.disconnect("default")
except Exception as e:
print(f"Milvus连接失败: {e}")
sys.exit(1)
EOF
if [ $? -eq 0 ]; then
echo -e "${GREEN}Milvus服务运行正常${NC}"
else
echo -e "${RED}Milvus连接失败${NC}"
exit 1
fi
echo ""
# 检查MinIO连接（通过Python）
echo -e "${YELLOW}[8/8] 检查MinIO连接...${NC}"
python << 'EOF'
import sys
try:
from minio import Minio
client = Minio("localhost:9000", "minioadmin", "minioadmin", secure=False)
# 检查bucket是否存在
bucket = "compliance-docs"
if not client.bucket_exists(bucket):
client.make_bucket(bucket)
print(f"MinIO bucket '{bucket}' 创建成功")
else:
print(f"MinIO bucket '{bucket}' 已存在")
print("MinIO连接成功")
except Exception as e:
print(f"MinIO连接失败: {e}")
sys.exit(1)
EOF
if [ $? -eq 0 ]; then
echo -e "${GREEN}MinIO服务运行正常${NC}"
else
echo -e "${RED}MinIO连接失败${NC}"
exit 1
fi
echo ""
# 预下载BGE-M3模型（可选）
echo -e "${YELLOW}[提示] BGE-M3嵌入模型...${NC}"
MODEL_CACHE=~/.cache/huggingface/hub/models--BAAI--bge-m3
if [ -d "$MODEL_CACHE" ]; then
echo -e "${GREEN}BGE-M3模型已存在${NC}"
else
echo -e "${YELLOW}模型未下载，首次运行时将自动下载（约2GB）${NC}"
echo "手动预下载: python -c \"from FlagEmbedding import BGEM3FlagModel; BGEM3FlagModel('BAAI/bge-m3')\""
fi
echo ""
# 输出启动命令
echo "========================================"
echo -e "${GREEN}环境检查完成！${NC}"
echo "========================================"
echo ""
echo "虚拟环境: $VENV_DIR"
echo ""
echo "启动API服务:"
echo " ./start_api.sh"
echo ""
echo "启动前端页面:"
echo " ./start_frontend.sh"
echo " 访问: http://localhost:3000/prototype.html"
echo ""
echo "后台启动:"
echo " ./start_api_background.sh"
echo ""
echo "API文档地址:"
echo " http://localhost:8000/docs"
echo " http://localhost:8000/health"
echo ""

4
requirements.txt
View File

@@ -26,11 +26,11 @@ python-docx>=0.8.11
FlagEmbedding>=1.2.0
sentence-transformers>=2.2.0
# 任务队列（可选）
# 任务队列
celery>=5.3.0
redis>=4.5.0
# 对象存储（可选）
# 对象存储
minio>=7.1.0
# 数据库

23
restart_all.sh
View File

@@ -1,23 +0,0 @@
#!/bin/bash
# restart_all.sh - 重启所有服务
set -e
# 颜色定义
CYAN='\033[0;36m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} AI+合规智能中枢 - 重启服务${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
echo -e "${YELLOW}正在停止所有服务...${NC}"
./stop_all.sh
echo ""
echo -e "${YELLOW}正在启动所有服务...${NC}"
./start_all.sh

11
scripts/clear_all.py
View File

@@ -13,10 +13,11 @@ import argparse
import sys
import os
# 添加项目路径
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# 添加 backend 到导入路径
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(PROJECT_ROOT, "backend"))
from src.config.settings import settings
from app.config.settings import settings
from loguru import logger
@@ -52,7 +53,7 @@ def clear_milvus(dry_run: bool = False):
# 重新创建collection（可选）
logger.info("重新创建collection...")
from src.services.storage.milvus_client import MilvusClient
from app.services.storage.milvus_client import MilvusClient
client = MilvusClient()
client.connect()
client.create_collection(recreate=True)
@@ -190,4 +191,4 @@ def main():
if __name__ == "__main__":
sys.exit(main())
sys.exit(main())

4
src/__init__.py
View File

@@ -1,4 +0,0 @@
# src/__init__.py
"""AI+合规智能中枢 - 法律法规文档解析入库功能"""
__version__ = "0.1.0"

BIN
src/__pycache__/__init__.cpython-312.pyc
View File

Binary file not shown.

2
src/api/__init__.py
View File

@@ -1,2 +0,0 @@
# src/api/__init__.py
"""API接口模块"""

BIN
src/api/__pycache__/__init__.cpython-312.pyc
View File

Binary file not shown.

BIN
src/api/__pycache__/main.cpython-312.pyc
View File

Binary file not shown.

111
src/api/main.py
View File

@@ -1,111 +0,0 @@
# src/api/main.py
"""FastAPI应用入口"""
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from contextlib import asynccontextmanager
from loguru import logger
import sys
import os
# 设置日志
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
from src.config.logging import setup_logging
from src.config.settings import settings
from src.api.routes import api_router
from src.api.models import ErrorResponse
from src.services.llm.llm_factory import LLMFactory
# 配置日志
setup_logging(level="INFO" if not settings.debug else "DEBUG")
@asynccontextmanager
async def lifespan(app: FastAPI):
"""应用生命周期管理"""
logger.info(f"启动 {settings.app_name} v{settings.app_version}")
logger.info(f"调试模式: {settings.debug}")
# 启动时预加载LLM客户端
logger.info("预加载LLM客户端...")
LLMFactory.preload_clients(["qwen", "deepseek"])
yield
# 关闭时清理LLM客户端
logger.info("应用关闭，执行清理...")
LLMFactory.cleanup()
# 创建FastAPI应用
app = FastAPI(
title=settings.app_name,
description="AI+合规智能中枢 - 法律法规文档解析入库功能\n\n支持PDF/DOCX文档解析、智能分块、向量嵌入、Milvus存储",
version=settings.app_version,
lifespan=lifespan,
docs_url="/docs",
redoc_url="/redoc"
)
# CORS中间件
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # 生产环境应配置具体域名
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# 注册路由
app.include_router(api_router, prefix="/api/v1")
# 全局异常处理
@app.exception_handler(Exception)
async def global_exception_handler(request: Request, exc: Exception):
"""全局异常处理"""
logger.error(f"未处理的异常: {exc}")
return JSONResponse(
status_code=500,
content=ErrorResponse(
error="InternalServerError",
message=str(exc)
).model_dump()
)
# 健康检查接口
@app.get("/health", tags=["health"])
async def health_check():
"""健康检查"""
return {
"status": "healthy",
"app": settings.app_name,
"version": settings.app_version
}
# 根路径
@app.get("/", tags=["root"])
async def root():
"""根路径"""
return {
"message": f"Welcome to {settings.app_name}",
"version": settings.app_version,
"docs": "/docs",
"health": "/health"
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(
"main:app",
host=settings.api_host,
port=settings.api_port,
reload=settings.debug,
log_level="info"
)

22
src/api/models/__init__.py
View File

@@ -1,22 +0,0 @@
# src/api/models/__init__.py
"""API数据模型"""
from .document import (
DocumentUploadRequest,
DocumentUploadResponse,
SearchRequest,
SearchResultItem,
SearchResponse,
DocumentStatusResponse,
ErrorResponse
)
__all__ = [
"DocumentUploadRequest",
"DocumentUploadResponse",
"SearchRequest",
"SearchResultItem",
"SearchResponse",
"DocumentStatusResponse",
"ErrorResponse"
]

63
src/api/models/document.py
View File

@@ -1,63 +0,0 @@
# src/api/models/document.py
"""文档相关Pydantic数据模型"""
from pydantic import BaseModel, Field
from typing import List, Optional, Dict, Any
from datetime import datetime
class DocumentUploadRequest(BaseModel):
"""文档上传请求"""
doc_name: Optional[str] = Field(None, description="文档名称")
regulation_type: Optional[str] = Field(None, description="法规类型")
version: Optional[str] = Field(None, description="文档版本")
class DocumentUploadResponse(BaseModel):
"""文档上传响应"""
doc_id: str = Field(..., description="文档ID")
doc_name: str = Field(..., description="文档名称")
status: str = Field(..., description="处理状态")
message: str = Field(default="", description="状态消息")
num_chunks: int = Field(default=0, description="分块数量")
summary: str = Field(default="", description="LLM生成的文档摘要")
summary_latency_ms: int = Field(default=0, description="摘要生成耗时(ms)")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")
class SearchRequest(BaseModel):
"""检索请求"""
query: str = Field(..., description="查询文本")
top_k: int = Field(default=10, description="返回结果数量")
filters: Optional[str] = Field(None, description="过滤条件")
class SearchResultItem(BaseModel):
"""单个检索结果"""
id: int = Field(..., description="记录ID")
content: str = Field(..., description="内容")
score: float = Field(..., description="相似度分数")
metadata: Dict[str, Any] = Field(default_factory=dict, description="元数据")
class SearchResponse(BaseModel):
"""检索响应"""
query: str = Field(..., description="查询文本")
total: int = Field(..., description="结果总数")
results: List[SearchResultItem] = Field(default_factory=list, description="结果列表")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")
class DocumentStatusResponse(BaseModel):
"""文档状态响应"""
doc_id: str = Field(..., description="文档ID")
status: str = Field(..., description="状态")
num_chunks: Optional[int] = Field(None, description="分块数量")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")
class ErrorResponse(BaseModel):
"""错误响应"""
error: str = Field(..., description="错误类型")
message: str = Field(..., description="错误消息")
timestamp: datetime = Field(default_factory=datetime.now, description="时间戳")

17
src/api/routes/__init__.py
View File

@@ -1,17 +0,0 @@
# src/api/routes/__init__.py
"""API路由模块"""
from fastapi import APIRouter
from .documents import router as documents_router
from .knowledge import router as knowledge_router
from .agent import router as agent_router
# 主路由
api_router = APIRouter()
# 注册子路由
api_router.include_router(documents_router)
api_router.include_router(knowledge_router)
api_router.include_router(agent_router)
__all__ = ["api_router", "documents_router", "knowledge_router", "agent_router"]

449
src/api/routes/agent.py
View File

@@ -1,449 +0,0 @@
# src/api/routes/agent.py
"""Agent API接口 - 问答对话接口"""
from fastapi import APIRouter, HTTPException, Depends
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from typing import List, Dict, Optional, AsyncGenerator
from loguru import logger
import json
import asyncio
from src.services.agent.qa_agent import QAAgent, AgentResponse, AgentConfig
from src.services.agent.session_manager import SessionManager, ChatSession
from src.config.settings import settings
router = APIRouter(prefix="/agent", tags=["agent"])
# 会话管理器（全局实例）
session_manager = SessionManager()
# ===== Pydantic Models =====
class AskRequest(BaseModel):
"""单次问答请求"""
query: str = Field(..., description="用户问题", min_length=1, max_length=2000)
filters: Optional[str] = Field(None, description="检索过滤条件")
provider: Optional[str] = Field(None, description="LLM提供商 (qwen/deepseek)")
model: Optional[str] = Field(None, description="LLM模型名称")
top_k: Optional[int] = Field(None, description="检索数量", ge=1, le=20)
prompt_template: Optional[str] = Field(None, description="Prompt模板名称")
class AskResponse(BaseModel):
"""问答响应"""
answer: str
sources: List[Dict] = []
model: str = ""
latency_ms: int = 0
retrieved_count: int = 0
context_tokens: int = 0
truncated: bool = False
error: Optional[str] = None
class ChatRequest(BaseModel):
"""多轮对话请求"""
query: str = Field(..., description="用户问题", min_length=1, max_length=2000)
session_id: Optional[str] = Field(None, description="会话ID（首次对话可不传）")
filters: Optional[str] = Field(None, description="检索过滤条件")
provider: Optional[str] = Field(None, description="LLM提供商")
model: Optional[str] = Field(None, description="LLM模型名称")
class ChatResponse(BaseModel):
"""多轮对话响应"""
session_id: str
answer: str
sources: List[Dict] = []
model: str = ""
latency_ms: int = 0
message_count: int = 0
class SessionInfo(BaseModel):
"""会话信息"""
session_id: str
message_count: int
created_at: int
updated_at: int
class FeedbackRequest(BaseModel):
"""反馈请求"""
session_id: str
message_index: int
rating: int = Field(..., ge=1, le=5, description="评分 1-5")
comment: Optional[str] = Field(None, description="反馈内容")
class TemplateListResponse(BaseModel):
"""模板列表响应"""
templates: Dict[str, str]
# ===== API Endpoints =====
@router.post("/ask", response_model=AskResponse)
async def ask_question(request: AskRequest):
"""
单次问答接口
不保存会话历史，适合单次查询场景。
"""
logger.info(f"收到问答请求: {request.query}")
try:
# 构建Agent配置
config = AgentConfig(
llm_provider=request.provider or settings.llm_provider,
llm_model=request.model or settings.llm_model,
top_k=request.top_k or settings.rag_top_k
)
# 创建Agent并执行问答
agent = QAAgent(config)
response = agent.ask(
query=request.query,
filters=request.filters,
prompt_template=request.prompt_template
)
agent.close()
return AskResponse(
answer=response.answer,
sources=response.sources,
model=response.model,
latency_ms=response.latency_ms,
retrieved_count=response.retrieved_count,
context_tokens=response.context_tokens,
truncated=response.truncated,
error=response.error
)
except Exception as e:
logger.error(f"问答失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@router.post("/chat", response_model=ChatResponse)
async def chat_with_session(request: ChatRequest):
"""
多轮对话接口
支持会话历史记录，适合连续对话场景。
"""
logger.info(f"收到对话请求: session={request.session_id}, query={request.query}")
try:
# 获取或创建会话
if request.session_id:
session = session_manager.get_session(request.session_id)
if not session:
raise HTTPException(status_code=404, detail="会话不存在或已过期")
else:
session = session_manager.create_session()
# 添加用户消息
session.add_user_message(request.query)
# 执行问答
config = AgentConfig(
llm_provider=request.provider or settings.llm_provider,
llm_model=request.model or settings.llm_model
)
agent = QAAgent(config)
response = agent.ask(
query=request.query,
filters=request.filters
)
agent.close()
# 添加助手消息
session.add_assistant_message(
response.answer,
response.sources
)
return ChatResponse(
session_id=session.session_id,
answer=response.answer,
sources=response.sources,
model=response.model,
latency_ms=response.latency_ms,
message_count=session.message_count
)
except HTTPException:
raise
except Exception as e:
logger.error(f"对话失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@router.get("/chat/stream")
async def chat_stream_get(
query: str,
session_id: Optional[str] = None,
filters: Optional[str] = None,
provider: Optional[str] = None,
model: Optional[str] = None
):
"""
流式对话接口（SSE）- GET版本
EventSource只能发送GET请求，因此提供此接口。
query参数通过URL传递。
SSE事件格式：
- event: session - 会话ID
- event: status - 状态更新（检索中、生成中）
- event: sources - 引用来源
- event: content - 回答内容片段
- event: done - 完成，包含统计信息
- event: error - 错误信息
"""
logger.info(f"收到GET流式对话请求: session={session_id}, query={query}")
async def generate_sse() -> AsyncGenerator[str, None]:
"""生成SSE事件流"""
try:
# 获取或创建会话
if session_id:
session = session_manager.get_session(session_id)
if not session:
yield f"event: error\ndata: 会话不存在或已过期\n\n"
return
else:
session = session_manager.create_session()
# 发送session_id
yield f"event: session\ndata: {json.dumps({'session_id': session.session_id})}\n\n"
# 添加用户消息
session.add_user_message(query)
# 创建Agent
config = AgentConfig(
llm_provider=provider or settings.llm_provider,
llm_model=model or settings.llm_model
)
agent = QAAgent(config)
# 执行流式问答
full_answer = ""
sources = []
done_data = {}
for event_data in agent.ask_stream(
query=query,
filters=filters
):
event_type = event_data.get("event", "content")
data = event_data.get("data", "")
# 收集完整回答和来源
if event_type == "content":
full_answer += str(data)
elif event_type == "sources":
sources = data
elif event_type == "done":
done_data = data
# 发送SSE事件
if isinstance(data, (dict, list)):
yield f"event: {event_type}\ndata: {json.dumps(data)}\n\n"
else:
yield f"event: {event_type}\ndata: {data}\n\n"
# 小延迟让其他任务有机会执行
await asyncio.sleep(0)
agent.close()
# 保存到会话历史
session.add_assistant_message(full_answer, sources)
except Exception as e:
logger.error(f"流式对话失败: {e}")
yield f"event: error\ndata: {str(e)}\n\n"
return StreamingResponse(
generate_sse(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no" # 禁用nginx缓冲
}
)
@router.post("/chat/stream")
async def chat_stream(request: ChatRequest):
"""
流式对话接口（SSE）
返回Server-Sent Events格式的流式响应，用户可实时看到思考过程和回答生成。
SSE事件格式：
- event: status - 状态更新（检索中、生成中）
- event: sources - 引用来源
- event: content - 回答内容片段
- event: done - 完成，包含统计信息
- event: error - 错误信息
"""
logger.info(f"收到流式对话请求: session={request.session_id}, query={request.query}")
async def generate_sse() -> AsyncGenerator[str, None]:
"""生成SSE事件流"""
try:
# 获取或创建会话
if request.session_id:
session = session_manager.get_session(request.session_id)
if not session:
yield f"event: error\ndata: 会话不存在或已过期\n\n"
return
else:
session = session_manager.create_session()
# 发送session_id
yield f"event: session\ndata: {json.dumps({'session_id': session.session_id})}\n\n"
# 添加用户消息
session.add_user_message(request.query)
# 创建Agent
config = AgentConfig(
llm_provider=request.provider or settings.llm_provider,
llm_model=request.model or settings.llm_model
)
agent = QAAgent(config)
# 执行流式问答
full_answer = ""
sources = []
done_data = {}
for event_data in agent.ask_stream(
query=request.query,
filters=request.filters
):
event_type = event_data.get("event", "content")
data = event_data.get("data", "")
# 收集完整回答和来源
if event_type == "content":
full_answer += str(data)
elif event_type == "sources":
sources = data
elif event_type == "done":
done_data = data
# 发送SSE事件
if isinstance(data, (dict, list)):
yield f"event: {event_type}\ndata: {json.dumps(data)}\n\n"
else:
yield f"event: {event_type}\ndata: {data}\n\n"
# 小延迟让其他任务有机会执行
await asyncio.sleep(0)
agent.close()
# 保存到会话历史
session.add_assistant_message(full_answer, sources)
except Exception as e:
logger.error(f"流式对话失败: {e}")
yield f"event: error\ndata: {str(e)}\n\n"
return StreamingResponse(
generate_sse(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no" # 禁用nginx缓冲
}
)
@router.get("/session/{session_id}", response_model=SessionInfo)
async def get_session_info(session_id: str):
"""获取会话信息"""
session = session_manager.get_session(session_id)
if not session:
raise HTTPException(status_code=404, detail="会话不存在或已过期")
return SessionInfo(
session_id=session.session_id,
message_count=session.message_count,
created_at=session.created_at,
updated_at=session.updated_at
)
@router.get("/session/{session_id}/history")
async def get_session_history(session_id: str, max_turns: int = 5):
"""获取会话历史"""
session = session_manager.get_session(session_id)
if not session:
raise HTTPException(status_code=404, detail="会话不存在或已过期")
history = session.get_history(max_turns)
return {"session_id": session_id, "history": history}
@router.delete("/session/{session_id}")
async def delete_session(session_id: str):
"""删除会话"""
success = session_manager.delete_session(session_id)
if not success:
raise HTTPException(status_code=404, detail="会话不存在")
return {"message": "会话已删除", "session_id": session_id}
@router.get("/sessions", response_model=List[SessionInfo])
async def list_sessions():
"""列出所有活跃会话"""
sessions = session_manager.list_sessions()
return [SessionInfo(**s) for s in sessions]
@router.post("/feedback")
async def submit_feedback(request: FeedbackRequest):
"""提交问答反馈"""
session = session_manager.get_session(request.session_id)
if not session:
raise HTTPException(status_code=404, detail="会话不存在")
# 记录反馈（实际应用中可存储到数据库）
logger.info(f"收到反馈: session={request.session_id}, rating={request.rating}, comment={request.comment}")
return {"message": "反馈已记录", "rating": request.rating}
@router.get("/templates", response_model=TemplateListResponse)
async def list_prompt_templates():
"""列出可用的Prompt模板"""
from src.services.rag.prompt_templates import PromptTemplates
templates = PromptTemplates.list_templates()
return TemplateListResponse(templates=templates)
@router.get("/models")
async def list_available_models():
"""列出可用的LLM模型"""
from src.services.llm import LLMFactory
factory = LLMFactory()
models = factory.list_available_providers()
return {"models": models}

291
src/api/routes/documents.py
View File

@@ -1,291 +0,0 @@
# src/api/routes/documents.py
"""文档上传与处理接口"""
from fastapi import APIRouter, UploadFile, File, Form, HTTPException
from fastapi.responses import FileResponse, StreamingResponse
from typing import Optional
import os
import uuid
import tempfile
from pathlib import Path
from loguru import logger
from io import BytesIO
from urllib.parse import quote
from ..models import DocumentUploadResponse, ErrorResponse
from src.services.document_processor import DocumentProcessor
from src.services.storage.minio_client import MinIOClient
from src.config.settings import settings
router = APIRouter(prefix="/documents", tags=["documents"])
# MinIO客户端（用于文档存储）
minio_client: Optional[MinIOClient] = None
def get_minio_client() -> MinIOClient:
"""获取MinIO客户端实例"""
global minio_client
if minio_client is None:
minio_client = MinIOClient()
minio_client.connect()
minio_client.ensure_bucket()
return minio_client
def _build_document_records(limit: Optional[int] = None):
"""构建文档列表记录，支持按最近更新时间倒序截断。"""
minio = get_minio_client()
document_records = []
objects = minio.client.list_objects(minio.bucket, recursive=True)
for obj in objects:
parts = obj.object_name.split("/", 1)
if len(parts) != 2:
continue
doc_id, filename = parts
last_modified = getattr(obj, "last_modified", None)
document_records.append({
"doc_id": doc_id,
"filename": filename,
"size": getattr(obj, "size", 0) or 0,
"object_name": obj.object_name,
"download_url": f"/api/v1/documents/download/{doc_id}",
"last_modified": last_modified.isoformat() if last_modified else None,
"_sort_key": last_modified.timestamp() if last_modified else 0,
})
document_records.sort(key=lambda item: item["_sort_key"], reverse=True)
if limit is not None:
document_records = document_records[:limit]
for item in document_records:
item.pop("_sort_key", None)
return document_records
@router.post("/upload", response_model=DocumentUploadResponse)
async def upload_document(
file: UploadFile = File(..., description="上传的文档文件"),
doc_name: Optional[str] = Form(None, description="文档名称"),
regulation_type: Optional[str] = Form(None, description="法规类型"),
version: Optional[str] = Form(None, description="文档版本"),
generate_summary: bool = Form(False, description="是否生成摘要（默认不生成，可节省约60秒）")
):
"""
上传文档并处理
支持格式：PDF、DOCX、DOC
处理流程：解析 → 分块 → 嵌入 → 入库（摘要可选）
文件存储：MinIO对象存储
参数说明：
- generate_summary: 是否生成LLM摘要，默认False。勾选后处理时间增加约60秒。
"""
# 验证文件类型
ext = os.path.splitext(file.filename)[1].lower()
if ext not in [".pdf", ".docx", ".doc"]:
raise HTTPException(
status_code=400,
detail=f"不支持的文件类型: {ext}，仅支持PDF、DOCX、DOC"
)
# 验证文件大小
if file.size and file.size > settings.max_file_size_mb * 1024 * 1024:
raise HTTPException(
status_code=400,
detail=f"文件过大，最大支持{settings.max_file_size_mb}MB"
)
# 生成文档ID
doc_id = str(uuid.uuid4())[:8]
# 文档名称
final_doc_name = doc_name or file.filename
# MinIO对象名称
object_name = f"{doc_id}/{file.filename}"
logger.info(f"接收到文件上传: {final_doc_name}, 类型: {ext}, doc_id={doc_id}")
try:
# 读取文件内容
content = await file.read()
# 保存临时文件用于处理
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, f"{doc_id}_{file.filename}")
with open(temp_path, "wb") as f:
f.write(content)
logger.info(f"临时文件已保存到: {temp_path}")
# 上传到MinIO
minio = get_minio_client()
upload_success = minio.upload_bytes(
data=content,
object_name=object_name,
content_type=minio._get_content_type(file.filename),
metadata={
"doc_id": doc_id # 仅传递ASCII安全的metadata
}
)
if upload_success:
logger.success(f"文件已上传到MinIO: {object_name}")
else:
logger.warning(f"MinIO上传失败，仅使用本地临时文件")
# 处理文档（传入相同的doc_id，保持一致性）
processor = DocumentProcessor(generate_summary=generate_summary)
result = processor.process(
file_path=temp_path,
doc_id=doc_id, # 使用相同的doc_id
doc_name=final_doc_name,
regulation_type=regulation_type or "",
version=version or ""
)
processor.close()
# 清理临时文件
try:
os.remove(temp_path)
except:
pass
if result.success:
return DocumentUploadResponse(
doc_id=result.doc_id,
doc_name=result.doc_name,
status="success",
message=result.message,
num_chunks=result.num_chunks,
summary=result.summary,
summary_latency_ms=result.summary_latency_ms
)
else:
raise HTTPException(
status_code=500,
detail=result.message
)
except Exception as e:
logger.error(f"文档处理失败: {e}")
raise HTTPException(
status_code=500,
detail=f"文档处理失败: {str(e)}"
)
@router.get("/status/{doc_id}", response_model=DocumentUploadResponse)
async def get_document_status(doc_id: str):
"""
查询文档处理状态
Args:
doc_id: 文档ID
"""
# TODO: 实现状态查询（需要数据库支持）
return DocumentUploadResponse(
doc_id=doc_id,
doc_name="",
status="unknown",
message="状态查询功能待实现"
)
@router.get("/download/{doc_id}")
async def download_document(doc_id: str):
"""
下载文档（从MinIO获取）
Args:
doc_id: 文档ID
Returns:
文件下载响应
"""
logger.info(f"请求下载文档: doc_id={doc_id}")
try:
minio = get_minio_client()
# 查找该doc_id下的文件（MinIO对象名称格式: {doc_id}/{filename}）
objects = minio.list_objects(prefix=f"{doc_id}/")
if not objects:
logger.warning(f"MinIO中未找到文档: doc_id={doc_id}")
raise HTTPException(
status_code=404,
detail=f"文档不存在: doc_id={doc_id}"
)
# 获取第一个匹配的对象
object_name = objects[0]
logger.info(f"找到MinIO对象: {object_name}")
# 获取文件数据
file_data = minio.get_object_data(object_name)
if file_data is None:
raise HTTPException(
status_code=500,
detail=f"获取文档数据失败"
)
# 解析原始文件名
original_name = object_name.split("/", 1)[1] if "/" in object_name else object_name
# 获取Content-Type
content_type = minio._get_content_type(original_name)
logger.success(f"文档下载成功: {original_name}, 大小={len(file_data)}")
# 返回文件流（URL编码文件名以支持中文）
encoded_name = quote(original_name)
return StreamingResponse(
BytesIO(file_data),
media_type=content_type,
headers={
"Content-Disposition": f"attachment; filename*=UTF-8''{encoded_name}"
}
)
except HTTPException:
raise
except Exception as e:
logger.error(f"文档下载失败: {e}")
raise HTTPException(
status_code=500,
detail=f"文档下载失败: {str(e)}"
)
@router.get("/list")
async def list_documents():
"""
列出所有已上传的文档（从MinIO获取）
"""
try:
documents = _build_document_records()
return {"documents": documents, "total": len(documents)}
except Exception as e:
logger.error(f"列出文档失败: {e}")
return {"documents": [], "total": 0, "error": str(e)}
@router.get("/management-list")
async def get_document_management_list():
"""
文档管理清单接口：仅返回最近的10条文档。
"""
try:
documents = _build_document_records(limit=10)
return {"documents": documents, "total": len(documents), "limit": 10}
except Exception as e:
logger.error(f"获取文档管理清单失败: {e}")
return {"documents": [], "total": 0, "limit": 10, "error": str(e)}

81
src/api/routes/knowledge.py
View File

@@ -1,81 +0,0 @@
# src/api/routes/knowledge.py
"""知识库检索接口"""
from fastapi import APIRouter, HTTPException
from loguru import logger
from ..models import SearchRequest, SearchResponse, SearchResultItem, ErrorResponse
from src.services.document_processor import DocumentProcessor
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
@router.post("/search", response_model=SearchResponse)
async def search_knowledge(request: SearchRequest):
"""
检索法规知识库
使用混合检索：Dense向量 + Sparse向量 + RRF融合
Args:
request: 检索请求参数
"""
if not request.query or len(request.query.strip()) == 0:
raise HTTPException(
status_code=400,
detail="查询文本不能为空"
)
logger.info(f"收到检索请求: {request.query}")
try:
# 执行检索
processor = DocumentProcessor()
results = processor.search(
query=request.query,
top_k=request.top_k,
filters=request.filters
)
processor.close()
# 转换结果格式
result_items = []
for r in results:
item = SearchResultItem(
id=r.get("id", 0),
content=r.get("content", ""),
score=r.get("score", 0.0),
metadata=r.get("metadata", {})
)
result_items.append(item)
return SearchResponse(
query=request.query,
total=len(result_items),
results=result_items
)
except Exception as e:
logger.error(f"检索失败: {e}")
raise HTTPException(
status_code=500,
detail=f"检索失败: {str(e)}"
)
@router.post("/retrieval", response_model=SearchResponse)
async def knowledge_retrieval(request: SearchRequest):
"""
知识检索接口（与架构文档对齐）
该接口实现完整的检索流程：
1. 意图识别
2. BM25关键词检索 + 向量语义检索（双路召回）
3. Cross-Encoder精排
4. 返回结果
Args:
request: 检索请求
"""
# 当前版本使用混合检索，后续可添加精排步骤
return await search_knowledge(request)

6
src/config/__init__.py
View File

@@ -1,6 +0,0 @@
# src/config/__init__.py
"""配置模块"""
from .settings import Settings, get_settings
__all__ = ["Settings", "get_settings"]

32
src/config/logging.py
View File

@@ -1,32 +0,0 @@
# src/config/logging.py
"""日志配置"""
from loguru import logger
import sys
def setup_logging(level: str = "INFO"):
"""设置日志配置"""
# 移除默认handler
logger.remove()
# 添加控制台输出
logger.add(
sys.stdout,
level=level,
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>",
colorize=True
)
# 添加文件输出
logger.add(
"logs/app_{time:YYYY-MM-DD}.log",
level=level,
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}",
rotation="00:00",
retention="7 days",
compression="zip"
)
return logger

95
src/config/settings.py
View File

@@ -1,95 +0,0 @@
# src/config/settings.py
"""配置管理 - 环境变量和默认配置"""
from pydantic_settings import BaseSettings
from pydantic import Field
from typing import Optional
from functools import lru_cache
class Settings(BaseSettings):
"""应用配置"""
# 应用基础配置
app_name: str = Field(default="AI Regulations Demo", description="Application name")
app_version: str = Field(default="0.1.0", description="应用版本")
debug: bool = Field(default=False, description="调试模式")
# Milvus向量数据库配置
milvus_host: str = Field(default="localhost", description="Milvus服务地址")
milvus_port: int = Field(default=19530, description="Milvus服务端口")
milvus_collection: str = Field(default="regulations", description="法规向量集合名称")
milvus_db_name: str = Field(default="default", description="Milvus数据库名称")
# 嵌入模型配置
embedding_model: str = Field(default="BAAI/bge-m3", description="嵌入模型名称")
embedding_dim: int = Field(default=1024, description="嵌入向量维度")
embedding_max_length: int = Field(default=8192, description="最大嵌入长度")
embedding_batch_size: int = Field(default=12, description="嵌入批处理大小")
embedding_use_fp16: bool = Field(default=True, description="使用FP16加速")
# MinIO对象存储配置
minio_endpoint: str = Field(default="localhost:9000", description="MinIO服务地址")
minio_access_key: str = Field(default="minioadmin", description="MinIO访问密钥")
minio_secret_key: str = Field(default="minioadmin123", description="MinIO秘密密钥")
minio_bucket: str = Field(default="upload-files", description="文档存储桶名称")
minio_secure: bool = Field(default=False, description="是否使用HTTPS")
# Redis配置
redis_host: str = Field(default="localhost", description="Redis服务地址")
redis_port: int = Field(default=6379, description="Redis服务端口")
redis_password: str = Field(default="", description="Redis密码")
redis_db: int = Field(default=0, description="Redis数据库编号")
# PostgreSQL配置
postgres_host: str = Field(default="localhost", description="PostgreSQL服务地址")
postgres_port: int = Field(default=5432, description="PostgreSQL服务端口")
postgres_user: str = Field(default="compliance", description="PostgreSQL用户名")
postgres_password: str = Field(default="compliance123", description="PostgreSQL密码")
postgres_db: str = Field(default="compliance_db", description="PostgreSQL数据库名称")
# 文档处理配置
chunk_size: int = Field(default=512, description="分块大小（字符数）")
chunk_overlap: int = Field(default=50, description="分块重叠大小")
max_file_size_mb: int = Field(default=100, description="最大文件大小(MB)")
# API配置
api_host: str = Field(default="0.0.0.0", description="API服务地址")
api_port: int = Field(default=8000, description="API服务端口")
# LLM配置
llm_provider: str = Field(default="deepseek", description="LLM提供商 (deepseek/qwen/qwen_vl)")
llm_model: str = Field(default="deepseek-v4-flash", description="LLM模型名称")
llm_max_tokens: int = Field(default=4096, description="LLM最大输出token数")
llm_temperature: float = Field(default=0.7, description="LLM温度参数")
# DeepSeek配置
deepseek_api_key: str = Field(default="", description="DeepSeek API密钥")
deepseek_base_url: str = Field(default="http://6.86.80.4:30080/v1", description="DeepSeek API地址")
deepseek_model: str = Field(default="deepseek-v4-flash", description="DeepSeek模型")
# Qwen配置（通过统一代理API）
qwen_api_key: str = Field(default="", description="Qwen API密钥")
qwen_base_url: str = Field(default="http://6.86.80.4:30080/v1", description="Qwen API地址")
qwen_model: str = Field(default="qwen3.5-flash", description="Qwen文本模型")
qwen_vl_model: str = Field(default="qwen3-vl-plus", description="Qwen视觉模型")
# RAG配置
rag_top_k: int = Field(default=5, description="检索召回数量")
rag_max_context_tokens: int = Field(default=2000, description="RAG最大上下文token数")
rag_summary_max_tokens: int = Field(default=10240, description="文档摘要最大token数")
class Config:
env_file = ".env"
env_file_encoding = "utf-8"
extra = "ignore"
@lru_cache
def get_settings() -> Settings:
"""获取配置实例（缓存）"""
return Settings()
# 导出默认配置实例
settings = get_settings()

2
src/services/__init__.py
View File

@@ -1,2 +0,0 @@
# src/services/__init__.py
"""业务服务模块"""

7
src/services/agent/__init__.py
View File

@@ -1,7 +0,0 @@
# src/services/agent/__init__.py
"""Agent服务模块"""
from .qa_agent import QAAgent, ask_compliance_question
from .session_manager import SessionManager, ChatSession
__all__ = ["QAAgent", "ask_compliance_question", "SessionManager", "ChatSession"]

412
src/services/agent/qa_agent.py
View File

@@ -1,412 +0,0 @@
# src/services/agent/qa_agent.py
"""RAG问答Agent - 合规智能问答核心实现"""
import time
from typing import List, Dict, Optional, Any, Generator
from dataclasses import dataclass, field
from loguru import logger
from src.services.llm import get_llm_client, BaseLLMClient, LLMResponse
from src.services.llm.llm_factory import LLMFactory
from src.services.rag.retriever import Retriever, RetrievedDocument
from src.services.rag.context_builder import ContextBuilder, RAGContext
from src.services.rag.prompt_templates import get_prompt_template, PromptTemplate
from src.config.settings import settings
@dataclass
class AgentResponse:
"""Agent响应结果"""
answer: str
sources: List[Dict] = field(default_factory=list)
model: str = ""
latency_ms: int = 0
retrieved_count: int = 0
context_tokens: int = 0
truncated: bool = False
error: Optional[str] = None
@property
def is_success(self) -> bool:
return self.error is None
@dataclass
class AgentConfig:
"""Agent配置"""
llm_provider: str = "deepseek"
llm_model: str = "deepseek-v4-flash"
top_k: int = 5
min_score: float = 0.3
max_context_tokens: int = 2000
temperature: float = 0.7
prompt_template: str = "compliance_qa"
include_metadata: bool = True
class QAAgent:
"""
合规问答Agent
核心流程：
1. 接收用户问题
2. Milvus混合检索相关法规条款
3. 构建RAG上下文
4. 调用LLM生成回答
5. 返回答案和引用来源
使用示例：
agent = QAAgent()
response = agent.ask("机动车安全技术检验有哪些要求？")
print(response.answer)
for source in response.sources:
print(f"引用: {source['doc_name']} - {source['clause_number']}")
"""
def __init__(self, config: Optional[AgentConfig] = None):
"""
初始化问答Agent
Args:
config: Agent配置（可选，使用默认配置）
"""
self.config = config or AgentConfig(
llm_provider=settings.llm_provider,
llm_model=settings.llm_model,
top_k=settings.rag_top_k,
max_context_tokens=settings.rag_max_context_tokens
)
# 初始化组件（延迟加载）
self.llm: Optional[BaseLLMClient] = None
self.retriever: Optional[Retriever] = None
self.context_builder: Optional[ContextBuilder] = None
logger.info(f"问答Agent初始化: provider={self.config.llm_provider}, model={self.config.llm_model}")
def _init_llm(self):
"""延迟初始化LLM客户端（优先使用全局缓存）"""
if self.llm is None:
# 尝试先获取全局缓存的客户端
cached = LLMFactory.get_global_client(self.config.llm_provider, self.config.llm_model)
if cached:
self.llm = cached
logger.debug(f"使用全局缓存的LLM客户端: {self.config.llm_provider} - {self.config.llm_model}")
else:
logger.info("创建新的LLM客户端...")
self.llm = get_llm_client(
provider=self.config.llm_provider,
model=self.config.llm_model,
temperature=self.config.temperature
)
def _init_retriever(self):
"""延迟初始化检索器"""
if self.retriever is None:
logger.info("初始化检索器...")
self.retriever = Retriever(
top_k=self.config.top_k,
min_score=self.config.min_score
)
def _init_context_builder(self):
"""延迟初始化上下文构建器"""
if self.context_builder is None:
logger.info("初始化上下文构建器...")
self.context_builder = ContextBuilder(
max_context_tokens=self.config.max_context_tokens,
include_metadata=self.config.include_metadata
)
def ask(
self,
query: str,
filters: Optional[str] = None,
prompt_template: Optional[str] = None
) -> AgentResponse:
"""
回答用户问题
Args:
query: 用户问题
filters: 检索过滤条件（如 "regulation_type=='车辆安全'"）
prompt_template: Prompt模板名称（可选，覆盖默认配置）
Returns:
AgentResponse: 包含答案和引用来源的响应对象
"""
start_time = time.time()
logger.info(f"收到问题: {query}")
try:
# Step 1: 检索相关法规
self._init_retriever()
documents = self.retriever.retrieve(query, filters)
retrieved_count = len(documents)
if retrieved_count == 0:
return AgentResponse(
answer="抱歉，未找到与您的问题相关的法规条款。请尝试用不同的关键词重新提问，或提供更具体的法规名称。",
retrieved_count=0,
error="no_retrieved_documents"
)
# Step 2: 构建RAG上下文
self._init_context_builder()
template_name = prompt_template or self.config.prompt_template
template = get_prompt_template(template_name)
context = self.context_builder.build(
query=query,
documents=documents,
system_prompt=template.system_prompt
)
# Step 3: 构建LLM输入消息
messages = self._build_messages(template, context)
# Step 4: 调用LLM生成回答
self._init_llm()
llm_response = self.llm.chat(
messages=messages,
temperature=self.config.temperature
)
if not llm_response.is_success:
return AgentResponse(
answer="",
retrieved_count=retrieved_count,
error=llm_response.error
)
latency_ms = int((time.time() - start_time) * 1000)
# Step 5: 返回结果
logger.success(f"问答完成: {latency_ms}ms, {retrieved_count}条引用")
return AgentResponse(
answer=llm_response.content,
sources=context.sources,
model=llm_response.model,
latency_ms=latency_ms,
retrieved_count=retrieved_count,
context_tokens=context.total_tokens,
truncated=context.truncated
)
except Exception as e:
logger.error(f"问答失败: {e}")
return AgentResponse(
answer="",
error=str(e)
)
def ask_with_context(
self,
query: str,
documents: List[RetrievedDocument],
prompt_template: Optional[str] = None
) -> AgentResponse:
"""
使用提供的文档回答问题（不执行检索）
Args:
query: 用户问题
documents: 已检索的文档列表
prompt_template: Prompt模板名称
Returns:
AgentResponse: 响应结果
"""
start_time = time.time()
try:
self._init_context_builder()
self._init_llm()
template_name = prompt_template or self.config.prompt_template
template = get_prompt_template(template_name)
context = self.context_builder.build(
query=query,
documents=documents,
system_prompt=template.system_prompt
)
messages = self._build_messages(template, context)
llm_response = self.llm.chat(messages)
latency_ms = int((time.time() - start_time) * 1000)
return AgentResponse(
answer=llm_response.content,
sources=context.sources,
model=llm_response.model,
latency_ms=latency_ms,
retrieved_count=len(documents),
context_tokens=context.total_tokens,
truncated=context.truncated
)
except Exception as e:
logger.error(f"问答失败: {e}")
return AgentResponse(answer="", error=str(e))
def _build_messages(
self,
template: PromptTemplate,
context: RAGContext
) -> List[Dict[str, str]]:
"""构建LLM输入消息"""
user_content = template.user_template.format(
context=context.context_text,
query=context.user_query
)
return [
{"role": "system", "content": template.system_prompt},
{"role": "user", "content": user_content}
]
def ask_stream(
self,
query: str,
filters: Optional[str] = None,
prompt_template: Optional[str] = None
) -> Generator[Dict[str, Any], None, None]:
"""
流式回答用户问题（SSE模式）
返回事件类型:
- {"event": "status", "data": "正在检索..."} - 状态更新
- {"event": "sources", "data": [...]} - 引用来源
- {"event": "content", "data": "文本片段"} - 回答内容
- {"event": "done", "data": {"latency_ms": ..., "model": ...}} - 完成
Args:
query: 用户问题
filters: 检索过滤条件
prompt_template: Prompt模板名称
Yields:
Dict: SSE事件数据
"""
start_time = time.time()
logger.info(f"收到流式问题: {query}")
try:
# Step 1: 检索相关法规
yield {"event": "status", "data": "正在检索相关法规..."}
self._init_retriever()
documents = self.retriever.retrieve(query, filters)
retrieved_count = len(documents)
if retrieved_count == 0:
yield {"event": "status", "data": "未找到相关法规"}
yield {"event": "content", "data": "抱歉，未找到与您的问题相关的法规条款。请尝试用不同的关键词重新提问。"}
yield {"event": "done", "data": {"latency_ms": 0, "retrieved_count": 0}}
return
# Step 2: 发送检索结果
yield {"event": "status", "data": f"找到{retrieved_count}条相关法规，正在生成回答..."}
sources = [
{
"doc_name": doc.doc_name,
"doc_id": doc.doc_id,
"clause_number": doc.clause_number,
"score": doc.score
}
for doc in documents[:5] # 只返回前5条引用
]
yield {"event": "sources", "data": sources}
# Step 3: 构建RAG上下文
self._init_context_builder()
template_name = prompt_template or self.config.prompt_template
template = get_prompt_template(template_name)
context = self.context_builder.build(
query=query,
documents=documents,
system_prompt=template.system_prompt
)
# Step 4: 构建LLM输入消息
messages = self._build_messages(template, context)
# Step 5: 流式调用LLM生成回答
self._init_llm()
full_answer = ""
# 检查LLM是否支持流式输出
if hasattr(self.llm, 'stream_chat'):
yield {"event": "status", "data": "思考中..."}
for chunk in self.llm.stream_chat(
messages=messages,
temperature=self.config.temperature
):
full_answer += chunk
yield {"event": "content", "data": chunk}
else:
# 如果不支持流式，回退到普通调用
yield {"event": "status", "data": "生成回答中..."}
llm_response = self.llm.chat(
messages=messages,
temperature=self.config.temperature
)
if llm_response.is_success:
full_answer = llm_response.content
yield {"event": "content", "data": full_answer}
# Step 6: 发送完成事件
latency_ms = int((time.time() - start_time) * 1000)
logger.success(f"流式问答完成: {latency_ms}ms, {retrieved_count}条引用")
yield {
"event": "done",
"data": {
"latency_ms": latency_ms,
"model": self.config.llm_model,
"retrieved_count": retrieved_count,
"context_tokens": context.total_tokens
}
}
except Exception as e:
logger.error(f"流式问答失败: {e}")
yield {"event": "error", "data": str(e)}
def close(self):
"""关闭Agent资源（不关闭LLM客户端，因为它全局缓存）"""
if self.retriever:
self.retriever.close()
logger.info("问答Agent已关闭")
def ask_compliance_question(
query: str,
provider: str = "deepseek",
model: str = "deepseek-v4-flash",
top_k: int = 10
) -> AgentResponse:
"""
便捷函数：问答合规问题
Args:
query: 用户问题
provider: LLM提供商
model: LLM模型
top_k: 检索数量
Returns:
AgentResponse: 响应结果
"""
config = AgentConfig(
llm_provider=provider,
llm_model=model,
top_k=top_k
)
agent = QAAgent(config)
response = agent.ask(query)
agent.close()
return response

247
src/services/agent/session_manager.py
View File

@@ -1,247 +0,0 @@
# src/services/agent/session_manager.py
"""多轮对话会话管理"""
import time
import uuid
from typing import Dict, List, Optional
from dataclasses import dataclass, field
from loguru import logger
@dataclass
class ChatMessage:
"""对话消息"""
role: str # "user" / "assistant" / "system"
content: str
timestamp: int
sources: List[Dict] = field(default_factory=list)
metadata: Dict = field(default_factory=dict)
@dataclass
class ChatSession:
"""对话会话"""
session_id: str
messages: List[ChatMessage] = field(default_factory=list)
created_at: int = field(default_factory=lambda: int(time.time()))
updated_at: int = field(default_factory=lambda: int(time.time()))
metadata: Dict = field(default_factory=dict)
def add_user_message(self, content: str) -> ChatMessage:
"""添加用户消息"""
message = ChatMessage(
role="user",
content=content,
timestamp=int(time.time())
)
self.messages.append(message)
self.updated_at = int(time.time())
return message
def add_assistant_message(
self,
content: str,
sources: List[Dict] = None
) -> ChatMessage:
"""添加助手消息"""
message = ChatMessage(
role="assistant",
content=content,
timestamp=int(time.time()),
sources=sources or []
)
self.messages.append(message)
self.updated_at = int(time.time())
return message
def get_history(self, max_turns: int = 5) -> List[Dict[str, str]]:
"""获取历史对话（用于LLM上下文）"""
history = []
# 获取最近N轮对话（每轮包含user + assistant）
recent_messages = self.messages[-(max_turns * 2):]
for msg in recent_messages:
history.append({
"role": msg.role,
"content": msg.content
})
return history
def clear_history(self):
"""清空对话历史"""
self.messages = []
self.updated_at = int(time.time())
logger.info(f"会话历史已清空: {self.session_id}")
@property
def message_count(self) -> int:
"""消息数量"""
return len(self.messages)
@property
def is_empty(self) -> bool:
"""是否为空会话"""
return len(self.messages) == 0
class SessionManager:
"""
会话管理器
功能：
- 创建/获取/删除会话
- 会话超时清理
- 会话历史记录管理
使用示例：
manager = SessionManager()
# 创建会话
session = manager.create_session()
# 添加消息
session.add_user_message("什么是机动车安全技术检验？")
session.add_assistant_message("根据GB 7258...", sources=[...])
# 获取历史（用于LLM多轮对话）
history = session.get_history(max_turns=3)
"""
def __init__(
self,
max_sessions: int = 100,
session_timeout_minutes: int = 30
):
"""
初始化会话管理器
Args:
max_sessions: 最大会话数量
session_timeout_minutes: 会话超时时间（分钟）
"""
self.max_sessions = max_sessions
self.session_timeout = session_timeout_minutes * 60
# 会话存储（内存）
self._sessions: Dict[str, ChatSession] = {}
logger.info(f"会话管理器初始化: max_sessions={max_sessions}, timeout={session_timeout_minutes}min")
def create_session(self, metadata: Dict = None) -> ChatSession:
"""
创建新会话
Args:
metadata: 会话元数据（可选）
Returns:
ChatSession: 新创建的会话
"""
# 检查会话数量限制
if len(self._sessions) >= self.max_sessions:
# 清理过期会话
self._cleanup_expired_sessions()
# 如果仍然超出限制，删除最老的会话
if len(self._sessions) >= self.max_sessions:
oldest_id = min(
self._sessions.keys(),
key=lambda x: self._sessions[x].created_at
)
self.delete_session(oldest_id)
logger.warning(f"删除最老会话以腾出空间: {oldest_id}")
session_id = str(uuid.uuid4())[:8]
session = ChatSession(
session_id=session_id,
metadata=metadata or {}
)
self._sessions[session_id] = session
logger.info(f"创建新会话: {session_id}")
return session
def get_session(self, session_id: str) -> Optional[ChatSession]:
"""
获取会话
Args:
session_id: 会话ID
Returns:
ChatSession: 会话对象（如不存在返回None）
"""
session = self._sessions.get(session_id)
if session:
# 检查是否过期
if self._is_session_expired(session):
self.delete_session(session_id)
logger.info(f"会话已过期，已删除: {session_id}")
return None
return session
def delete_session(self, session_id: str) -> bool:
"""
删除会话
Args:
session_id: 会话ID
Returns:
bool: 是否成功删除
"""
if session_id in self._sessions:
del self._sessions[session_id]
logger.info(f"删除会话: {session_id}")
return True
return False
def list_sessions(self) -> List[Dict]:
"""
列出所有会话
Returns:
List[Dict]: 会话列表摘要
"""
return [
{
"session_id": s.session_id,
"message_count": s.message_count,
"created_at": s.created_at,
"updated_at": s.updated_at
}
for s in self._sessions.values()
]
def _is_session_expired(self, session: ChatSession) -> bool:
"""检查会话是否过期"""
current_time = int(time.time())
return (current_time - session.updated_at) > self.session_timeout
def _cleanup_expired_sessions(self) -> int:
"""清理过期会话"""
expired_ids = [
sid for sid, session in self._sessions.items()
if self._is_session_expired(session)
]
for sid in expired_ids:
self.delete_session(sid)
if expired_ids:
logger.info(f"清理过期会话: {len(expired_ids)}个")
return len(expired_ids)
def get_session_count(self) -> int:
"""获取当前会话数量"""
return len(self._sessions)
def clear_all_sessions(self):
"""清空所有会话"""
self._sessions.clear()
logger.info("所有会话已清空")

404
src/services/document_processor.py
View File

@@ -1,404 +0,0 @@
# src/services/document_processor.py
"""文档处理主流程 - 解析→摘要→分块→嵌入→入库"""
import os
from typing import List, Dict, Optional
from dataclasses import dataclass
from loguru import logger
import uuid
from .parser.pdf_parser import PDFParser
from .parser.docx_parser import DocxParser
from .parser.mineru_parser import ParserOrchestrator
from .embedding.text_chunker import RegulationChunker, TextChunk
from .embedding.bge_m3_embedder import BGEM3Embedder, EmbeddingResult
from .storage.milvus_client import MilvusClient
from .llm.document_summarizer import DocumentSummarizer, DocumentSummary
from src.config.settings import settings
@dataclass
class ProcessingResult:
"""文档处理结果"""
doc_id: str
doc_name: str
success: bool
num_chunks: int = 0
message: str = ""
markdown_text: str = ""
summary: str = ""
summary_latency_ms: int = 0
class DocumentProcessor:
"""
文档处理服务 - 完整处理流程
流程：
1. 文档解析（PDF/DOCX → Markdown）
2. 智能分块（章节级+条款级）
3. LLM摘要生成（可选）
4. 向量嵌入（BGE-M3 Dense+Sparse）
5. 存储入库（Milvus向量数据库）
"""
def __init__(
self,
chunk_size: int = None,
embedding_model: str = None,
use_mineru: bool = True,
generate_summary: bool = False, # 默认不生成摘要，节省约60秒
llm_provider: str = None,
llm_model: str = None
):
"""
初始化文档处理器
Args:
chunk_size: 分块大小
embedding_model: 嵌入模型名称
use_mineru: 是否优先使用MinerU解析
generate_summary: 是否生成文档摘要（默认False，可节省约60秒处理时间）
llm_provider: LLM提供商
llm_model: LLM模型名称
"""
self.chunk_size = chunk_size or settings.chunk_size
self.embedding_model = embedding_model or settings.embedding_model
self.use_mineru = use_mineru
self.generate_summary = generate_summary
self.llm_provider = llm_provider or settings.llm_provider
self.llm_model = llm_model or settings.llm_model
# 初始化各组件
logger.info("初始化文档处理组件...")
# 解析器
self.parser = ParserOrchestrator()
# 分块器
self.chunker = RegulationChunker(chunk_size=self.chunk_size)
# 嵌入模型（延迟加载）
self.embedder: Optional[BGEM3Embedder] = None
# Milvus客户端（延迟连接）
self.milvus: Optional[MilvusClient] = None
# 摘要生成器（延迟加载）
self.summarizer: Optional[DocumentSummarizer] = None
logger.success("文档处理器初始化完成")
def _init_embedder(self):
"""延迟初始化嵌入模型"""
if self.embedder is None:
logger.info("加载嵌入模型...")
self.embedder = BGEM3Embedder(model_name=self.embedding_model)
def _init_milvus(self):
"""延迟初始化Milvus连接"""
if self.milvus is None:
logger.info("连接Milvus...")
self.milvus = MilvusClient()
self.milvus.connect()
self.milvus.create_collection(recreate=False)
self.milvus.load_collection()
def _init_summarizer(self):
"""延迟初始化摘要生成器"""
if self.summarizer is None:
logger.info("初始化摘要生成器...")
self.summarizer = DocumentSummarizer(
provider=self.llm_provider,
model=self.llm_model
)
def process(
self,
file_path: str,
doc_id: Optional[str] = None,
doc_name: Optional[str] = None,
regulation_type: str = "",
version: str = ""
) -> ProcessingResult:
"""
处理单个文档
Args:
file_path: 文档文件路径
doc_id: 文档ID（可选，默认自动生成）
doc_name: 文档名称（可选，默认从文件名获取）
regulation_type: 法规类型
version: 文档版本
Returns:
ProcessingResult: 处理结果
"""
# 生成或使用传入的文档ID
if doc_id is None:
doc_id = str(uuid.uuid4())[:8]
# 获取文档名称
if doc_name is None:
doc_name = os.path.basename(file_path)
logger.info(f"开始处理文档: {doc_name} (ID: {doc_id})")
# 初始化结果变量
summary = ""
summary_latency_ms = 0
try:
# 1. 文档解析
logger.info("Step 1: 文档解析")
markdown_text = self._parse_document(file_path)
if not markdown_text:
return ProcessingResult(
doc_id=doc_id,
doc_name=doc_name,
success=False,
message="文档解析失败，内容为空"
)
# 2. LLM摘要生成（可选）
if self.generate_summary:
logger.info("Step 2: LLM摘要生成")
self._init_summarizer()
summary_result = self.summarizer.summarize(
doc_name,
markdown_text,
regulation_type
)
if summary_result.is_success:
summary = summary_result.summary
summary_latency_ms = summary_result.latency_ms
logger.success(f"摘要生成完成: {summary_latency_ms}ms")
else:
logger.warning(f"摘要生成失败: {summary_result.error}")
else:
logger.info("Step 2: 跳过摘要生成（未勾选，节省约60秒）")
# 3. 智能分块
logger.info("Step 3: 智能分块")
chunks = self._chunk_document(
markdown_text,
doc_id,
doc_name,
regulation_type,
version
)
if not chunks:
return ProcessingResult(
doc_id=doc_id,
doc_name=doc_name,
success=False,
message="分块失败，无有效内容",
markdown_text=markdown_text,
summary=summary
)
# 4. 向量嵌入
logger.info("Step 4: 向量嵌入")
embeddings = self._embed_chunks(chunks)
if embeddings is None:
return ProcessingResult(
doc_id=doc_id,
doc_name=doc_name,
success=False,
message="向量嵌入失败",
markdown_text=markdown_text,
summary=summary
)
# 5. 存储入库
logger.info("Step 5: 存储入库")
inserted_ids = self._insert_to_milvus(chunks, embeddings)
logger.success(f"文档处理完成: {doc_name}, 共{len(inserted_ids)}条记录")
return ProcessingResult(
doc_id=doc_id,
doc_name=doc_name,
success=True,
num_chunks=len(inserted_ids),
message="处理成功",
markdown_text=markdown_text,
summary=summary,
summary_latency_ms=summary_latency_ms
)
except Exception as e:
logger.error(f"文档处理失败: {e}")
return ProcessingResult(
doc_id=doc_id,
doc_name=doc_name,
success=False,
message=f"处理失败: {str(e)}"
)
def _parse_document(self, file_path: str) -> str:
"""解析文档"""
ext = os.path.splitext(file_path)[1].lower()
try:
if ext == ".pdf":
# PDF文档解析（优先MinerU，回退PyMuPDF）
markdown_text = self.parser.parse_pdf(file_path, prefer_mineru=self.use_mineru)
elif ext in [".docx", ".doc"]:
# Word文档解析
markdown_text = self.parser.parse_docx(file_path)
else:
logger.warning(f"不支持的文件类型: {ext}")
return ""
logger.success(f"文档解析完成，内容长度: {len(markdown_text)}字符")
return markdown_text
except Exception as e:
logger.error(f"文档解析失败: {e}")
return ""
def _chunk_document(
self,
markdown_text: str,
doc_id: str,
doc_name: str,
regulation_type: str,
version: str
) -> List[TextChunk]:
"""分块文档"""
try:
chunks = self.chunker.chunk_document(
markdown_text,
doc_id=doc_id,
doc_name=doc_name,
regulation_type=regulation_type,
version=version
)
logger.success(f"分块完成，共{len(chunks)}个chunk")
return chunks
except Exception as e:
logger.error(f"分块失败: {e}")
return []
def _embed_chunks(self, chunks: List[TextChunk]) -> Optional[EmbeddingResult]:
"""嵌入分块"""
try:
# 延迟初始化嵌入模型
self._init_embedder()
# 提取文本内容
texts = [chunk.content for chunk in chunks]
# 执行嵌入
embeddings = self.embedder.embed(texts)
logger.success(f"嵌入完成，向量数: {len(embeddings.dense_embeddings)}")
return embeddings
except Exception as e:
logger.error(f"嵌入失败: {e}")
return None
def _insert_to_milvus(
self,
chunks: List[TextChunk],
embeddings: EmbeddingResult
) -> List[int]:
"""插入Milvus"""
try:
# 延迟初始化Milvus
self._init_milvus()
# 执行插入
inserted_ids = self.milvus.insert_chunks(chunks, embeddings)
logger.success(f"入库完成，共{len(inserted_ids)}条记录")
return inserted_ids
except Exception as e:
logger.error(f"入库失败: {e}")
return []
def search(
self,
query: str,
top_k: int = 10,
filters: Optional[str] = None
) -> List[Dict]:
"""
检索法规内容
Args:
query: 查询文本
top_k: 返回结果数量
filters: 过滤条件
Returns:
List[Dict]: 检索结果
"""
logger.info(f"执行检索: {query}")
try:
# 延迟初始化
self._init_embedder()
self._init_milvus()
# 生成查询向量
query_embedding = self.embedder.embed_single(query)
# 执行混合检索
results = self.milvus.hybrid_search(
query_dense=query_embedding['dense'].tolist(),
query_sparse=query_embedding['sparse'],
top_k=top_k,
filters=filters
)
# 转换为字典格式
result_dicts = []
for r in results:
result_dicts.append({
"id": r.id,
"content": r.content,
"score": r.score,
"metadata": r.metadata
})
logger.success(f"检索完成，返回{len(result_dicts)}条结果")
return result_dicts
except Exception as e:
logger.error(f"检索失败: {e}")
return []
def close(self):
"""关闭连接"""
if self.milvus:
self.milvus.disconnect()
logger.info("文档处理器已关闭")
def process_document(
file_path: str,
doc_name: Optional[str] = None,
regulation_type: str = "",
version: str = ""
) -> ProcessingResult:
"""便捷函数：处理单个文档"""
processor = DocumentProcessor()
result = processor.process(file_path, doc_name, regulation_type, version)
processor.close()
return result
def search_regulations(query: str, top_k: int = 10) -> List[Dict]:
"""便捷函数：检索法规"""
processor = DocumentProcessor()
results = processor.search(query, top_k)
processor.close()
return results

7
src/services/embedding/__init__.py
View File

@@ -1,7 +0,0 @@
# src/services/embedding/__init__.py
"""嵌入和分块服务"""
from .text_chunker import RegulationChunker
from .bge_m3_embedder import BGEM3Embedder
__all__ = ["RegulationChunker", "BGEM3Embedder"]

296
src/services/embedding/bge_m3_embedder.py
View File

@@ -1,296 +0,0 @@
# src/services/embedding/bge_m3_embedder.py
"""BGE-M3嵌入服务 - Dense+Sparse双路向量生成"""
import numpy as np
from typing import List, Dict, Optional, Union
from dataclasses import dataclass, field
from loguru import logger
import torch
import os
# 设置HuggingFace镜像（国内网络）
if 'HF_ENDPOINT' not in os.environ:
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
# 本地模型路径（按优先级检查）
LOCAL_MODEL_PATHS = [
os.path.expanduser("~/.cache/modelscope/Xorbits/bge-m3"), # ModelScope下载路径
os.path.expanduser("~/.cache/huggingface/hub/models--BAAI--bge-m3/snapshots/main"), # HuggingFace本地路径
]
@dataclass
class EmbeddingResult:
"""嵌入结果"""
dense_embeddings: np.ndarray # Dense向量（语义检索）
sparse_embeddings: List[Dict[int, float]] # Sparse向量（关键词匹配）
texts: List[str]
dim: int = 1024
class BGEM3Embedder:
"""
BGE-M3多语言嵌入模型服务
BGE-M3是BAAI发布的多语言嵌入模型，支持：
- Dense向量：用于语义相似度检索
- Sparse向量：用于关键词精确匹配（BM25风格）
- ColBERT向量：用于细粒度交互匹配（可选）
特点：
- 支持100+语言（中英双语优化）
- 8192 tokens超长上下文
- Dense+Sparse双路检索能力
GitHub: https://github.com/FlagOpen/FlagEmbedding
"""
def __init__(
self,
model_name: str = "BAAI/bge-m3",
use_fp16: bool = True,
device: Optional[str] = None,
batch_size: int = 12,
max_length: int = 8192,
local_model_path: Optional[str] = None
):
"""
初始化BGE-M3嵌入模型
Args:
model_name: 模型名称（如果使用本地路径，此参数会被忽略）
use_fp16: 是否使用FP16加速
device: 设备类型（cuda/cpu），默认自动选择
batch_size: 批处理大小
max_length: 最大序列长度
local_model_path: 本地模型路径（可选，优先使用）
"""
self.use_fp16 = use_fp16
self.batch_size = batch_size
self.max_length = max_length
# 确定模型路径（优先使用本地路径）
if local_model_path and os.path.exists(local_model_path):
self.model_path = local_model_path
self.model_name = "local"
logger.info(f"使用本地模型路径: {local_model_path}")
else:
# 检查多个可能的本地路径
found_local = False
for path in LOCAL_MODEL_PATHS:
if os.path.exists(path) and os.path.exists(os.path.join(path, "config.json")):
self.model_path = path
self.model_name = "local"
logger.info(f"使用本地模型路径: {path}")
found_local = True
break
if not found_local:
self.model_path = model_name
self.model_name = model_name
logger.info(f"使用远程模型: {model_name}")
# 自动选择设备
if device is None:
self.device = "cuda" if torch.cuda.is_available() else "cpu"
else:
self.device = device
logger.info(f"初始化BGE-M3模型, 设备: {self.device}")
self.model = None
self._load_model()
def _load_model(self):
"""加载嵌入模型"""
try:
from FlagEmbedding import BGEM3FlagModel
self.model = BGEM3FlagModel(
self.model_path,
use_fp16=self.use_fp16,
device=self.device
)
logger.success(f"BGE-M3模型加载成功")
except ImportError:
logger.warning("FlagEmbedding库未安装，请运行: pip install FlagEmbedding")
raise
except Exception as e:
logger.error(f"模型加载失败: {e}")
raise
def embed(
self,
texts: List[str],
return_dense: bool = True,
return_sparse: bool = True,
return_colbert_vecs: bool = False
) -> EmbeddingResult:
"""
对文本列表生成嵌入向量
Args:
texts: 文本列表
return_dense: 是否返回Dense向量
return_sparse: 是否返回Sparse向量
return_colbert_vecs: 是否返回ColBERT向量
Returns:
EmbeddingResult: 嵌入结果
"""
if not texts:
logger.warning("输入文本列表为空")
return EmbeddingResult(
dense_embeddings=np.array([]),
sparse_embeddings=[],
texts=[],
dim=0
)
logger.info(f"开始嵌入{len(texts)}个文本块")
try:
# 执行嵌入
embeddings = self.model.encode(
texts,
batch_size=self.batch_size,
max_length=self.max_length,
return_dense=return_dense,
return_sparse=return_sparse,
return_colbert_vecs=return_colbert_vecs
)
# 提取结果
dense_embeddings = embeddings.get('dense_vecs', np.array([]))
sparse_embeddings = embeddings.get('lexical_weights', [])
# 获取维度
dim = dense_embeddings.shape[1] if len(dense_embeddings) > 0 else 1024
logger.success(f"嵌入完成，向量维度: {dim}")
return EmbeddingResult(
dense_embeddings=dense_embeddings,
sparse_embeddings=sparse_embeddings,
texts=texts,
dim=dim
)
except Exception as e:
logger.error(f"嵌入失败: {e}")
raise
def embed_single(self, text: str) -> Dict[str, Union[np.ndarray, Dict]]:
"""
对单个文本生成嵌入向量
Args:
text: 输入文本
Returns:
Dict: 包含dense和sparse向量
"""
result = self.embed([text])
return {
'dense': result.dense_embeddings[0],
'sparse': result.sparse_embeddings[0] if result.sparse_embeddings else {},
'dim': result.dim
}
def embed_dense(self, texts: List[str]) -> np.ndarray:
"""只生成Dense向量"""
result = self.embed(texts, return_sparse=False, return_colbert_vecs=False)
return result.dense_embeddings
def embed_sparse(self, texts: List[str]) -> List[Dict[int, float]]:
"""只生成Sparse向量"""
result = self.embed(texts, return_dense=False, return_colbert_vecs=False)
return result.sparse_embeddings
def embed_query(self, query: str) -> Dict:
"""
对查询文本生成嵌入（用于检索）
Args:
query: 查询文本
Returns:
Dict: 包含dense和sparse向量
"""
return self.embed_single(query)
def compute_similarity(
self,
query_embedding: np.ndarray,
doc_embeddings: np.ndarray,
metric: str = "cosine"
) -> np.ndarray:
"""
计算查询与文档的相似度
Args:
query_embedding: 查询向量
doc_embeddings: 文档向量矩阵
metric: 相似度度量（cosine/dot）
Returns:
np.ndarray: 相似度分数数组
"""
if metric == "cosine":
# 余弦相似度
query_norm = np.linalg.norm(query_embedding)
doc_norms = np.linalg.norm(doc_embeddings, axis=1)
similarities = np.dot(doc_embeddings, query_embedding) / (doc_norms * query_norm)
elif metric == "dot":
# 点积相似度
similarities = np.dot(doc_embeddings, query_embedding)
else:
raise ValueError(f"不支持的相似度度量: {metric}")
return similarities
def sparse_similarity(
self,
query_sparse: Dict[int, float],
doc_sparse: Dict[int, float]
) -> float:
"""
计算Sparse向量的相似度（BM25风格）
Args:
query_sparse: 查询的Sparse向量（词ID -> 权重）
doc_sparse: 文档的Sparse向量
Returns:
float: 相似度分数
"""
# 计算交集词的点积
common_keys = set(query_sparse.keys()) & set(doc_sparse.keys())
score = sum(query_sparse[k] * doc_sparse[k] for k in common_keys)
return score
def embed_texts(
texts: List[str],
model_name: str = "BAAI/bge-m3",
**kwargs
) -> EmbeddingResult:
"""便捷函数：对文本列表生成嵌入"""
embedder = BGEM3Embedder(model_name=model_name, **kwargs)
return embedder.embed(texts)
def embed_single_text(
text: str,
model_name: str = "BAAI/bge-m3",
**kwargs
) -> Dict:
"""便捷函数：对单个文本生成嵌入"""
embedder = BGEM3Embedder(model_name=model_name, **kwargs)
return embedder.embed_single(text)

449
src/services/embedding/text_chunker.py
View File

@@ -1,449 +0,0 @@
# src/services/embedding/text_chunker.py
"""智能分块器 - 章节级+条款级双粒度切割"""
import re
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass, field
from loguru import logger
@dataclass
class ChunkMetadata:
"""分块元数据"""
doc_id: str = ""
doc_name: str = ""
chunk_id: str = ""
section_number: str = "" # 章节编号（如 "第一章"）
section_title: str = "" # 章节标题
clause_number: str = "" # 条款编号（如 "第一条"）
page_number: int = 0
start_position: int = 0 # 在原文中的起始位置
end_position: int = 0 # 在原文中的结束位置
regulation_type: str = "" # 法规类型
version: str = ""
@dataclass
class TextChunk:
"""文本分块"""
content: str
metadata: ChunkMetadata
token_count: int = 0 # 估算的token数量
class RegulationChunker:
"""
法规文档智能分块器
实现章节级/条款级双粒度切割，适配国标GB文档结构：
- 国标文档通常有明确的层级结构：章 > 节 > 条
- 每个条款应作为一个独立的语义单元
- 保留条款完整性，避免跨条款截断
"""
# 法规标题模式
CHAPTER_PATTERN = re.compile(r'^第[一二三四五六七八九十百]+章\s+[^\n]+')
SECTION_PATTERN = re.compile(r'^第[一二三四五六七八九十百]+节\s+[^\n]+')
CLAUSE_PATTERN = re.compile(r'^第[一二三四五六七八九十百]+条\s')
# 条款子项模式
SUB_ITEM_PATTERN = re.compile(r'^[\(（][一二三四五六七八九十]+[\)）]\s')
NUMBER_ITEM_PATTERN = re.compile(r'^[\d]+[\.、]\s')
def __init__(
self,
chunk_size: int = 512,
chunk_overlap: int = 50,
max_chunk_size: int = 2048,
min_chunk_size: int = 100
):
"""
初始化分块器
Args:
chunk_size: 默认分块大小（字符数）
chunk_overlap: 分块重叠大小
max_chunk_size: 最大分块大小（防止单个条款过长）
min_chunk_size: 最小分块大小（防止碎片化）
"""
self.chunk_size = chunk_size
self.chunk_overlap = chunk_overlap
self.max_chunk_size = max_chunk_size
self.min_chunk_size = min_chunk_size
def chunk_document(
self,
markdown_text: str,
doc_id: str = "",
doc_name: str = "",
regulation_type: str = "",
version: str = ""
) -> List[TextChunk]:
"""
对法规文档进行智能分块
Args:
markdown_text: Markdown格式的文档内容
doc_id: 文档ID
doc_name: 文档名称
regulation_type: 法规类型
version: 文档版本
Returns:
List[TextChunk]: 分块列表
"""
logger.info(f"开始分块文档: {doc_name}")
# 1. 按章节分割（一级分块）
sections = self._split_by_sections(markdown_text)
# 2. 在每个章节内按条款分割（二级分块）
chunks = []
global_position = 0
for section_num, section_title, section_content, section_start in sections:
# 在章节内按条款分割
clause_chunks = self._split_by_clauses(
section_content,
section_num,
section_title,
section_start + global_position
)
for chunk_content, clause_num, clause_title, start_pos, end_pos in clause_chunks:
# 处理过长的条款（进一步细分）
if len(chunk_content) > self.max_chunk_size:
sub_chunks = self._split_long_clause(
chunk_content,
clause_num,
clause_title
)
for sub_content, sub_start, sub_end in sub_chunks:
chunk = self._create_chunk(
sub_content,
doc_id,
doc_name,
section_num,
section_title,
clause_num,
sub_start + start_pos,
sub_end + start_pos,
regulation_type,
version
)
chunks.append(chunk)
else:
chunk = self._create_chunk(
chunk_content,
doc_id,
doc_name,
section_num,
section_title,
clause_num,
start_pos,
end_pos,
regulation_type,
version
)
chunks.append(chunk)
logger.success(f"分块完成，共{len(chunks)}个chunk")
return chunks
def _split_by_sections(self, markdown_text: str) -> List[Tuple[str, str, str, int]]:
"""
按章节分割文档
Returns:
List of (section_number, section_title, section_content, start_position)
"""
sections = []
lines = markdown_text.split('\n')
current_section_num = ""
current_section_title = ""
current_section_content = []
current_section_start = 0
for i, line in enumerate(lines):
# 检测章节标题
chapter_match = self.CHAPTER_PATTERN.match(line.strip())
section_match = self.SECTION_PATTERN.match(line.strip())
if chapter_match or section_match:
# 保存上一个章节
if current_section_content:
content = '\n'.join(current_section_content)
sections.append((
current_section_num,
current_section_title,
content,
current_section_start
))
# 开始新章节
current_section_start = sum(len(l) + 1 for l in lines[:i])
current_section_content = []
if chapter_match:
current_section_num = line.strip()
current_section_title = self._extract_title(line.strip())
else:
current_section_num = line.strip()
current_section_title = self._extract_title(line.strip())
current_section_content.append(line)
# 保存最后一个章节
if current_section_content:
content = '\n'.join(current_section_content)
sections.append((
current_section_num,
current_section_title,
content,
current_section_start
))
# 如果没有检测到章节，将整个文档作为一个大章节
if not sections:
sections.append((
"",
"全文",
markdown_text,
0
))
return sections
def _split_by_clauses(
self,
section_content: str,
section_num: str,
section_title: str,
section_start: int
) -> List[Tuple[str, str, str, int, int]]:
"""
在章节内按条款分割
Returns:
List of (content, clause_number, clause_title, start_position, end_position)
"""
clauses = []
lines = section_content.split('\n')
current_clause_num = ""
current_clause_title = ""
current_clause_content = []
current_clause_start = section_start
for i, line in enumerate(lines):
# 检测条款标题
clause_match = self.CLAUSE_PATTERN.match(line.strip())
if clause_match:
# 保存上一个条款
if current_clause_content:
content = '\n'.join(current_clause_content)
end_pos = current_clause_start + len(content)
clauses.append((
content,
current_clause_num,
current_clause_title,
current_clause_start,
end_pos
))
# 开始新条款
current_clause_start = section_start + sum(len(l) + 1 for l in lines[:i])
current_clause_content = []
current_clause_num = self._extract_clause_number(line.strip())
current_clause_title = line.strip()
current_clause_content.append(line)
# 保存最后一个条款
if current_clause_content:
content = '\n'.join(current_clause_content)
end_pos = current_clause_start + len(content)
clauses.append((
content,
current_clause_num,
current_clause_title,
current_clause_start,
end_pos
))
# 如果没有检测到条款，将整个章节作为一个条款
if not clauses:
clauses.append((
section_content,
"",
section_title,
section_start,
section_start + len(section_content)
))
return clauses
def _split_long_clause(
self,
content: str,
clause_num: str,
clause_title: str
) -> List[Tuple[str, int, int]]:
"""
分割过长的条款内容
按条款子项或段落分割，保持语义完整性
"""
sub_chunks = []
lines = content.split('\n')
# 检测是否有子项结构
has_sub_items = any(
self.SUB_ITEM_PATTERN.match(line.strip()) or
self.NUMBER_ITEM_PATTERN.match(line.strip())
for line in lines
)
if has_sub_items:
# 按子项分割
current_sub_content = []
current_sub_start = 0
for i, line in enumerate(lines):
is_sub_item = (
self.SUB_ITEM_PATTERN.match(line.strip()) or
self.NUMBER_ITEM_PATTERN.match(line.strip())
)
if is_sub_item and current_sub_content:
sub_content = '\n'.join(current_sub_content)
sub_end = current_sub_start + len(sub_content)
if len(sub_content) >= self.min_chunk_size:
sub_chunks.append((sub_content, current_sub_start, sub_end))
current_sub_content = []
current_sub_start = sum(len(l) + 1 for l in lines[:i])
current_sub_content.append(line)
# 保存最后一个子项
if current_sub_content:
sub_content = '\n'.join(current_sub_content)
sub_end = current_sub_start + len(sub_content)
sub_chunks.append((sub_content, current_sub_start, sub_end))
else:
# 按段落分割（滑动窗口）
paragraphs = []
current_para = []
for line in lines:
if line.strip():
current_para.append(line)
else:
if current_para:
paragraphs.append('\n'.join(current_para))
current_para = []
if current_para:
paragraphs.append('\n'.join(current_para))
# 合并段落直到达到chunk_size
current_chunk = []
current_length = 0
chunk_start = 0
for para in paragraphs:
if current_length + len(para) > self.chunk_size and current_chunk:
chunk_content = '\n'.join(current_chunk)
chunk_end = chunk_start + len(chunk_content)
sub_chunks.append((chunk_content, chunk_start, chunk_end))
current_chunk = []
current_length = 0
chunk_start = chunk_end
current_chunk.append(para)
current_length += len(para)
# 保存最后一个chunk
if current_chunk:
chunk_content = '\n'.join(current_chunk)
chunk_end = chunk_start + len(chunk_content)
sub_chunks.append((chunk_content, chunk_start, chunk_end))
return sub_chunks
def _extract_title(self, header_line: str) -> str:
"""从标题行提取标题内容"""
# 移除"第X章"、"第X节"前缀
title = re.sub(r'^第[一二三四五六七八九十百]+[章节]\s+', '', header_line)
return title.strip()
def _extract_clause_number(self, clause_line: str) -> str:
"""从条款行提取条款编号"""
match = self.CLAUSE_PATTERN.match(clause_line)
if match:
return match.group(0).strip()
return ""
def _create_chunk(
self,
content: str,
doc_id: str,
doc_name: str,
section_num: str,
section_title: str,
clause_num: str,
start_pos: int,
end_pos: int,
regulation_type: str,
version: str
) -> TextChunk:
"""创建文本分块"""
# 清理内容
content = content.strip()
# 计算估算token数（中文约1.5字符/token）
token_count = int(len(content) * 0.7) # 简化估算
# 生成chunk_id
chunk_id = f"{doc_id}_{section_num}_{clause_num}_{start_pos}"
metadata = ChunkMetadata(
doc_id=doc_id,
doc_name=doc_name,
chunk_id=chunk_id,
section_number=section_num,
section_title=section_title,
clause_number=clause_num,
start_position=start_pos,
end_position=end_pos,
regulation_type=regulation_type,
version=version
)
return TextChunk(
content=content,
metadata=metadata,
token_count=token_count
)
def chunk_regulation_document(
markdown_text: str,
doc_id: str = "",
doc_name: str = "",
regulation_type: str = "",
version: str = "",
chunk_size: int = 512
) -> List[TextChunk]:
"""便捷函数：对法规文档进行分块"""
chunker = RegulationChunker(chunk_size=chunk_size)
return chunker.chunk_document(
markdown_text,
doc_id,
doc_name,
regulation_type,
version
)

15
src/services/llm/__init__.py
View File

@@ -1,15 +0,0 @@
# src/services/llm/__init__.py
"""LLM服务模块"""
from .llm_factory import LLMFactory, get_llm_client
from .base_client import BaseLLMClient, LLMResponse, LLMConfig, LLMProvider
from .deepseek_client import DeepSeekClient
from .qwen_client import QwenClient, QwenVLClient
from .document_summarizer import DocumentSummarizer, summarize_document, DocumentSummary
__all__ = [
"LLMFactory", "get_llm_client",
"BaseLLMClient", "LLMResponse", "LLMConfig", "LLMProvider",
"DeepSeekClient", "QwenClient", "QwenVLClient",
"DocumentSummarizer", "summarize_document", "DocumentSummary"
]

116
src/services/llm/base_client.py
View File

@@ -1,116 +0,0 @@
# src/services/llm/base_client.py
"""LLM客户端基类 - 统一接口定义"""
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import List, Dict, Optional, Any
from enum import Enum
class LLMProvider(Enum):
"""LLM提供商"""
DEEPSEEK = "deepseek"
QWEN = "qwen"
QWEN_VL = "qwen_vl"
@dataclass
class LLMResponse:
"""LLM响应结果"""
content: str
model: str
usage: Dict[str, int] = field(default_factory=dict)
finish_reason: str = "stop"
latency_ms: int = 0
error: Optional[str] = None
@property
def is_success(self) -> bool:
return self.error is None
@dataclass
class LLMConfig:
"""LLM配置"""
provider: LLMProvider
model: str
api_key: str
base_url: str
max_tokens: int = 4096
temperature: float = 0.7
top_p: float = 0.9
timeout: int = 300 # 默认超时300秒（摘要/Skills生成可能需要较长时间）
class BaseLLMClient(ABC):
"""LLM客户端基类"""
def __init__(self, config: LLMConfig):
self.config = config
self._client = None
@abstractmethod
def _init_client(self):
"""初始化客户端"""
pass
@abstractmethod
def chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> LLMResponse:
"""
对话补全
Args:
messages: 对话消息列表 [{"role": "user/assistant/system", "content": "..."}]
max_tokens: 最大输出token数
temperature: 温度参数
**kwargs: 其他参数
Returns:
LLMResponse: 响应结果
"""
pass
def complete(
self,
prompt: str,
system_prompt: Optional[str] = None,
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> LLMResponse:
"""
单轮补全（便捷方法）
Args:
prompt: 用户输入
system_prompt: 系统提示词
max_tokens: 最大输出token数
temperature: 温度参数
Returns:
LLMResponse: 响应结果
"""
messages = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
return self.chat(messages, max_tokens, temperature, **kwargs)
@abstractmethod
def get_available_models(self) -> List[str]:
"""获取可用模型列表"""
pass
def estimate_tokens(self, text: str) -> int:
"""估算文本token数（粗略估计）"""
# 中文字符约1.5 token，英文约0.25 token
chinese_chars = sum(1 for c in text if '一' <= c <= '鿿')
other_chars = len(text) - chinese_chars
return int(chinese_chars * 1.5 + other_chars * 0.25)

130
src/services/llm/deepseek_client.py
View File

@@ -1,130 +0,0 @@
# src/services/llm/deepseek_client.py
"""DeepSeek LLM客户端 - OpenAI兼容API"""
import time
from typing import List, Dict, Optional
from loguru import logger
import httpx
from .base_client import BaseLLMClient, LLMResponse, LLMConfig, LLMProvider
class DeepSeekClient(BaseLLMClient):
"""
DeepSeek API客户端（OpenAI兼容格式）
支持模型：
- deepseek-chat
- deepseek-coder
- deepseek-reasoner
- deepseek-v3
- deepseek-v3.2
- deepseek-v4-flash
"""
SUPPORTED_MODELS = [
"deepseek-chat",
"deepseek-coder",
"deepseek-reasoner",
"deepseek-v3",
"deepseek-v3.2",
"deepseek-v4-flash"
]
def __init__(self, config: LLMConfig):
if config.provider != LLMProvider.DEEPSEEK:
raise ValueError(f"配置provider应为DEEPSEEK，实际为{config.provider}")
super().__init__(config)
self._init_client()
def _init_client(self):
"""初始化HTTP客户端"""
self._client = httpx.Client(
base_url=self.config.base_url,
headers={
"Authorization": f"Bearer {self.config.api_key}",
"Content-Type": "application/json"
},
timeout=self.config.timeout
)
logger.info(f"DeepSeek客户端初始化完成: {self.config.base_url} - {self.config.model}")
def chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> LLMResponse:
"""对话补全"""
start_time = time.time()
try:
payload = {
"model": self.config.model,
"messages": messages,
"max_tokens": max_tokens or self.config.max_tokens,
"temperature": temperature or self.config.temperature,
"top_p": kwargs.get("top_p", self.config.top_p),
"stream": False
}
response = self._client.post("/chat/completions", json=payload)
response.raise_for_status()
data = response.json()
latency_ms = int((time.time() - start_time) * 1000)
choices = data.get("choices", [{}])
message = choices[0].get("message", {})
return LLMResponse(
content=message.get("content", ""),
model=data.get("model", self.config.model),
usage=data.get("usage", {}),
finish_reason=choices[0].get("finish_reason", "stop"),
latency_ms=latency_ms
)
except httpx.HTTPStatusError as e:
logger.error(f"DeepSeek API错误: {e.response.status_code} - {e.response.text}")
return LLMResponse(
content="",
model=self.config.model,
error=f"API错误: {e.response.status_code} - {e.response.text[:200]}"
)
except Exception as e:
logger.error(f"DeepSeek调用失败: {e}")
return LLMResponse(
content="",
model=self.config.model,
error=str(e)
)
def get_available_models(self) -> List[str]:
"""获取可用模型列表"""
return self.SUPPORTED_MODELS
def close(self):
"""关闭客户端"""
if self._client:
self._client.close()
def create_deepseek_client(
api_key: str,
model: str = "deepseek-v4-flash",
base_url: str = "http://6.86.80.4:30080/v1",
**kwargs
) -> DeepSeekClient:
"""便捷函数：创建DeepSeek客户端"""
config = LLMConfig(
provider=LLMProvider.DEEPSEEK,
model=model,
api_key=api_key,
base_url=base_url,
**kwargs
)
return DeepSeekClient(config)

231
src/services/llm/document_summarizer.py
View File

@@ -1,231 +0,0 @@
# src/services/llm/document_summarizer.py
"""文档摘要生成服务 - LLM生成法规文档摘要"""
from typing import Dict, Optional
from dataclasses import dataclass
from loguru import logger
from src.services.llm import get_llm_client, BaseLLMClient
from src.services.rag.prompt_templates import get_prompt_template
from src.config.settings import settings
@dataclass
class DocumentSummary:
"""文档摘要结果"""
doc_name: str
summary: str
applicable_scope: str
key_clauses: list
key_terms: list
compliance_points: list
model: str
latency_ms: int
error: Optional[str] = None
@property
def is_success(self) -> bool:
return self.error is None
class DocumentSummarizer:
"""
文档摘要生成器
功能：
- 生成法规文档的核心要点摘要
- 提取适用范围
- 突出关键条款
- 列出合规要点
使用示例：
summarizer = DocumentSummarizer()
result = summarizer.summarize("GB 7258-2017", markdown_content)
print(result.summary)
"""
def __init__(
self,
provider: str = None,
model: str = None,
max_tokens: int = None
):
"""
初始化摘要生成器
Args:
provider: LLM提供商
model: LLM模型名称
max_tokens: 最大输出token数
"""
self.provider = provider or settings.llm_provider
self.model = model or settings.llm_model
self.max_tokens = max_tokens or settings.rag_summary_max_tokens
# LLM客户端（延迟加载）
self.llm: Optional[BaseLLMClient] = None
logger.info(f"摘要生成器初始化: provider={self.provider}, model={self.model}")
def _init_llm(self):
"""延迟初始化LLM"""
if self.llm is None:
self.llm = get_llm_client(
provider=self.provider,
model=self.model
)
def summarize(
self,
doc_name: str,
content: str,
regulation_type: str = "",
max_tokens: Optional[int] = None
) -> DocumentSummary:
"""
生成文档摘要
Args:
doc_name: 文档名称
content: 文档内容（Markdown格式）
regulation_type: 法规类型
max_tokens: 最大输出token数
Returns:
DocumentSummary: 摘要结果
"""
import time
start_time = time.time()
logger.info(f"生成文档摘要: {doc_name}")
try:
self._init_llm()
# 使用摘要模板
template = get_prompt_template("document_summary")
# 构建用户消息
user_content = template.user_template.format(
doc_name=doc_name,
content=content[:8000] # 截取前8000字符（避免超出token限制）
)
# 调用LLM
response = self.llm.chat(
messages=[
{"role": "system", "content": template.system_prompt},
{"role": "user", "content": user_content}
],
max_tokens=max_tokens or self.max_tokens,
temperature=0.3 # 低温度保证摘要准确性
)
latency_ms = int((time.time() - start_time) * 1000)
if not response.is_success:
return DocumentSummary(
doc_name=doc_name,
summary="",
applicable_scope="",
key_clauses=[],
key_terms=[],
compliance_points=[],
model=self.model,
latency_ms=latency_ms,
error=response.error
)
# 解析摘要结构
summary_data = self._parse_summary(response.content)
logger.success(f"摘要生成完成: {doc_name}, {latency_ms}ms")
return DocumentSummary(
doc_name=doc_name,
summary=summary_data.get("summary", response.content),
applicable_scope=summary_data.get("applicable_scope", ""),
key_clauses=summary_data.get("key_clauses", []),
key_terms=summary_data.get("key_terms", []),
compliance_points=summary_data.get("compliance_points", []),
model=response.model,
latency_ms=latency_ms
)
except Exception as e:
logger.error(f"摘要生成失败: {e}")
return DocumentSummary(
doc_name=doc_name,
summary="",
applicable_scope="",
key_clauses=[],
key_terms=[],
compliance_points=[],
model=self.model,
latency_ms=0,
error=str(e)
)
def _parse_summary(self, content: str) -> Dict:
"""解析摘要内容（提取结构化信息）"""
result = {
"summary": content,
"applicable_scope": "",
"key_clauses": [],
"key_terms": [],
"compliance_points": []
}
# 简单解析（提取关键信息）
lines = content.split("\n")
for line in lines:
line = line.strip()
# 提取适用范围
if "适用范围" in line or "适用对象" in line:
result["applicable_scope"] = line.split("：")[-1].strip() if "：" in line else line.split(":")[-1].strip()
# 提取关键条款
if line.startswith("- 【条款") or line.startswith("【条款"):
result["key_clauses"].append(line)
# 提取关键术语
if "关键术语" in line or "术语定义" in line:
# 继续读取后续几行
pass
# 提取合规要点
if "合规要点" in line or "必须满足" in line:
pass
return result
def batch_summarize(
self,
documents: list
) -> list:
"""
批量生成摘要
Args:
documents: 文档列表 [{"doc_name": str, "content": str}, ...]
Returns:
list: 摘要结果列表
"""
results = []
for doc in documents:
result = self.summarize(doc["doc_name"], doc["content"])
results.append(result)
return results
def summarize_document(
doc_name: str,
content: str,
**kwargs
) -> DocumentSummary:
"""便捷函数：生成文档摘要"""
summarizer = DocumentSummarizer(**kwargs)
return summarizer.summarize(doc_name, content)

258
src/services/llm/llm_factory.py
View File

@@ -1,258 +0,0 @@
# src/services/llm/llm_factory.py
"""LLM工厂 - 统一创建和管理LLM客户端"""
from typing import Optional, Dict, Any
from loguru import logger
from functools import lru_cache
from .base_client import BaseLLMClient, LLMConfig, LLMProvider, LLMResponse
from .deepseek_client import DeepSeekClient
from .qwen_client import QwenClient, QwenVLClient
# 默认模型映射
DEFAULT_MODELS = {
LLMProvider.DEEPSEEK: "deepseek-v4-flash",
LLMProvider.QWEN: "qwen3.5-flash",
LLMProvider.QWEN_VL: "qwen3-vl-plus"
}
# API基础URL（使用统一代理服务）
DEFAULT_BASE_URLS = {
LLMProvider.DEEPSEEK: "http://6.86.80.4:30080/v1",
LLMProvider.QWEN: "http://6.86.80.4:30080/v1",
LLMProvider.QWEN_VL: "http://6.86.80.4:30080/v1"
}
class LLMFactory:
"""
LLM客户端工厂（支持全局缓存）
支持的提供商和模型：
- DeepSeek: deepseek-chat (DeepSeek-V3), deepseek-coder
- Qwen: qwen-turbo, qwen-plus, qwen-max, qwen-long
- QwenVL: qwen-vl-plus, qwen-vl-max (多模态)
使用示例：
factory = LLMFactory()
# 使用默认配置
client = factory.create("deepseek")
# 自定义配置
client = factory.create("qwen", model="qwen-max", temperature=0.5)
# 调用LLM
response = client.complete("你好，介绍一下自己")
"""
# 全局客户端缓存（类级别，跨实例共享）
_global_instances: Dict[str, BaseLLMClient] = {}
def __init__(self):
self._config_cache: Dict[str, Any] = {}
def create(
self,
provider: str,
api_key: Optional[str] = None,
model: Optional[str] = None,
base_url: Optional[str] = None,
max_tokens: int = 4096,
temperature: float = 0.7,
**kwargs
) -> BaseLLMClient:
"""
创建LLM客户端
Args:
provider: 提供商名称 ("deepseek", "qwen", "qwen_vl")
api_key: API密钥（如未提供，从环境变量获取）
model: 模型名称（如未提供，使用默认模型）
base_url: API基础URL
max_tokens: 最大输出token数
temperature: 温度参数
**kwargs: 其他配置参数
Returns:
BaseLLMClient: LLM客户端实例
"""
provider_enum = self._parse_provider(provider)
# 获取配置
api_key = api_key or self._get_api_key(provider_enum)
model = model or DEFAULT_MODELS.get(provider_enum)
base_url = base_url or DEFAULT_BASE_URLS.get(provider_enum)
if not api_key:
raise ValueError(f"缺少API密钥，请设置环境变量或传入api_key参数")
# 检查全局缓存
cache_key = f"{provider}_{model}"
if cache_key in LLMFactory._global_instances:
logger.debug(f"使用缓存的LLM客户端: {cache_key}")
return LLMFactory._global_instances[cache_key]
config = LLMConfig(
provider=provider_enum,
model=model,
api_key=api_key,
base_url=base_url,
max_tokens=max_tokens,
temperature=temperature,
**kwargs
)
# 创建客户端
client = self._create_client(config)
# 缓存到全局实例
LLMFactory._global_instances[cache_key] = client
logger.info(f"LLM客户端创建成功并缓存: {provider} - {model}")
return client
def _parse_provider(self, provider: str) -> LLMProvider:
"""解析提供商名称"""
provider_map = {
"deepseek": LLMProvider.DEEPSEEK,
"deepseek-v3": LLMProvider.DEEPSEEK,
"deepseek_chat": LLMProvider.DEEPSEEK,
"qwen": LLMProvider.QWEN,
"qwen-turbo": LLMProvider.QWEN,
"qwen-plus": LLMProvider.QWEN,
"qwen-max": LLMProvider.QWEN,
"qwen3.5-flash": LLMProvider.QWEN,
"qwen3.5-plus": LLMProvider.QWEN,
"qwen_vl": LLMProvider.QWEN_VL,
"qwen-vl": LLMProvider.QWEN_VL,
"qwen-vl-plus": LLMProvider.QWEN_VL,
"qwen-vl-max": LLMProvider.QWEN_VL
}
provider_lower = provider.lower()
if provider_lower not in provider_map:
raise ValueError(f"不支持的提供商: {provider}，支持的: {list(provider_map.keys())}")
return provider_map[provider_lower]
def _get_api_key(self, provider: LLMProvider) -> Optional[str]:
"""从环境变量获取API密钥"""
import os
key_map = {
LLMProvider.DEEPSEEK: ["DEEPSEEK_API_KEY", "OPENAI_API_KEY"],
LLMProvider.QWEN: ["QWEN_API_KEY", "DASHSCOPE_API_KEY", "ALIBABA_API_KEY"],
LLMProvider.QWEN_VL: ["QWEN_API_KEY", "DASHSCOPE_API_KEY", "ALIBABA_API_KEY"]
}
for key_name in key_map.get(provider, []):
api_key = os.getenv(key_name)
if api_key:
return api_key
return None
def _create_client(self, config: LLMConfig) -> BaseLLMClient:
"""创建具体客户端"""
client_map = {
LLMProvider.DEEPSEEK: DeepSeekClient,
LLMProvider.QWEN: QwenClient,
LLMProvider.QWEN_VL: QwenVLClient
}
client_class = client_map.get(config.provider)
if not client_class:
raise ValueError(f"不支持的提供商: {config.provider}")
return client_class(config)
def get_cached(self, provider: str, model: Optional[str] = None) -> Optional[BaseLLMClient]:
"""获取缓存的客户端"""
provider_enum = self._parse_provider(provider)
model = model or DEFAULT_MODELS.get(provider_enum)
cache_key = f"{provider}_{model}"
return LLMFactory._global_instances.get(cache_key)
def list_available_providers(self) -> Dict[str, list]:
"""列出可用的提供商和模型"""
return {
"deepseek": DeepSeekClient.SUPPORTED_MODELS,
"qwen": QwenClient.SUPPORTED_MODELS,
"qwen_vl": QwenVLClient.SUPPORTED_MODELS
}
@classmethod
def preload_clients(cls, providers: list = None):
"""
预加载LLM客户端（应用启动时调用）
Args:
providers: 要预加载的提供商列表，默认加载qwen和deepseek
"""
if providers is None:
providers = ["qwen", "deepseek"]
factory = cls()
for provider in providers:
try:
client = factory.create(provider)
logger.success(f"预加载LLM客户端成功: {provider}")
except Exception as e:
logger.warning(f"预加载LLM客户端失败: {provider} - {e}")
@classmethod
def get_global_client(cls, provider: str, model: Optional[str] = None) -> Optional[BaseLLMClient]:
"""获取全局缓存的客户端"""
provider_lower = provider.lower()
# 处理模型名作为provider的情况（如 qwen3.5-flash）
if provider_lower.startswith("qwen"):
provider_lower = "qwen"
model = model or DEFAULT_MODELS.get(LLMProvider.QWEN if provider_lower == "qwen" else LLMProvider.DEEPSEEK)
cache_key = f"{provider_lower}_{model}"
return cls._global_instances.get(cache_key)
@classmethod
def cleanup(cls):
"""清理所有缓存的客户端"""
for cache_key, client in cls._global_instances.items():
try:
client.close()
logger.debug(f"关闭LLM客户端: {cache_key}")
except Exception as e:
logger.warning(f"关闭LLM客户端失败: {cache_key} - {e}")
cls._global_instances.clear()
logger.info("所有LLM客户端已清理")
@lru_cache
def get_llm_factory() -> LLMFactory:
"""获取LLM工厂实例（缓存）"""
return LLMFactory()
def get_llm_client(
provider: str = "qwen",
model: Optional[str] = None,
**kwargs
) -> BaseLLMClient:
"""
便捷函数：获取LLM客户端（优先使用缓存）
Args:
provider: 提供商名称
model: 模型名称
**kwargs: 其他配置
Returns:
BaseLLMClient: LLM客户端实例
"""
factory = get_llm_factory()
# 先尝试获取缓存的实例
cached = factory.get_cached(provider, model)
if cached:
return cached
return factory.create(provider, model=model, **kwargs)

392
src/services/llm/qwen_client.py
View File

@@ -1,392 +0,0 @@
# src/services/llm/qwen_client.py
"""Qwen LLM客户端 - 支持OpenAI兼容API格式"""
import time
import json
from typing import List, Dict, Optional, Generator, AsyncGenerator
from loguru import logger
import httpx
from .base_client import BaseLLMClient, LLMResponse, LLMConfig, LLMProvider
class QwenClient(BaseLLMClient):
"""
Qwen API客户端（OpenAI兼容格式）
支持通过new-api等代理服务调用：
- qwen-turbo
- qwen-plus
- qwen-max
- qwen3.5-flash (推荐：快速响应)
- qwen3.5-plus
- qwen-long
- qwen2.5系列
"""
SUPPORTED_MODELS = [
"qwen-turbo",
"qwen-plus",
"qwen-max",
"qwen-max-longcontext",
"qwen-long",
"qwen3.5-flash",
"qwen3.5-plus",
"qwen3-plus",
"qwen2.5-72b-instruct",
"qwen2.5-32b-instruct",
"qwen2.5-14b-instruct",
"qwen2.5-7b-instruct"
]
def __init__(self, config: LLMConfig):
if config.provider not in [LLMProvider.QWEN, LLMProvider.QWEN_VL]:
raise ValueError(f"配置provider应为Qwen，实际为{config.provider}")
super().__init__(config)
self._init_client()
def _init_client(self):
"""初始化HTTP客户端"""
# OpenAI兼容API格式
self._client = httpx.Client(
base_url=self.config.base_url,
headers={
"Authorization": f"Bearer {self.config.api_key}",
"Content-Type": "application/json"
},
timeout=self.config.timeout
)
logger.info(f"Qwen客户端初始化完成: {self.config.base_url} - {self.config.model}")
def chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> LLMResponse:
"""对话补全（OpenAI兼容格式）"""
start_time = time.time()
try:
# OpenAI兼容格式的请求体
payload = {
"model": self.config.model,
"messages": messages,
"max_tokens": max_tokens or self.config.max_tokens,
"temperature": temperature or self.config.temperature,
"top_p": kwargs.get("top_p", self.config.top_p),
"stream": False
}
# OpenAI兼容接口路径
response = self._client.post("/chat/completions", json=payload)
response.raise_for_status()
data = response.json()
latency_ms = int((time.time() - start_time) * 1000)
# OpenAI兼容格式的响应解析
choices = data.get("choices", [{}])
message = choices[0].get("message", {})
return LLMResponse(
content=message.get("content", ""),
model=data.get("model", self.config.model),
usage=data.get("usage", {}),
finish_reason=choices[0].get("finish_reason", "stop"),
latency_ms=latency_ms
)
except httpx.HTTPStatusError as e:
logger.error(f"Qwen API错误: {e.response.status_code} - {e.response.text}")
return LLMResponse(
content="",
model=self.config.model,
error=f"API错误: {e.response.status_code} - {e.response.text[:200]}"
)
except Exception as e:
logger.error(f"Qwen调用失败: {e}")
return LLMResponse(
content="",
model=self.config.model,
error=str(e)
)
def stream_chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> Generator[str, None, None]:
"""
流式对话补全（SSE格式）
Yields:
str: 每次返回一个文本片段
使用示例:
for chunk in client.stream_chat(messages):
print(chunk, end="", flush=True)
"""
try:
# OpenAI兼容格式的请求体，启用流式输出
payload = {
"model": self.config.model,
"messages": messages,
"max_tokens": max_tokens or self.config.max_tokens,
"temperature": temperature or self.config.temperature,
"top_p": kwargs.get("top_p", self.config.top_p),
"stream": True # 启用流式输出
}
# 使用stream模式发送请求
with self._client.stream("POST", "/chat/completions", json=payload) as response:
for line in response.iter_lines():
if line:
line = line.strip()
# SSE格式: data: {...}
if line.startswith("data: "):
data_str = line[6:] # 移除 "data: " 前缀
if data_str == "[DONE]":
break
try:
data = json.loads(data_str)
choices = data.get("choices", [])
if not choices:
continue # 跳过空的choices
delta = choices[0].get("delta", {})
content = delta.get("content", "")
if content:
yield content
except json.JSONDecodeError:
continue
except httpx.HTTPStatusError as e:
logger.error(f"Qwen流式API错误: {e.response.status_code}")
yield f"[ERROR: API返回错误 {e.response.status_code}]"
except Exception as e:
logger.error(f"Qwen流式调用失败: {e}")
yield f"[ERROR: {str(e)}]"
async def async_stream_chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> AsyncGenerator[str, None]:
"""
异步流式对话补全（用于FastAPI SSE响应）
Yields:
str: 每次返回一个文本片段
"""
import asyncio
# 使用同步流式方法，包装为异步
for chunk in self.stream_chat(messages, max_tokens, temperature, **kwargs):
yield chunk
# 给async循环一个小延迟，让其他任务有机会执行
await asyncio.sleep(0)
def get_available_models(self) -> List[str]:
"""获取可用模型列表"""
return self.SUPPORTED_MODELS
def close(self):
"""关闭客户端"""
if self._client:
self._client.close()
class QwenVLClient(BaseLLMClient):
"""
Qwen VL多模态客户端（OpenAI兼容格式）
支持模型：
- qwen-vl-plus
- qwen-vl-max
- qwen3-vl-plus
- qwen2-vl-7b-instruct
- qwen2-vl-72b-instruct
"""
SUPPORTED_MODELS = [
"qwen-vl-plus",
"qwen-vl-max",
"qwen3-vl-plus",
"qwen2-vl-7b-instruct",
"qwen2-vl-72b-instruct"
]
def __init__(self, config: LLMConfig):
if config.provider != LLMProvider.QWEN_VL:
raise ValueError(f"配置provider应为QWEN_VL，实际为{config.provider}")
super().__init__(config)
self._init_client()
def _init_client(self):
"""初始化HTTP客户端"""
self._client = httpx.Client(
base_url=self.config.base_url,
headers={
"Authorization": f"Bearer {self.config.api_key}",
"Content-Type": "application/json"
},
timeout=self.config.timeout
)
logger.info(f"QwenVL客户端初始化完成: {self.config.base_url} - {self.config.model}")
def chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> LLMResponse:
"""多模态对话补全（OpenAI兼容格式）
支持图片输入，消息格式：
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
{"type": "text", "text": "描述这张图片"}
]
}
"""
start_time = time.time()
try:
# OpenAI兼容格式的请求体
payload = {
"model": self.config.model,
"messages": messages,
"max_tokens": max_tokens or self.config.max_tokens,
"temperature": temperature or self.config.temperature,
"top_p": kwargs.get("top_p", self.config.top_p),
"stream": False
}
response = self._client.post("/chat/completions", json=payload)
response.raise_for_status()
data = response.json()
latency_ms = int((time.time() - start_time) * 1000)
choices = data.get("choices", [{}])
message = choices[0].get("message", {})
return LLMResponse(
content=message.get("content", ""),
model=data.get("model", self.config.model),
usage=data.get("usage", {}),
finish_reason=choices[0].get("finish_reason", "stop"),
latency_ms=latency_ms
)
except httpx.HTTPStatusError as e:
logger.error(f"QwenVL API错误: {e.response.status_code} - {e.response.text}")
return LLMResponse(
content="",
model=self.config.model,
error=f"API错误: {e.response.status_code} - {e.response.text[:200]}"
)
except Exception as e:
logger.error(f"QwenVL调用失败: {e}")
return LLMResponse(
content="",
model=self.config.model,
error=str(e)
)
def stream_chat(
self,
messages: List[Dict[str, str]],
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
**kwargs
) -> Generator[str, None, None]:
"""流式多模态对话补全"""
try:
payload = {
"model": self.config.model,
"messages": messages,
"max_tokens": max_tokens or self.config.max_tokens,
"temperature": temperature or self.config.temperature,
"top_p": kwargs.get("top_p", self.config.top_p),
"stream": True
}
with self._client.stream("POST", "/chat/completions", json=payload) as response:
for line in response.iter_lines():
if line:
line = line.strip()
if line.startswith("data: "):
data_str = line[6:]
if data_str == "[DONE]":
break
try:
data = json.loads(data_str)
choices = data.get("choices", [])
if not choices:
continue # 跳过空的choices
delta = choices[0].get("delta", {})
content = delta.get("content", "")
if content:
yield content
except json.JSONDecodeError:
continue
except Exception as e:
logger.error(f"QwenVL流式调用失败: {e}")
yield f"[ERROR: {str(e)}]"
def get_available_models(self) -> List[str]:
"""获取可用模型列表"""
return self.SUPPORTED_MODELS
def close(self):
"""关闭客户端"""
if self._client:
self._client.close()
def create_qwen_client(
api_key: str,
model: str = "qwen3.5-flash",
base_url: str = "http://6.86.80.4:30080/v1",
**kwargs
) -> QwenClient:
"""便捷函数：创建Qwen客户端"""
config = LLMConfig(
provider=LLMProvider.QWEN,
model=model,
api_key=api_key,
base_url=base_url,
**kwargs
)
return QwenClient(config)
def create_qwen_vl_client(
api_key: str,
model: str = "qwen3-vl-plus",
base_url: str = "http://6.86.80.4:30080/v1",
**kwargs
) -> QwenVLClient:
"""便捷函数：创建QwenVL客户端"""
config = LLMConfig(
provider=LLMProvider.QWEN_VL,
model=model,
api_key=api_key,
base_url=base_url,
**kwargs
)
return QwenVLClient(config)

7
src/services/parser/__init__.py
View File

@@ -1,7 +0,0 @@
# src/services/parser/__init__.py
"""文档解析服务"""
from .pdf_parser import PDFParser
from .docx_parser import DocxParser
__all__ = ["PDFParser", "DocxParser"]

287
src/services/parser/docx_parser.py
View File

@@ -1,287 +0,0 @@
# src/services/parser/docx_parser.py
"""Word文档解析 - 使用python-docx"""
from docx import Document
from docx.enum.text import WD_ALIGN_PARAGRAPH
from typing import List, Dict, Optional
from dataclasses import dataclass, field
from loguru import logger
import re
@dataclass
class DocxParagraph:
"""段落内容"""
text: str
level: int = 0 # 标题级别，0表示正文
is_list: bool = False
list_number: Optional[str] = None
@dataclass
class DocxTable:
"""表格内容"""
rows: List[List[str]]
markdown: str = ""
@dataclass
class DocxDocumentContent:
"""Word文档完整内容"""
file_path: str
paragraphs: List[DocxParagraph]
tables: List[DocxTable]
metadata: Dict[str, str] = field(default_factory=dict)
markdown_text: str = ""
class DocxParser:
"""Word文档解析器 - 基于python-docx"""
def __init__(self):
self.document = None
def parse(self, file_path: str) -> DocxDocumentContent:
"""
解析Word文档
Args:
file_path: Word文档路径
Returns:
DocxDocumentContent: 解析后的文档内容
"""
logger.info(f"开始解析Word文档: {file_path}")
try:
self.document = Document(file_path)
doc_content = DocxDocumentContent(
file_path=file_path,
paragraphs=[],
tables=[]
)
# 提取文档元数据
doc_content.metadata = self._extract_metadata()
# 提取段落
doc_content.paragraphs = self._extract_paragraphs()
# 提取表格
doc_content.tables = self._extract_tables()
# 生成Markdown格式文本
doc_content.markdown_text = self._generate_markdown(doc_content)
logger.success(f"Word文档解析完成，共{len(doc_content.paragraphs)}个段落")
return doc_content
except Exception as e:
logger.error(f"Word文档解析失败: {e}")
raise
def _extract_metadata(self) -> Dict[str, str]:
"""提取文档元数据"""
metadata = {}
try:
core_props = self.document.core_properties
metadata = {
"title": core_props.title or "",
"author": core_props.author or "",
"subject": core_props.subject or "",
"keywords": core_props.keywords or "",
"created": str(core_props.created) if core_props.created else "",
"modified": str(core_props.modified) if core_props.modified else "",
}
except Exception as e:
logger.warning(f"提取元数据失败: {e}")
return metadata
def _extract_paragraphs(self) -> List[DocxParagraph]:
"""提取所有段落"""
paragraphs = []
for para in self.document.paragraphs:
text = para.text.strip()
if not text:
continue
# 判断标题级别
level = self._get_paragraph_level(para)
# 判断是否是列表项
is_list, list_number = self._detect_list_item(para)
paragraph = DocxParagraph(
text=text,
level=level,
is_list=is_list,
list_number=list_number
)
paragraphs.append(paragraph)
return paragraphs
def _get_paragraph_level(self, para) -> int:
"""
判断段落标题级别
Returns:
int: 标题级别，0表示正文
"""
# 方法1：检查段落样式
style_name = para.style.name if para.style else ""
if "Heading" in style_name or "标题" in style_name:
# 从样式名称中提取级别
match = re.search(r'Heading\s*(\d)|标题\s*(\d)', style_name)
if match:
level = int(match.group(1) or match.group(2))
return level
# 方法2：检查段落格式（字号）
# 标题通常字号较大
if para.paragraph_format:
# 可以根据字号判断，这里简化处理
pass
# 方法3：根据内容模式判断（法规文档特征）
text = para.text.strip()
# 第一章、第X章 -> 二级标题
if re.match(r'^第[一二三四五六七八九十百]+章\s', text):
return 2
# 第X节 -> 三级标题
elif re.match(r'^第[一二三四五六七八九十百]+节\s', text):
return 3
# 第X条 -> 四级标题
elif re.match(r'^第[一二三四五六七八九十百]+条\s', text):
return 4
return 0 # 正文
def _detect_list_item(self, para) -> tuple[bool, Optional[str]]:
"""检测是否是列表项"""
text = para.text.strip()
# 数字列表：1.、2.、（1）、[1]等
if re.match(r'^[\d]+[.、)\]]\s', text):
match = re.match(r'^([\d]+[.、)\]])\s', text)
return True, match.group(1) if match else None
# 中文数字列表：一、二、（一）等
if re.match(r'^[一二三四五六七八九十]+[、.)]\s', text):
match = re.match(r'^([一二三四五六七八九十]+[、.)])\s', text)
return True, match.group(1) if match else None
# 检查段落格式中的列表编号
if para.paragraph_format and hasattr(para.paragraph_format, 'left_indent'):
# 有缩进的可能是列表项
pass
return False, None
def _extract_tables(self) -> List[DocxTable]:
"""提取所有表格"""
tables = []
for table in self.document.tables:
rows = []
for row in table.rows:
cells = []
for cell in row.cells:
cells.append(cell.text.strip())
rows.append(cells)
# 转换为Markdown表格
markdown = self._table_to_markdown(rows)
table_content = DocxTable(rows=rows, markdown=markdown)
tables.append(table_content)
return tables
def _table_to_markdown(self, rows: List[List[str]]) -> str:
"""将表格转换为Markdown格式"""
if not rows or len(rows) < 1:
return ""
lines = []
# 表头
if len(rows) >= 1:
header = rows[0]
lines.append("| " + " | ".join(cell for cell in header) + " |")
lines.append("| " + " | ".join("---" for _ in header) + " |")
# 数据行
for row in rows[1:]:
lines.append("| " + " | ".join(cell for cell in row) + " |")
return "\n".join(lines)
def _generate_markdown(self, doc_content: DocxDocumentContent) -> str:
"""生成Markdown格式文本"""
lines = []
# 文档标题
title = doc_content.metadata.get("title", "")
if title:
lines.append(f"# {title}\n")
else:
# 从第一个段落获取标题（如果是标题样式）
for para in doc_content.paragraphs[:5]:
if para.level == 1:
lines.append(f"# {para.text}\n")
break
else:
lines.append(f"# {doc_content.file_path}\n")
# 元数据信息
lines.append("\n## 文档信息\n")
for key, value in doc_content.metadata.items():
if value:
lines.append(f"- **{key}**: {value}")
# 正文内容
lines.append("\n## 正文\n")
table_index = 0
for para in doc_content.paragraphs:
if para.level > 0:
# 标题
prefix = "#" * para.level
lines.append(f"\n{prefix} {para.text}\n")
elif para.is_list:
# 列表项
lines.append(f"- {para.text}")
else:
# 正文
lines.append(para.text)
# 添加表格
if doc_content.tables:
lines.append("\n## 表格\n")
for i, table in enumerate(doc_content.tables):
lines.append(f"\n### 表格 {i + 1}\n")
lines.append(table.markdown + "\n")
return "\n".join(lines)
def parse_to_markdown(self, file_path: str) -> str:
"""直接解析并返回Markdown文本"""
doc_content = self.parse(file_path)
return doc_content.markdown_text
def parse_docx(file_path: str) -> DocxDocumentContent:
"""便捷函数：解析Word文档"""
parser = DocxParser()
return parser.parse(file_path)
def parse_docx_to_markdown(file_path: str) -> str:
"""便捷函数：解析Word并返回Markdown"""
parser = DocxParser()
return parser.parse_to_markdown(file_path)

204
src/services/parser/mineru_parser.py
View File

@@ -1,204 +0,0 @@
# src/services/parser/mineru_parser.py
"""MinerU多模态PDF解析 - 版面感知解析"""
from typing import Optional, Dict
from dataclasses import dataclass, field
from loguru import logger
import os
@dataclass
class MinerUResult:
"""MinerU解析结果"""
file_path: str
markdown_text: str
metadata: Dict[str, str] = field(default_factory=dict)
success: bool = True
error_message: str = ""
class MinerUParser:
"""
MinerU多模态PDF解析器
MinerU (magic-pdf) 是一个开源的高质量PDF解析工具，
支持版面感知解析，能够识别文档中的标题、正文、表格、图片等元素，
并输出结构化的Markdown格式。
GitHub: https://github.com/opendatalab/MinerU
"""
def __init__(self):
self.available = self._check_mineru_available()
def _check_mineru_available(self) -> bool:
"""检查MinerU是否可用"""
try:
from magic_pdf.pipe.UNIPipe import UNIPipe
return True
except ImportError:
logger.warning("MinerU (magic-pdf) 未安装，请运行: pip install magic-pdf[full]")
return False
def parse(self, file_path: str, output_dir: Optional[str] = None) -> MinerUResult:
"""
使用MinerU解析PDF文档
Args:
file_path: PDF文件路径
output_dir: 输出目录（可选，用于保存解析产物）
Returns:
MinerUResult: 解析结果
"""
logger.info(f"尝试使用MinerU解析: {file_path}")
if not self.available:
return MinerUResult(
file_path=file_path,
markdown_text="",
success=False,
error_message="MinerU未安装"
)
try:
from magic_pdf.pipe.UNIPipe import UNIPipe
from magic_pdf.libs.MakeContentConfig import DropMode
# 设置输出目录
if output_dir is None:
output_dir = os.path.dirname(file_path)
# 创建解析管道
# OCR模式可以根据PDF类型选择
# auto: 自动判断是否需要OCR
# txt: 纯文本PDF（无OCR）
# ocr: 扫描件PDF（OCR）
pipe = UNIPipe(file_path, output_dir)
# 执行解析
# pipe_mk() 返回Markdown格式文本
markdown_content = pipe.pipe_mk()
logger.success(f"MinerU解析成功")
return MinerUResult(
file_path=file_path,
markdown_text=markdown_content,
metadata=self._extract_metadata(pipe),
success=True
)
except Exception as e:
logger.error(f"MinerU解析失败: {e}")
return MinerUResult(
file_path=file_path,
markdown_text="",
success=False,
error_message=str(e)
)
def _extract_metadata(self, pipe) -> Dict[str, str]:
"""从解析管道提取元数据"""
metadata = {}
try:
# MinerU解析管道中可能包含的元数据信息
if hasattr(pipe, 'pdf_mid_data') and pipe.pdf_mid_data:
mid_data = pipe.pdf_mid_data
# 提取可能的元数据字段
metadata = {
"page_count": str(mid_data.get("page_count", "")),
"language": str(mid_data.get("language", "")),
"is_scanned": str(mid_data.get("is_scanned", "")),
}
except Exception as e:
logger.warning(f"提取MinerU元数据失败: {e}")
return metadata
def parse_to_markdown(self, file_path: str) -> str:
"""直接解析并返回Markdown文本"""
result = self.parse(file_path)
return result.markdown_text if result.success else ""
class ParserOrchestrator:
"""
解析服务编排 - 按优先级选择解析器
解析策略：
1. 优先尝试MinerU（版面感知能力强）
2. MinerU失败时回退到基础PyMuPDF解析
"""
def __init__(self):
from .pdf_parser import PDFParser
self.mineru_parser = MinerUParser()
self.pdf_parser = PDFParser()
self.mineru_available = self.mineru_parser.available
def parse_pdf(self, file_path: str, prefer_mineru: bool = True) -> str:
"""
解析PDF文档，按优先级选择解析器
Args:
file_path: PDF文件路径
prefer_mineru: 是否优先使用MinerU
Returns:
str: Markdown格式文本
"""
markdown_text = ""
if prefer_mineru and self.mineru_available:
# 优先尝试MinerU
result = self.mineru_parser.parse(file_path)
if result.success:
markdown_text = result.markdown_text
logger.info("使用MinerU解析成功")
return markdown_text
else:
logger.warning(f"MinerU解析失败，回退到PyMuPDF: {result.error_message}")
# 回退到PyMuPDF基础解析
logger.info("使用PyMuPDF基础解析")
markdown_text = self.pdf_parser.parse_to_markdown(file_path)
return markdown_text
def parse_docx(self, file_path: str) -> str:
"""解析Word文档"""
from .docx_parser import DocxParser
docx_parser = DocxParser()
return docx_parser.parse_to_markdown(file_path)
def parse(self, file_path: str) -> str:
"""
根据文件类型选择解析器
Args:
file_path: 文件路径
Returns:
str: Markdown格式文本
"""
ext = os.path.splitext(file_path)[1].lower()
if ext == ".pdf":
return self.parse_pdf(file_path)
elif ext in [".docx", ".doc"]:
return self.parse_docx(file_path)
else:
raise ValueError(f"不支持的文件类型: {ext}")
def parse_with_mineru(file_path: str) -> MinerUResult:
"""便捷函数：使用MinerU解析"""
parser = MinerUParser()
return parser.parse(file_path)
def parse_pdf_smart(file_path: str) -> str:
"""便捷函数：智能解析PDF（自动选择最佳解析器）"""
orchestrator = ParserOrchestrator()
return orchestrator.parse_pdf(file_path)

268
src/services/parser/pdf_parser.py
View File

@@ -1,268 +0,0 @@
# src/services/parser/pdf_parser.py
"""PDF文档解析 - 使用PyMuPDF基础解析"""
import fitz # PyMuPDF
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass, field
from loguru import logger
import re
@dataclass
class PDFPageContent:
"""PDF页面内容"""
page_number: int
text: str
tables: List[str] = field(default_factory=list)
images: List[str] = field(default_factory=list) # 图片路径列表
blocks: List[Dict] = field(default_factory=list)
@dataclass
class PDFDocumentContent:
"""PDF文档完整内容"""
file_path: str
total_pages: int
pages: List[PDFPageContent]
metadata: Dict[str, str] = field(default_factory=dict)
markdown_text: str = ""
class PDFParser:
"""PDF文档解析器 - 基于PyMuPDF"""
def __init__(self):
self.pdf = None
def parse(self, file_path: str, extract_tables: bool = True, extract_images: bool = False) -> PDFDocumentContent:
"""
解析PDF文档
Args:
file_path: PDF文件路径
extract_tables: 是否提取表格
extract_images: 是否提取图片
Returns:
PDFDocumentContent: 解析后的文档内容
"""
logger.info(f"开始解析PDF文档: {file_path}")
try:
self.pdf = fitz.open(file_path)
doc_content = PDFDocumentContent(
file_path=file_path,
total_pages=self.pdf.page_count,
pages=[]
)
# 提取文档元数据
doc_content.metadata = self._extract_metadata()
# 逐页解析
for page_num in range(self.pdf.page_count):
page = self.pdf[page_num]
page_content = self._parse_page(page, page_num + 1, extract_tables, extract_images)
doc_content.pages.append(page_content)
# 生成Markdown格式文本
doc_content.markdown_text = self._generate_markdown(doc_content)
self.pdf.close()
logger.success(f"PDF解析完成，共{doc_content.total_pages}页")
return doc_content
except Exception as e:
logger.error(f"PDF解析失败: {e}")
raise
def _extract_metadata(self) -> Dict[str, str]:
"""提取PDF元数据"""
metadata = {}
try:
meta = self.pdf.metadata
metadata = {
"title": meta.get("title", ""),
"author": meta.get("author", ""),
"subject": meta.get("subject", ""),
"keywords": meta.get("keywords", ""),
"creator": meta.get("creator", ""),
"producer": meta.get("producer", ""),
"creation_date": meta.get("creationDate", ""),
"mod_date": meta.get("modDate", ""),
}
except Exception as e:
logger.warning(f"提取元数据失败: {e}")
return metadata
def _parse_page(self, page: fitz.Page, page_num: int,
extract_tables: bool, extract_images: bool) -> PDFPageContent:
"""解析单页内容"""
page_content = PDFPageContent(page_number=page_num, text="")
# 提取文本块（保留结构）
blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
page_content.blocks = blocks
# 提取纯文本
text = page.get_text("text", flags=fitz.TEXT_PRESERVE_WHITESPACE)
page_content.text = text.strip()
# 提取表格（使用PyMuPDF的表格提取功能）
if extract_tables:
tables = self._extract_tables_from_page(page)
page_content.tables = tables
# 提取图片
if extract_images:
images = self._extract_images_from_page(page, page_num)
page_content.images = images
return page_content
def _extract_tables_from_page(self, page: fitz.Page) -> List[str]:
"""
从页面提取表格（基于文本块分析）
注意：PyMuPDF基础版表格提取能力有限，复杂表格建议使用MinerU
"""
tables = []
try:
# 使用PyMuPDF的表格提取方法（2.4+版本）
# 对于更复杂的表格，需要在mineru_parser中使用更高级的方法
tabs = page.find_tables()
if tabs:
for tab in tabs:
table_text = tab.extract()
# 将表格转换为Markdown格式
markdown_table = self._table_to_markdown(table_text)
tables.append(markdown_table)
except AttributeError:
# 旧版本PyMuPDF没有表格提取功能
logger.warning("PyMuPDF版本不支持表格提取，请升级到2.4+版本")
except Exception as e:
logger.warning(f"表格提取失败: {e}")
return tables
def _table_to_markdown(self, table_data: List[List[str]]) -> str:
"""将表格数据转换为Markdown格式"""
if not table_data or len(table_data) < 1:
return ""
lines = []
# 表头
if len(table_data) >= 1:
header = table_data[0]
lines.append("| " + " | ".join(str(cell).strip() for cell in header) + " |")
lines.append("| " + " | ".join("---" for _ in header) + " |")
# 数据行
for row in table_data[1:]:
lines.append("| " + " | ".join(str(cell).strip() for cell in row) + " |")
return "\n".join(lines)
def _extract_images_from_page(self, page: fitz.Page, page_num: int) -> List[str]:
"""提取页面图片"""
images = []
# 图片提取功能（可选实现）
# 这里仅记录图片信息，实际图片需要额外保存
try:
image_list = page.get_images()
for img_index, img in enumerate(image_list):
xref = img[0]
images.append(f"image_p{page_num}_i{img_index}_xref{xref}")
except Exception as e:
logger.warning(f"图片提取失败: {e}")
return images
def _generate_markdown(self, doc_content: PDFDocumentContent) -> str:
"""生成Markdown格式文本"""
lines = []
# 文档标题
title = doc_content.metadata.get("title", "")
if title:
lines.append(f"# {title}\n")
else:
lines.append(f"# {doc_content.file_path}\n")
# 元数据信息
lines.append("\n## 文档信息\n")
for key, value in doc_content.metadata.items():
if value and key in ["author", "subject", "keywords", "creation_date"]:
lines.append(f"- **{key}**: {value}")
# 正文内容
lines.append("\n## 正文\n")
for page in doc_content.pages:
# 页码标记
lines.append(f"\n---\n**第 {page.page_number} 页**\n")
# 处理文本内容，识别标题结构
text = self._process_page_text(page.text, page.blocks)
lines.append(text)
# 添加表格
for table in page.tables:
lines.append("\n" + table + "\n")
return "\n".join(lines)
def _process_page_text(self, text: str, blocks: List[Dict]) -> str:
"""处理页面文本，识别标题结构"""
# 基于字体大小识别标题
processed_text = text
# 尝试识别标题（基于字号）
# 法规文档通常有明确的层级结构：章、节、条
processed_text = self._detect_headers(text, blocks)
return processed_text
def _detect_headers(self, text: str, blocks: List[Dict]) -> str:
"""检测并标记标题（基于字号或内容模式）"""
lines = text.split("\n")
processed_lines = []
for line in lines:
line = line.strip()
if not line:
continue
# 法规标题模式检测
# 第一章、第X章、第X节、第X条等
if re.match(r'^第[一二三四五六七八九十百]+章\s', line):
processed_lines.append(f"\n## {line}\n")
elif re.match(r'^第[一二三四五六七八九十百]+节\s', line):
processed_lines.append(f"\n### {line}\n")
elif re.match(r'^第[一二三四五六七八九十百]+条\s', line):
processed_lines.append(f"\n#### {line}\n")
elif re.match(r'^[一二三四五六七八九十]+\s*[、.]', line):
# 条款子项
processed_lines.append(f"- {line}")
else:
processed_lines.append(line)
return "\n".join(processed_lines)
def parse_to_markdown(self, file_path: str) -> str:
"""直接解析并返回Markdown文本"""
doc_content = self.parse(file_path)
return doc_content.markdown_text
def parse_pdf(file_path: str, **kwargs) -> PDFDocumentContent:
"""便捷函数：解析PDF文档"""
parser = PDFParser()
return parser.parse(file_path, **kwargs)
def parse_pdf_to_markdown(file_path: str) -> str:
"""便捷函数：解析PDF并返回Markdown"""
parser = PDFParser()
return parser.parse_to_markdown(file_path)

12
src/services/rag/__init__.py
View File

@@ -1,12 +0,0 @@
# src/services/rag/__init__.py
"""RAG服务模块"""
from .retriever import Retriever, retrieve_regulations
from .context_builder import ContextBuilder, build_rag_context
from .prompt_templates import PromptTemplates, get_prompt_template
__all__ = [
"Retriever", "retrieve_regulations",
"ContextBuilder", "build_rag_context",
"PromptTemplates", "get_prompt_template"
]

230
src/services/rag/context_builder.py
View File

@@ -1,230 +0,0 @@
# src/services/rag/context_builder.py
"""RAG上下文构建服务 - 构建LLM输入上下文"""
from typing import List, Dict, Optional
from dataclasses import dataclass
from loguru import logger
from .retriever import RetrievedDocument
from src.config.settings import settings
@dataclass
class RAGContext:
"""RAG构建的上下文"""
system_prompt: str
context_text: str
user_query: str
total_tokens: int
sources: List[Dict]
truncated: bool = False
class ContextBuilder:
"""
RAG上下文构建器
功能：
- 格式化检索结果为上下文文本
- 控制上下文长度（token限制）
- 构建完整的LLM输入格式
"""
def __init__(
self,
max_context_tokens: int = None,
include_metadata: bool = True,
citation_format: str = "【条款{clause}】"
):
"""
初始化上下文构建器
Args:
max_context_tokens: 最大上下文token数
include_metadata: 是否包含元数据（文档名、条款号等）
citation_format: 引用格式模板
"""
self.max_context_tokens = max_context_tokens or settings.rag_max_context_tokens
self.include_metadata = include_metadata
self.citation_format = citation_format
logger.info(f"上下文构建器初始化: max_tokens={self.max_context_tokens}")
def build(
self,
query: str,
documents: List[RetrievedDocument],
system_prompt: Optional[str] = None,
max_tokens: Optional[int] = None
) -> RAGContext:
"""
构建RAG上下文
Args:
query: 用户查询
documents: 检索到的文档列表
system_prompt: 系统提示词（可选）
max_tokens: 最大token数（可选，覆盖默认值）
Returns:
RAGContext: 构建的上下文对象
"""
max_tokens = max_tokens or self.max_context_tokens
# 格式化文档内容
context_text, sources, truncated = self._format_documents(
documents,
max_tokens
)
# 构建系统提示词
system_prompt = system_prompt or self._default_system_prompt()
# 估算总token数
total_tokens = self._estimate_tokens(system_prompt + context_text + query)
logger.info(f"上下文构建完成: {len(documents)}条文档, {total_tokens}tokens, truncated={truncated}")
return RAGContext(
system_prompt=system_prompt,
context_text=context_text,
user_query=query,
total_tokens=total_tokens,
sources=sources,
truncated=truncated
)
def _format_documents(
self,
documents: List[RetrievedDocument],
max_tokens: int
) -> tuple:
"""
格式化文档内容
Args:
documents: 文档列表
max_tokens: 最大token数
Returns:
(context_text, sources, truncated)
"""
context_parts = []
sources = []
current_tokens = 0
truncated = False
for i, doc in enumerate(documents):
# 格式化单个文档
formatted = self._format_single_doc(doc, i + 1)
# 估算token数
doc_tokens = self._estimate_tokens(formatted)
# 检查是否超出限制
if current_tokens + doc_tokens > max_tokens:
truncated = True
logger.warning(f"上下文截断: 已达到{max_tokens}tokens限制")
break
context_parts.append(formatted)
current_tokens += doc_tokens
# 记录来源
sources.append({
"index": i + 1,
"doc_id": doc.doc_id,
"doc_name": doc.doc_name,
"section_title": doc.section_title,
"clause_number": doc.clause_number,
"page_number": doc.page_number,
"score": doc.score
})
context_text = "\n\n".join(context_parts)
return context_text, sources, truncated
def _format_single_doc(
self,
doc: RetrievedDocument,
index: int
) -> str:
"""格式化单个文档"""
parts = []
# 索引编号
parts.append(f"[{index}]")
# 元数据（可选）
if self.include_metadata:
meta_parts = []
if doc.doc_name:
meta_parts.append(f"文档: {doc.doc_name}")
if doc.section_title:
meta_parts.append(f"章节: {doc.section_title}")
if doc.clause_number:
clause_text = self.citation_format.format(clause=doc.clause_number)
meta_parts.append(clause_text)
if meta_parts:
parts.append(" | ".join(meta_parts))
# 内容
parts.append(doc.content)
return "\n".join(parts)
def _default_system_prompt(self) -> str:
"""默认系统提示词"""
return """你是合规专家助手，基于提供的法规条款回答问题。
回答要求：
1. 直接回答问题，必须引用具体条款编号（如【条款5.2.1】）
2. 如引用的条款不完整，说明需要进一步查阅原文
3. 给出明确的合规建议和操作指导
4. 如果检索内容不足以回答问题，如实说明
回答格式：
- 先给出直接结论
- 然后引用支撑条款
- 最后给出合规建议"""
def _estimate_tokens(self, text: str) -> int:
"""估算文本token数"""
# 中文字符约1.5 token，英文约0.25 token
chinese_chars = sum(1 for c in text if '一' <= c <= '鿿')
other_chars = len(text) - chinese_chars
return int(chinese_chars * 1.5 + other_chars * 0.25)
def build_messages(
self,
context: RAGContext
) -> List[Dict[str, str]]:
"""
构建LLM消息格式
Args:
context: RAG上下文对象
Returns:
List[Dict]: [{"role": "system/user/assistant", "content": "..."}]
"""
messages = [
{"role": "system", "content": context.system_prompt},
{"role": "user", "content": f"参考以下法规条款回答问题。\n\n{context.context_text}\n\n问题：{context.user_query}"}
]
return messages
def build_rag_context(
query: str,
documents: List[RetrievedDocument],
**kwargs
) -> RAGContext:
"""便捷函数：构建RAG上下文"""
builder = ContextBuilder()
return builder.build(query, documents, **kwargs)

296
src/services/rag/prompt_templates.py
View File

@@ -1,296 +0,0 @@
# src/services/rag/prompt_templates.py
"""RAG Prompt模板 - 合规问答专用Prompt"""
from typing import Dict, Optional
from dataclasses import dataclass
@dataclass
class PromptTemplate:
"""Prompt模板"""
name: str
system_prompt: str
user_template: str
description: str
class PromptTemplates:
"""
合规问答Prompt模板库
包含多种场景的Prompt模板：
- 合规问答（标准）
- 条款解读（详细解释）
- 合规检查（判断合规状态）
- 差异对比（新旧法规对比）
- 报告生成（合规报告）
"""
# 合规问答标准模板
COMPLIANCE_QA = PromptTemplate(
name="compliance_qa",
system_prompt="""你是合规专家助手，专门解答法规合规问题。
角色定位：
- 深谙国家法规标准（GB标准、行业标准）
- 熟悉车辆安全、数据安全、EHS等领域合规要求
- 提供专业、准确、可操作的合规建议
回答规范：
1. 必须引用具体条款编号（如【条款5.2.1】）
2. 优先引用高相关性条款（score ≥ 0.5）
3. 如条款内容不完整，明确提示需要查阅原文
4. 给出明确的合规结论和建议
5. 如检索内容不足以回答，如实说明
回答格式：
【结论】直接给出合规判断或答案
【条款依据】
- 【条款X.X.X】简要内容摘要（相关性: 高/中）
- ...
【合规建议】
1. 具体操作建议
2. 需要注意的风险点
3. 后续行动建议""",
user_template="""请根据以下法规条款回答问题。
【法规条款】
{context}
【用户问题】
{query}""",
description="标准合规问答模板"
)
# 条款解读模板（详细解释）
CLAUSE_INTERPRETATION = PromptTemplate(
name="clause_interpretation",
system_prompt="""你是法规解读专家，负责详细解释法规条款的含义和应用。
解读要求：
1. 逐句解释条款原文的含义
2. 说明条款的目的和背景
3. 举例说明条款的实际应用场景
4. 解释常见的误解和注意事项
解读格式：
【条款原文】完整引用条款
【逐句解读】
- "原文句1"：解读含义
- "原文句2"：解读含义
...
【应用场景】
具体举例说明该条款在实际工作中如何应用
【注意事项】
常见误解、执行难点、合规风险点""",
user_template="""请解读以下法规条款：
条款编号：{clause_number}
条款内容：{content}
用户关注点：{query}""",
description="条款详细解读模板"
)
# 合规检查模板（判断合规状态）
COMPLIANCE_CHECK = PromptTemplate(
name="compliance_check",
system_prompt="""你是合规检查专家，负责评估企业行为或产品的合规状态。
检查流程：
1. 理解企业行为/产品描述
2. 识别相关的法规条款
3. 逐条对照检查合规状态
4. 给出综合合规结论和整改建议
合规状态分类：
- ✅ 符合：完全满足法规要求
- ⚠️ 需评估：需要进一步核实或补充材料
- ❌ 不符合：明确违反法规要求
- ❓ 无适用条款：检索内容不足以判断
检查格式：
【合规检查报告】
一、检查对象
{描述企业行为/产品}
二、条款对照检查
| 条款编号 | 要求摘要 | 检查状态 | 说明 |
|--------|---------|---------|------|
| 【条款X.X.X】 | ... | ✅/⚠️/❌/❓ | ... |
三、综合结论
合规等级：A/B/C/D（完全合规/基本合规/部分合规/不合规）
四、整改建议（如需要）
1. ...
2. ...""",
user_template="""请对以下企业行为进行合规检查：
【行为/产品描述】
{query}
【相关法规条款】
{context}""",
description="合规检查评估模板"
)
# 差异对比模板（新旧法规对比）
COMPARISON = PromptTemplate(
name="comparison",
system_prompt="""你是法规变更分析专家，负责对比新旧法规版本的差异。
对比任务：
1. 识别新旧版本的条款差异
2. 分类差异类型（新增/修改/删除）
3. 分析差异的影响范围
4. 给出企业应对建议
差异分类：
- 🆕 新增条款：原版本不存在
- 🔄 修改条款：内容有实质性变更
- ❌ 删除条款：原条款被移除
- ⚖️ 调整条款：仅格式/编号调整，实质内容不变
对比格式：
【法规变更对比分析】
一、变更概述
- 旧版本：{version_old}
- 新版本：{version_new}
- 变更条款数：{count}
二、差异明细
| 类型 | 条款编号 | 旧版本内容 | 新版本内容 | 变化要点 |
|-----|---------|-----------|-----------|---------|
| 🆕 | X.X.X | - | ... | 新增要求... |
三、影响分析
- 高影响：{条款列表}
- 中影响：{条款列表}
- 低影响：{条款列表}
四、应对建议
1. 立即整改项
2. 逐步调整项
3. 持续关注项""",
user_template="""请对比分析以下法规差异：
【用户问题】
{query}
【旧版本条款】
{context_old}
【新版本条款】
{context_new}""",
description="法规版本对比模板"
)
# 报告生成模板
REPORT_GENERATION = PromptTemplate(
name="report_generation",
system_prompt="""你是合规报告撰写专家，负责生成结构化的合规分析报告。
报告要求：
1. 结构清晰、逻辑严谨
2. 数据准确、引用规范
3. 结论明确、建议可操作
4. 语言专业、表达简洁
报告结构：
1. 概述（背景、范围）
2. 分析内容（主体分析）
3. 发现问题（合规差距）
4. 整改建议（具体措施）
5. 附录（引用条款原文）""",
user_template="""请生成以下合规报告：
【报告主题】
{query}
【分析依据】
{context}
【报告要求】
{requirements}""",
description="合规报告生成模板"
)
# 文档摘要生成模板
DOCUMENT_SUMMARY = PromptTemplate(
name="document_summary",
system_prompt="""你是法规文档摘要专家，负责生成法规文档的核心要点摘要。
摘要要求：
1. 精炼核心内容，不超过1024字
2. 突出关键合规要求和条款编号
3. 说明适用范围和生效条件
4. 列出重要定义和术语解释
摘要格式：
【法规名称】{doc_name}
【适用范围】{适用范围描述}
【核心条款摘要】
- 【条款X.X.X】{关键要求}（重要度：高）
- ...
【关键术语】
- 术语1：定义解释
- ...
【合规要点】
1. 必须满足的核心要求
2. 需要特别注意的条款""",
user_template="""请生成以下法规文档的摘要：
【文档名称】
{doc_name}
【文档内容】
{content}
请生成不超过1024字的摘要。""",
description="文档摘要生成模板"
)
@classmethod
def get_template(cls, name: str) -> Optional[PromptTemplate]:
"""获取指定模板"""
templates = {
"compliance_qa": cls.COMPLIANCE_QA,
"clause_interpretation": cls.CLAUSE_INTERPRETATION,
"compliance_check": cls.COMPLIANCE_CHECK,
"comparison": cls.COMPARISON,
"report": cls.REPORT_GENERATION,
"document_summary": cls.DOCUMENT_SUMMARY
}
return templates.get(name)
@classmethod
def list_templates(cls) -> Dict[str, str]:
"""列出所有模板"""
return {
"compliance_qa": cls.COMPLIANCE_QA.description,
"clause_interpretation": cls.CLAUSE_INTERPRETATION.description,
"compliance_check": cls.COMPLIANCE_CHECK.description,
"comparison": cls.COMPARISON.description,
"report": cls.REPORT_GENERATION.description,
"document_summary": cls.DOCUMENT_SUMMARY.description
}
def get_prompt_template(name: str) -> PromptTemplate:
"""便捷函数：获取Prompt模板"""
template = PromptTemplates.get_template(name)
if not template:
raise ValueError(f"不存在的模板: {name}")
return template

193
src/services/rag/retriever.py
View File

@@ -1,193 +0,0 @@
# src/services/rag/retriever.py
"""RAG检索服务 - 封装Milvus检索"""
from typing import List, Dict, Optional, Any
from dataclasses import dataclass, field
from loguru import logger
from src.services.embedding.bge_m3_embedder import BGEM3Embedder
from src.services.storage.milvus_client import MilvusClient, SearchResult
from src.config.settings import settings
@dataclass
class RetrievedDocument:
"""检索到的文档"""
content: str
doc_id: str # 文档ID，用于下载
doc_name: str
section_title: str
clause_number: str
page_number: int
score: float
metadata: Dict[str, Any] = field(default_factory=dict)
class Retriever:
"""
RAG检索器
功能：
- 向量检索（Dense + Sparse混合）
- 重排序（可选）
- 过滤和筛选
"""
def __init__(
self,
top_k: int = None,
rerank: bool = False,
min_score: float = 0.3
):
"""
初始化检索器
Args:
top_k: 检索召回数量
rerank: 是否启用重排序
min_score: 最低相关性分数阈值
"""
self.top_k = top_k or settings.rag_top_k
self.rerank = rerank
self.min_score = min_score
# 嵌入模型（延迟加载）
self.embedder: Optional[BGEM3Embedder] = None
# Milvus客户端（延迟连接）
self.milvus: Optional[MilvusClient] = None
logger.info(f"检索器初始化: top_k={self.top_k}, rerank={self.rerank}")
def _init_embedder(self):
"""延迟初始化嵌入模型"""
if self.embedder is None:
logger.info("加载嵌入模型...")
self.embedder = BGEM3Embedder(model_name=settings.embedding_model)
def _init_milvus(self):
"""延迟初始化Milvus"""
if self.milvus is None:
logger.info("连接Milvus...")
self.milvus = MilvusClient()
self.milvus.connect()
self.milvus.create_collection(recreate=False)
self.milvus.load_collection()
def retrieve(
self,
query: str,
filters: Optional[str] = None,
top_k: Optional[int] = None
) -> List[RetrievedDocument]:
"""
检索相关文档
Args:
query: 查询文本
filters: 过滤条件（如 "regulation_type=='车辆安全'"）
top_k: 返回数量（可选，覆盖默认值）
Returns:
List[RetrievedDocument]: 检索结果列表
"""
logger.info(f"执行检索: {query}")
# 初始化组件
self._init_embedder()
self._init_milvus()
# 生成查询向量
query_embedding = self.embedder.embed_single(query)
# 执行混合检索
results = self.milvus.hybrid_search(
query_dense=query_embedding['dense'].tolist(),
query_sparse=query_embedding['sparse'],
top_k=top_k or self.top_k,
filters=filters
)
# 转换为RetrievedDocument格式
documents = []
for r in results:
if r.score >= self.min_score:
doc = RetrievedDocument(
content=r.content,
doc_id=r.metadata.get("doc_id", ""),
doc_name=r.metadata.get("doc_name", ""),
section_title=r.metadata.get("section_title", ""),
clause_number=r.metadata.get("clause_number", ""),
page_number=r.metadata.get("page_number", 0),
score=r.score,
metadata=r.metadata
)
documents.append(doc)
logger.success(f"检索完成，返回{len(documents)}条结果（阈值过滤后）")
return documents
def retrieve_with_scores(
self,
query: str,
filters: Optional[str] = None
) -> List[Dict]:
"""
检索并返回完整结果（包含分数）
Args:
query: 查询文本
filters: 过滤条件
Returns:
List[Dict]: 包含分数的检索结果
"""
documents = self.retrieve(query, filters)
return [
{
"content": doc.content,
"doc_id": doc.doc_id,
"doc_name": doc.doc_name,
"section_title": doc.section_title,
"clause_number": doc.clause_number,
"page_number": doc.page_number,
"score": doc.score
}
for doc in documents
]
def search_by_doc_name(
self,
query: str,
doc_name: str
) -> List[RetrievedDocument]:
"""按文档名称过滤检索"""
filters = f'doc_name=="{doc_name}"'
return self.retrieve(query, filters)
def search_by_regulation_type(
self,
query: str,
regulation_type: str
) -> List[RetrievedDocument]:
"""按法规类型过滤检索"""
filters = f'regulation_type=="{regulation_type}"'
return self.retrieve(query, filters)
def close(self):
"""关闭连接"""
if self.milvus:
self.milvus.disconnect()
logger.info("检索器已关闭")
def retrieve_regulations(
query: str,
top_k: int = 10,
filters: Optional[str] = None
) -> List[RetrievedDocument]:
"""便捷函数：检索法规"""
retriever = Retriever(top_k=top_k)
results = retriever.retrieve(query, filters)
retriever.close()
return results

7
src/services/storage/__init__.py
View File

@@ -1,7 +0,0 @@
# src/services/storage/__init__.py
"""存储服务"""
from .milvus_client import MilvusClient
from .minio_client import MinIOClient
__all__ = ["MilvusClient", "MinIOClient"]

485
src/services/storage/milvus_client.py
View File

@@ -1,485 +0,0 @@
# src/services/storage/milvus_client.py
"""Milvus向量数据库客户端 - 存储与检索服务"""
from pymilvus import (
connections,
Collection,
FieldSchema,
CollectionSchema,
DataType,
utility
)
from typing import List, Dict, Optional, Any
from dataclasses import dataclass, field
from loguru import logger
import time
import numpy as np
from ..embedding.text_chunker import TextChunk
from ..embedding.bge_m3_embedder import EmbeddingResult
from src.config.settings import settings
@dataclass
class SearchResult:
"""检索结果"""
id: int
content: str
score: float
metadata: Dict[str, Any] = field(default_factory=dict)
@dataclass
class MilvusDocument:
"""Milvus文档数据结构"""
doc_id: str
chunk_id: str
content: str
dense_vector: List[float]
sparse_vector: Dict[int, float]
doc_name: str
section_title: str
clause_number: str
page_number: int
regulation_type: str
version: str
create_time: int
class MilvusClient:
"""Milvus向量数据库客户端"""
COLLECTION_NAME = "regulations"
SCHEMA_FIELDS = [
FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
FieldSchema(name="doc_id", dtype=DataType.VARCHAR, max_length=64),
FieldSchema(name="chunk_id", dtype=DataType.VARCHAR, max_length=128),
FieldSchema(name="content", dtype=DataType.VARCHAR, max_length=8192),
FieldSchema(name="dense_vector", dtype=DataType.FLOAT_VECTOR, dim=1024),
FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR),
FieldSchema(name="doc_name", dtype=DataType.VARCHAR, max_length=256),
FieldSchema(name="section_title", dtype=DataType.VARCHAR, max_length=512),
FieldSchema(name="clause_number", dtype=DataType.VARCHAR, max_length=64),
FieldSchema(name="page_number", dtype=DataType.INT64),
FieldSchema(name="regulation_type", dtype=DataType.VARCHAR, max_length=32),
FieldSchema(name="version", dtype=DataType.VARCHAR, max_length=32),
FieldSchema(name="create_time", dtype=DataType.INT64),
]
def __init__(
self,
host: str = None,
port: int = None,
collection_name: str = None,
db_name: str = None
):
self.host = host or settings.milvus_host
self.port = port or settings.milvus_port
self.collection_name = collection_name or settings.milvus_collection
self.db_name = db_name or settings.milvus_db_name
self.collection: Optional[Collection] = None
self.connected = False
logger.info(f"Milvus客户端配置: {self.host}:{self.port}, Collection: {self.collection_name}")
def connect(self) -> bool:
"""连接到Milvus服务器"""
try:
connections.connect(
alias="default",
host=self.host,
port=self.port,
db_name=self.db_name
)
self.connected = True
logger.success(f"Milvus连接成功: {self.host}:{self.port}")
return True
except Exception as e:
logger.error(f"Milvus连接失败: {e}")
self.connected = False
return False
def disconnect(self):
"""断开连接"""
try:
connections.disconnect("default")
self.connected = False
logger.info("Milvus连接已断开")
except Exception as e:
logger.warning(f"断开连接时出错: {e}")
def create_collection(self, recreate: bool = False) -> bool:
"""创建Collection"""
if not self.connected:
logger.warning("未连接到Milvus，请先调用connect()")
return False
try:
if utility.has_collection(self.collection_name):
if recreate:
logger.info(f"删除已存在的Collection: {self.collection_name}")
utility.drop_collection(self.collection_name)
else:
logger.info(f"Collection已存在: {self.collection_name}")
self.collection = Collection(self.collection_name)
return True
schema = CollectionSchema(
fields=self.SCHEMA_FIELDS,
description="法规文档向量存储",
enable_dynamic_field=True
)
self.collection = Collection(
name=self.collection_name,
schema=schema
)
self._create_indexes()
logger.success(f"Collection创建成功: {self.collection_name}")
return True
except Exception as e:
logger.error(f"Collection创建失败: {e}")
return False
def _create_indexes(self):
"""创建向量索引"""
if not self.collection:
return
try:
dense_index_params = {
"metric_type": "COSINE",
"index_type": "IVF_FLAT",
"params": {"nlist": 128}
}
self.collection.create_index(
field_name="dense_vector",
index_params=dense_index_params
)
sparse_index_params = {
"metric_type": "IP",
"index_type": "SPARSE_INVERTED_INDEX",
"params": {"drop_ratio_build": 0.2}
}
self.collection.create_index(
field_name="sparse_vector",
index_params=sparse_index_params
)
logger.success("向量索引创建成功")
except Exception as e:
logger.warning(f"创建索引时出错: {e}")
def load_collection(self):
"""加载Collection到内存"""
if self.collection:
self.collection.load()
logger.info(f"Collection已加载: {self.collection_name}")
def release_collection(self):
"""释放Collection内存"""
if self.collection:
self.collection.release()
logger.info(f"Collection已释放: {self.collection_name}")
def insert_chunks(
self,
chunks: List[TextChunk],
embeddings: EmbeddingResult
) -> List[int]:
"""插入文档分块和嵌入向量"""
if not self.collection:
logger.warning("Collection未初始化")
return []
if len(chunks) != len(embeddings.texts):
logger.warning(f"Chunks数量与嵌入数量不匹配")
return []
logger.info(f"准备插入{len(chunks)}个文档分块")
try:
data = []
current_time = int(time.time())
for chunk, dense_emb, sparse_emb in zip(
chunks,
embeddings.dense_embeddings,
embeddings.sparse_embeddings
):
row = {
"doc_id": chunk.metadata.doc_id,
"chunk_id": chunk.metadata.chunk_id,
"content": chunk.content,
"dense_vector": dense_emb.tolist(),
"sparse_vector": sparse_emb,
"doc_name": chunk.metadata.doc_name,
"section_title": chunk.metadata.section_title,
"clause_number": chunk.metadata.clause_number,
"page_number": chunk.metadata.page_number,
"regulation_type": chunk.metadata.regulation_type,
"version": chunk.metadata.version,
"create_time": current_time
}
data.append(row)
result = self.collection.insert(data)
self.collection.flush()
logger.success(f"插入完成，共{len(result.primary_keys)}条记录")
return result.primary_keys
except Exception as e:
logger.error(f"插入数据失败: {e}")
return []
def hybrid_search(
self,
query_dense: List[float],
query_sparse: Dict[int, float],
top_k: int = 10,
filters: Optional[str] = None
) -> List[SearchResult]:
"""混合检索：Dense + Sparse"""
if not self.collection:
logger.warning("Collection未初始化")
return []
try:
self.collection.load()
# 使用简单的Dense检索（兼容所有版本）
dense_results = self.dense_search(query_dense, top_k, filters)
# 可选：合并Sparse结果
if query_sparse:
sparse_results = self.sparse_search(query_sparse, top_k, filters)
merged = self._merge_results(dense_results, sparse_results, top_k)
logger.success(f"混合检索完成，返回{len(merged)}条结果")
return merged
return dense_results
except Exception as e:
logger.error(f"混合检索失败: {e}")
return []
def _merge_results(
self,
dense_results: List[SearchResult],
sparse_results: List[SearchResult],
top_k: int,
dense_weight: float = 0.6
) -> List[SearchResult]:
"""手动融合Dense和Sparse结果"""
sparse_weight = 1 - dense_weight
merged_dict = {}
for r in dense_results:
merged_dict[r.id] = {
"result": r,
"dense_score": r.score * dense_weight,
"sparse_score": 0
}
for r in sparse_results:
if r.id in merged_dict:
merged_dict[r.id]["sparse_score"] = r.score * sparse_weight
else:
merged_dict[r.id] = {
"result": r,
"dense_score": 0,
"sparse_score": r.score * sparse_weight
}
final_results = []
for id_, data in merged_dict.items():
result = data["result"]
final_score = data["dense_score"] + data["sparse_score"]
final_results.append(SearchResult(
id=result.id,
content=result.content,
score=final_score,
metadata=result.metadata
))
final_results.sort(key=lambda x: x.score, reverse=True)
return final_results[:top_k]
def dense_search(
self,
query_dense: List[float],
top_k: int = 10,
filters: Optional[str] = None
) -> List[SearchResult]:
"""纯Dense向量检索"""
if not self.collection:
return []
try:
self.collection.load()
search_params = {
"metric_type": "COSINE",
"params": {"nprobe": 16}
}
results = self.collection.search(
data=[query_dense],
anns_field="dense_vector",
param=search_params,
limit=top_k,
filter=filters,
output_fields=[
"doc_id", "chunk_id", "content",
"doc_name", "section_title", "clause_number",
"page_number", "regulation_type", "version"
]
)
search_results = []
for hits in results:
for hit in hits:
result = SearchResult(
id=hit.id,
content=hit.entity.get("content", ""),
score=hit.score,
metadata={
"doc_id": hit.entity.get("doc_id", ""),
"chunk_id": hit.entity.get("chunk_id", ""),
"doc_name": hit.entity.get("doc_name", ""),
"section_title": hit.entity.get("section_title", ""),
"clause_number": hit.entity.get("clause_number", ""),
"page_number": hit.entity.get("page_number", 0),
"regulation_type": hit.entity.get("regulation_type", ""),
"version": hit.entity.get("version", ""),
}
)
search_results.append(result)
return search_results
except Exception as e:
logger.error(f"Dense检索失败: {e}")
return []
def sparse_search(
self,
query_sparse: Dict[int, float],
top_k: int = 10,
filters: Optional[str] = None
) -> List[SearchResult]:
"""纯Sparse向量检索"""
if not self.collection:
return []
try:
self.collection.load()
search_params = {
"metric_type": "IP",
"params": {"drop_ratio_search": 0.2}
}
results = self.collection.search(
data=[query_sparse],
anns_field="sparse_vector",
param=search_params,
limit=top_k,
filter=filters,
output_fields=[
"doc_id", "chunk_id", "content",
"doc_name", "section_title", "clause_number",
"page_number", "regulation_type", "version"
]
)
search_results = []
for hits in results:
for hit in hits:
result = SearchResult(
id=hit.id,
content=hit.entity.get("content", ""),
score=hit.score,
metadata={
"doc_id": hit.entity.get("doc_id", ""),
"chunk_id": hit.entity.get("chunk_id", ""),
"doc_name": hit.entity.get("doc_name", ""),
"section_title": hit.entity.get("section_title", ""),
"clause_number": hit.entity.get("clause_number", ""),
"page_number": hit.entity.get("page_number", 0),
"regulation_type": hit.entity.get("regulation_type", ""),
"version": hit.entity.get("version", ""),
}
)
search_results.append(result)
return search_results
except Exception as e:
logger.error(f"Sparse检索失败: {e}")
return []
def delete_by_doc_id(self, doc_id: str) -> int:
"""根据doc_id删除记录"""
if not self.collection:
return 0
try:
expr = f'doc_id=="{doc_id}"'
result = self.collection.delete(expr)
logger.info(f"删除记录: doc_id={doc_id}, 数量={len(result.primary_keys)}")
return len(result.primary_keys)
except Exception as e:
logger.error(f"删除失败: {e}")
return 0
def get_collection_stats(self) -> Dict[str, Any]:
"""获取Collection统计信息"""
if not self.collection:
return {}
try:
stats = {
"name": self.collection_name,
"num_entities": self.collection.num_entities,
"description": self.collection.description,
}
return stats
except Exception as e:
logger.warning(f"获取统计信息失败: {e}")
return {}
def create_milvus_client() -> MilvusClient:
"""便捷函数：创建Milvus客户端"""
client = MilvusClient()
client.connect()
client.create_collection(recreate=False)
return client
def insert_documents(
client: MilvusClient,
chunks: List[TextChunk],
embeddings: EmbeddingResult
) -> List[int]:
"""便捷函数：插入文档"""
return client.insert_chunks(chunks, embeddings)
def search_regulations(
client: MilvusClient,
query_dense: List[float],
query_sparse: Dict[int, float],
top_k: int = 10
) -> List[SearchResult]:
"""便捷函数：检索法规"""
return client.hybrid_search(query_dense, query_sparse, top_k)

352
src/services/storage/minio_client.py
View File

@@ -1,352 +0,0 @@
# src/services/storage/minio_client.py
"""MinIO对象存储客户端 - 文档文件存储"""
from minio import Minio
from minio.error import S3Error
from typing import Optional, Dict, Any
from loguru import logger
from io import BytesIO
import os
from src.config.settings import settings
class MinIOClient:
"""MinIO对象存储客户端"""
def __init__(
self,
endpoint: str = None,
access_key: str = None,
secret_key: str = None,
bucket: str = None,
secure: bool = None
):
"""
初始化MinIO客户端
Args:
endpoint: MinIO服务地址
access_key: 访问密钥
secret_key: 秘密密钥
bucket: 存储桶名称
secure: 是否使用HTTPS
"""
self.endpoint = endpoint or settings.minio_endpoint
self.access_key = access_key or settings.minio_access_key
self.secret_key = secret_key or settings.minio_secret_key
self.bucket = bucket or settings.minio_bucket
self.secure = secure or settings.minio_secure
self.client: Optional[Minio] = None
self.connected = False
logger.info(f"MinIO客户端配置: {self.endpoint}, bucket={self.bucket}")
def connect(self) -> bool:
"""连接MinIO服务"""
try:
self.client = Minio(
self.endpoint,
access_key=self.access_key,
secret_key=self.secret_key,
secure=self.secure
)
self.connected = True
logger.success(f"MinIO连接成功: {self.endpoint}")
return True
except Exception as e:
logger.error(f"MinIO连接失败: {e}")
self.connected = False
return False
def ensure_bucket(self) -> bool:
"""确保存储桶存在"""
if not self.connected:
logger.warning("未连接MinIO，请先调用connect()")
return False
try:
if not self.client.bucket_exists(self.bucket):
self.client.make_bucket(self.bucket)
logger.success(f"创建存储桶: {self.bucket}")
else:
logger.info(f"存储桶已存在: {self.bucket}")
return True
except S3Error as e:
logger.error(f"存储桶操作失败: {e}")
return False
def upload_file(
self,
file_path: str,
object_name: str,
metadata: Dict[str, Any] = None
) -> bool:
"""
上传本地文件到MinIO
Args:
file_path: 本地文件路径
object_name: MinIO对象名称
metadata: 元数据
Returns:
bool: 是否成功
"""
if not self.connected:
self.connect()
self.ensure_bucket()
try:
file_size = os.stat(file_path).st_size
content_type = self._get_content_type(file_path)
with open(file_path, 'rb') as f:
self.client.put_object(
self.bucket,
object_name,
f,
file_size,
content_type=content_type,
metadata=metadata
)
logger.success(f"文件上传成功: {object_name}, 大小={file_size}")
return True
except S3Error as e:
logger.error(f"文件上传失败: {e}")
return False
def upload_bytes(
self,
data: bytes,
object_name: str,
content_type: str = "application/octet-stream",
metadata: Dict[str, Any] = None
) -> bool:
"""
上传字节数据到MinIO
Args:
data: 文件字节数据
object_name: MinIO对象名称
content_type: 内容类型
metadata: 元数据（注意：MinIO仅支持US-ASCII字符）
Returns:
bool: 是否成功
"""
if not self.connected:
self.connect()
self.ensure_bucket()
try:
data_stream = BytesIO(data)
# 处理metadata：仅保留ASCII安全字符
safe_metadata = None
if metadata:
safe_metadata = {}
for key, value in metadata.items():
if isinstance(value, str):
# 只保留ASCII字符或转换为安全格式
try:
value.encode('ascii')
safe_metadata[key] = value
except UnicodeEncodeError:
# 中文字符跳过或用占位符
safe_metadata[key] = ""
else:
safe_metadata[key] = str(value)
self.client.put_object(
self.bucket,
object_name,
data_stream,
len(data),
content_type=content_type,
metadata=safe_metadata
)
logger.success(f"数据上传成功: {object_name}, 大小={len(data)}")
return True
except S3Error as e:
logger.error(f"数据上传失败: {e}")
return False
def download_file(
self,
object_name: str,
file_path: str
) -> bool:
"""
从MinIO下载文件到本地
Args:
object_name: MinIO对象名称
file_path: 本地保存路径
Returns:
bool: 是否成功
"""
if not self.connected:
self.connect()
try:
self.client.fget_object(
self.bucket,
object_name,
file_path
)
logger.success(f"文件下载成功: {object_name} -> {file_path}")
return True
except S3Error as e:
logger.error(f"文件下载失败: {e}")
return False
def get_object_url(
self,
object_name: str,
expires: int = 3600
) -> Optional[str]:
"""
获取对象下载URL（临时URL）
Args:
object_name: MinIO对象名称
expires: URL有效期（秒）
Returns:
str: 下载URL
"""
if not self.connected:
self.connect()
try:
url = self.client.presigned_get_object(
self.bucket,
object_name,
expires=expires
)
return url
except S3Error as e:
logger.error(f"获取URL失败: {e}")
return None
def get_object_data(self, object_name: str) -> Optional[bytes]:
"""
获取对象数据（字节）
Args:
object_name: MinIO对象名称
Returns:
bytes: 文件数据
"""
if not self.connected:
self.connect()
try:
response = self.client.get_object(self.bucket, object_name)
data = response.read()
response.close()
response.release_conn()
return data
except S3Error as e:
logger.error(f"获取对象数据失败: {e}")
return None
def delete_object(self, object_name: str) -> bool:
"""
删除对象
Args:
object_name: MinIO对象名称
Returns:
bool: 是否成功
"""
if not self.connected:
self.connect()
try:
self.client.remove_object(self.bucket, object_name)
logger.info(f"对象删除成功: {object_name}")
return True
except S3Error as e:
logger.error(f"对象删除失败: {e}")
return False
def list_objects(self, prefix: str = "") -> list:
"""
列出存储桶中的对象
Args:
prefix: 对象名称前缀
Returns:
list: 对象列表
"""
if not self.connected:
self.connect()
try:
objects = self.client.list_objects(self.bucket, prefix=prefix)
return [obj.object_name for obj in objects]
except S3Error as e:
logger.error(f"列出对象失败: {e}")
return []
def object_exists(self, object_name: str) -> bool:
"""
检查对象是否存在
Args:
object_name: MinIO对象名称
Returns:
bool: 是否存在
"""
if not self.connected:
self.connect()
try:
self.client.stat_object(self.bucket, object_name)
return True
except S3Error:
return False
def _get_content_type(self, file_path: str) -> str:
"""根据文件扩展名获取Content-Type"""
ext = os.path.splitext(file_path)[1].lower()
content_types = {
'.pdf': 'application/pdf',
'.doc': 'application/msword',
'.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
'.txt': 'text/plain',
'.json': 'application/json',
'.xml': 'application/xml',
}
return content_types.get(ext, 'application/octet-stream')
def close(self):
"""关闭连接（MinIO客户端无需显式关闭）"""
self.connected = False
logger.info("MinIO客户端已关闭")
def create_minio_client() -> MinIOClient:
"""便捷函数：创建MinIO客户端"""
client = MinIOClient()
client.connect()
client.ensure_bucket()
return client

2
src/workers/__init__.py
View File

@@ -1,2 +0,0 @@
# src/workers/__init__.py
"""异步任务Worker模块"""

217
start_all.sh
View File

@@ -1,217 +0,0 @@
#!/bin/bash
# start_all.sh - 整合启动脚本（API + 前端）
# 前端使用 Vite 开发模式 (npm run dev)
set -e
VENV_DIR=".venv"
LOG_DIR="logs"
PID_FILE_API="$LOG_DIR/api.pid"
PID_FILE_FRONTEND="$LOG_DIR/frontend.pid"
LOG_FILE_API="$LOG_DIR/api.log"
# 创建日志目录
mkdir -p $LOG_DIR
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m'
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} AI+合规智能中枢 - 服务启动${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
# ===== 环境检查 =====
# 检查虚拟环境
if [ ! -d "$VENV_DIR" ]; then
echo -e "${RED}错误: 虚拟环境不存在${NC}"
echo "请先运行: ./quick_start.sh"
exit 1
fi
# 激活虚拟环境
source $VENV_DIR/bin/activate
echo -e "${GREEN}✓ 虚拟环境已激活: $VENV_DIR${NC}"
# 检查.env文件
if [ ! -f ".env" ]; then
echo -e "${YELLOW}警告: .env文件不存在，使用默认配置${NC}"
else
echo -e "${GREEN}✓ 配置文件已加载: .env${NC}"
fi
# 加载环境变量
export $(grep -v '^#' .env | xargs 2>/dev/null || true)
# 启动参数
API_HOST=${API_HOST:-0.0.0.0}
API_PORT=${API_PORT:-8000}
FRONTEND_PORT=${FRONTEND_PORT:-3000}
echo ""
# ===== 启动API服务 =====
echo -e "${YELLOW}[1/2] 启动API服务...${NC}"
# 检查是否已运行
if [ -f "$PID_FILE_API" ]; then
OLD_PID=$(cat $PID_FILE_API)
if ps -p $OLD_PID > /dev/null 2>&1; then
echo -e "${YELLOW}API服务已在运行 (PID: $OLD_PID)${NC}"
else
rm -f $PID_FILE_API
fi
fi
# 启动API（后台模式）
if [ ! -f "$PID_FILE_API" ]; then
nohup $VENV_DIR/bin/python -m uvicorn src.api.main:app \
--host $API_HOST --port $API_PORT \
> $LOG_FILE_API 2>&1 &
API_PID=$!
echo $API_PID > $PID_FILE_API
sleep 2
if ps -p $API_PID > /dev/null 2>&1; then
echo -e "${GREEN}✓ API服务启动成功 (PID: $API_PID)${NC}"
else
echo -e "${RED}✗ API服务启动失败${NC}"
echo "请查看日志: $LOG_FILE_API"
exit 1
fi
fi
echo ""
# ===== 启动前端服务 =====
echo -e "${YELLOW}[2/2] 启动前端服务...${NC}"
# 检查前端目录
FRONTEND_DIR="frontend"
if [ ! -d "$FRONTEND_DIR" ]; then
echo -e "${RED}错误: 前端目录不存在${NC}"
exit 1
fi
# 检查是否已运行
if [ -f "$PID_FILE_FRONTEND" ]; then
OLD_PID=$(cat $PID_FILE_FRONTEND)
if ps -p $OLD_PID > /dev/null 2>&1; then
echo -e "${YELLOW}前端服务已在运行 (PID: $OLD_PID)${NC}"
else
rm -f $PID_FILE_FRONTEND
fi
fi
# 启动前端（后台模式）
if [ ! -f "$PID_FILE_FRONTEND" ]; then
cd $FRONTEND_DIR
# 检查Node版本
NODE_VERSION=$(node -v 2>/dev/null || echo "v0")
echo -e "${CYAN}Node版本: $NODE_VERSION${NC}"
# 检查node_modules是否存在
if [ ! -d "node_modules" ]; then
echo -e "${YELLOW}首次启动，安装前端依赖...${NC}"
npm install
if [ $? -ne 0 ]; then
echo -e "${RED}✗ 前端依赖安装失败${NC}"
echo "请手动安装: cd frontend && npm install"
exit 1
fi
echo -e "${GREEN}✓ 前端依赖安装完成${NC}"
else
echo -e "${GREEN}✓ 前端依赖已存在${NC}"
# 检查vite是否已安装
if [ ! -d "node_modules/vite" ]; then
echo -e "${YELLOW}vite未安装，重新安装依赖...${NC}"
npm install
if [ $? -ne 0 ]; then
echo -e "${RED}✗ 前端依赖安装失败${NC}"
exit 1
fi
fi
fi
# 使用 npx vite 启动（确保能找到 vite 命令）
echo -e "${CYAN}使用 Vite 开发模式 (端口: $FRONTEND_PORT)${NC}"
nohup npx vite --host 0.0.0.0 --port $FRONTEND_PORT > ../$LOG_DIR/frontend.log 2>&1 &
FRONTEND_PID=$!
cd ..
echo $FRONTEND_PID > $PID_FILE_FRONTEND
sleep 3
# 检查进程
if ps -p $FRONTEND_PID > /dev/null 2>&1; then
echo -e "${GREEN}✓ 前端服务启动成功 (PID: $FRONTEND_PID)${NC}"
else
# Vite 可能启动较慢，等待更长时间再检查
echo -e "${YELLOW}等待 Vite 启动完成...${NC}"
sleep 5
# 检查是否有 vite 进程
VITE_PID=$(pgrep -f "vite" 2>/dev/null || true)
if [ -n "$VITE_PID" ]; then
echo $VITE_PID > $PID_FILE_FRONTEND
echo -e "${GREEN}✓ Vite 已启动 (PID: $VITE_PID)${NC}"
else
echo -e "${RED}✗ 前端服务启动失败${NC}"
echo "请查看日志: $LOG_DIR/frontend.log"
echo ""
echo "常见问题排查:"
echo " 1. Node版本过低: 需要 Node 20+"
echo " 当前版本: $NODE_VERSION"
echo " 升级命令: curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash - && sudo yum install -y nodejs"
echo ""
echo " 2. 查看详细错误:"
echo " cat $LOG_DIR/frontend.log"
exit 1
fi
fi
fi
echo ""
# ===== 输出访问地址 =====
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} 服务已启动${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
echo -e "${GREEN}API服务:${NC}"
echo " 地址: http://localhost:$API_PORT"
echo " 文档: http://localhost:$API_PORT/docs"
echo " 健康检查: http://localhost:$API_PORT/health"
echo ""
echo -e "${GREEN}前端测试页面:${NC}"
echo " 地址: http://localhost:$FRONTEND_PORT"
echo " 模式: Vite 开发模式"
echo ""
echo -e "${GREEN}日志文件:${NC}"
echo " API: $LOG_FILE_API"
echo " 前端: $LOG_DIR/frontend.log"
echo ""
echo -e "${YELLOW}管理命令:${NC}"
echo " 查看状态: ./status.sh"
echo " 查看日志: tail -f $LOG_DIR/frontend.log"
echo " 停止服务: ./stop_all.sh"
echo " 重启服务: ./restart_all.sh"
echo ""

41
start_api.sh
View File

@@ -1,41 +0,0 @@
#!/bin/bash
# start_api.sh - 启动迁移后的 backend API 服务
set -e
VENV_DIR=".venv"
BACKEND_PATH="$PWD/backend"
mkdir -p logs
echo "========================================"
echo "启动 AI+合规智能中枢 API 服务"
echo "========================================"
echo ""
if [ ! -d "$VENV_DIR" ]; then
echo "错误: 虚拟环境不存在，请先运行 ./quick_start.sh"
exit 1
fi
source "$VENV_DIR/bin/activate"
echo "已激活虚拟环境: $VENV_DIR"
echo ""
if [ ! -f ".env" ]; then
echo "警告: 根目录 .env 不存在，使用默认配置"
fi
HOST=${API_HOST:-0.0.0.0}
PORT=${API_PORT:-8000}
export PYTHONPATH="$BACKEND_PATH${PYTHONPATH:+:$PYTHONPATH}"
echo "API地址: http://$HOST:$PORT"
echo "API文档: http://$HOST:$PORT/docs"
echo "健康检查: http://$HOST:$PORT/health"
echo ""
echo "正在启动..."
echo ""
python -m uvicorn app.main:app --host "$HOST" --port "$PORT" --reload

67
start_api_background.sh
View File

@@ -1,67 +0,0 @@
#!/bin/bash
# start_api_background.sh - 后台启动迁移后的 backend API 服务
set -e
VENV_DIR=".venv"
BACKEND_PATH="$PWD/backend"
mkdir -p logs
PID_FILE=logs/api.pid
LOG_FILE=logs/api.log
echo "========================================"
echo "后台启动 AI+合规智能中枢 API 服务"
echo "========================================"
echo ""
if [ ! -d "$VENV_DIR" ]; then
echo "错误: 虚拟环境不存在，请先运行 ./quick_start.sh"
exit 1
fi
if [ -f "$PID_FILE" ]; then
PID=$(cat "$PID_FILE")
if ps -p "$PID" > /dev/null 2>&1; then
echo "服务已在运行 (PID: $PID)"
echo "如需重启，请先运行: ./stop_api.sh"
exit 1
else
rm -f "$PID_FILE"
fi
fi
HOST=${API_HOST:-0.0.0.0}
PORT=${API_PORT:-8000}
echo "服务地址: http://$HOST:$PORT"
echo "日志文件: $LOG_FILE"
echo ""
echo "正在后台启动..."
PYTHONPATH="$BACKEND_PATH${PYTHONPATH:+:$PYTHONPATH}" \
nohup "$VENV_DIR/bin/python" -m uvicorn app.main:app --host "$HOST" --port "$PORT" > "$LOG_FILE" 2>&1 &
PID=$!
echo "$PID" > "$PID_FILE"
sleep 3
if ps -p "$PID" > /dev/null 2>&1; then
echo "服务启动成功 (PID: $PID)"
echo ""
echo "API地址: http://$HOST:$PORT"
echo "API文档: http://$HOST:$PORT/docs"
echo "健康检查: http://$HOST:$PORT/health"
echo ""
echo "查看日志:"
echo " tail -f $LOG_FILE"
echo ""
echo "停止服务:"
echo " ./stop_api.sh"
else
echo "服务启动失败，请查看日志: $LOG_FILE"
rm -f "$PID_FILE"
exit 1
fi

47
start_frontend.sh
View File

@@ -1,47 +0,0 @@
#!/bin/bash
# start_frontend.sh - 启动前端测试页面静态服务
set -e
FRONTEND_DIR="frontend"
PORT=${FRONTEND_PORT:-3000}
echo "========================================"
echo "启动前端测试页面服务"
echo "========================================"
echo ""
# 检查前端目录
if [ ! -d "$FRONTEND_DIR" ]; then
echo "错误: 前端目录不存在: $FRONTEND_DIR"
exit 1
fi
# 检查index.html (React + T-Systems风格前端)
if [ ! -f "$FRONTEND_DIR/index.html" ]; then
echo "错误: 前端页面不存在: $FRONTEND_DIR/index.html"
exit 1
fi
echo "前端页面: index.html (React + T-Systems风格)"
echo "前端地址: http://localhost:$PORT"
echo ""
echo "确保API服务已启动:"
echo " ./start_api.sh"
echo ""
echo "正在启动前端服务..."
# 使用Python内置HTTP服务器
cd $FRONTEND_DIR
# 查找Python命令
if command -v python3 &> /dev/null; then
PYTHON_CMD=python3
elif command -v python &> /dev/null; then
PYTHON_CMD=python
else
echo "错误: 未找到Python"
exit 1
fi
$PYTHON_CMD -m http.server $PORT --bind 0.0.0.0

157
status.sh
View File

@@ -1,157 +0,0 @@
#!/bin/bash
# status.sh - 查看服务运行状态
# 支持: Vite开发模式 或 预构建静态服务
set -e
LOG_DIR="logs"
PID_FILE_API="$LOG_DIR/api.pid"
PID_FILE_FRONTEND="$LOG_DIR/frontend.pid"
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m'
# 加载配置
if [ -f ".env" ]; then
export $(grep -v '^#' .env | grep -E '^(API_PORT|FRONTEND_PORT|FRONTEND_MODE)' | xargs 2>/dev/null || true)
fi
API_PORT=${API_PORT:-8000}
FRONTEND_PORT=${FRONTEND_PORT:-5173}
FRONTEND_MODE=${FRONTEND_MODE:-dev}
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} AI+合规智能中枢 - 服务状态${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
# ===== API状态 =====
echo -e "${YELLOW}API服务:${NC}"
API_RUNNING=false
if [ -f "$PID_FILE_API" ]; then
PID=$(cat $PID_FILE_API)
if ps -p $PID > /dev/null 2>&1; then
API_RUNNING=true
echo -e " 状态: ${GREEN}运行中 ✓${NC}"
echo " PID: $PID"
# 检查端口
if command -v curl > /dev/null 2>&1; then
HEALTH=$(curl -s http://localhost:$API_PORT/health 2>/dev/null || echo '{"status":"error"}')
if echo "$HEALTH" | grep -q "healthy"; then
echo -e " 健康检查: ${GREEN}正常 ✓${NC}"
else
echo -e " 健康检查: ${RED}异常${NC}"
fi
fi
else
echo -e " 状态: ${RED}已停止${NC} (PID文件存在但进程不存在)"
rm -f $PID_FILE_API
fi
else
# 尝试查找uvicorn进程
UVICORN_PID=$(pgrep -f "uvicorn app.main:app" 2>/dev/null || true)
if [ -n "$UVICORN_PID" ]; then
echo -e " 状态: ${GREEN}运行中 ✓${NC} (无PID文件)"
echo " PID: $UVICORN_PID"
API_RUNNING=true
else
echo -e " 状态: ${RED}已停止${NC}"
fi
fi
echo " 地址: http://localhost:$API_PORT"
echo " 文档: http://localhost:$API_PORT/docs"
echo ""
# ===== 前端状态 =====
echo -e "${YELLOW}前端服务 (模式: $FRONTEND_MODE):${NC}"
FRONTEND_RUNNING=false
if [ -f "$PID_FILE_FRONTEND" ]; then
PID=$(cat $PID_FILE_FRONTEND)
if ps -p $PID > /dev/null 2>&1; then
FRONTEND_RUNNING=true
echo -e " 状态: ${GREEN}运行中 ✓${NC}"
echo " PID: $PID"
# 显示前端模式
if [ "$FRONTEND_MODE" = "dev" ]; then
echo -e " 模式: ${CYAN}Vite开发模式${NC}"
else
echo -e " 模式: ${CYAN}预构建静态服务${NC}"
fi
else
echo -e " 状态: ${RED}已停止${NC} (PID文件存在但进程不存在)"
rm -f $PID_FILE_FRONTEND
fi
else
# 查找Vite/npm进程
VITE_PID=$(pgrep -f "vite" 2>/dev/null || true)
HTTP_PID=$(pgrep -f "http.server $FRONTEND_PORT" 2>/dev/null || true)
if [ -n "$VITE_PID" ]; then
echo -e " 状态: ${GREEN}运行中 ✓${NC} (无PID文件)"
echo " PID: $VITE_PID (Vite)"
FRONTEND_RUNNING=true
elif [ -n "$HTTP_PID" ]; then
echo -e " 状态: ${GREEN}运行中 ✓${NC} (无PID文件)"
echo " PID: $HTTP_PID (静态)"
FRONTEND_RUNNING=true
else
echo -e " 状态: ${RED}已停止${NC}"
fi
fi
echo " 地址: http://localhost:$FRONTEND_PORT"
echo ""
# ===== Docker容器状态 =====
echo -e "${YELLOW}Docker服务:${NC}"
if command -v docker > /dev/null 2>&1; then
CONTAINERS="milvus minio redis postgres"
for container in $CONTAINERS; do
if docker ps --format '{{.Names}}' | grep -q "^${container}$"; then
echo -e " $container: ${GREEN}运行中 ✓${NC}"
elif docker ps -a --format '{{.Names}}' | grep -q "^${container}$"; then
echo -e " $container: ${RED}已停止${NC}"
else
echo -e " $container: ${YELLOW}不存在${NC}"
fi
done
else
echo -e " ${YELLOW}Docker未安装${NC}"
fi
echo ""
# ===== 总结 =====
echo -e "${CYAN}========================================${NC}"
if [ "$API_RUNNING" = true ] && [ "$FRONTEND_RUNNING" = true ]; then
echo -e "${GREEN} 所有服务正常运行 ✓${NC}"
else
if [ "$API_RUNNING" = false ]; then
echo -e "${RED} API服务未运行${NC}"
fi
if [ "$FRONTEND_RUNNING" = false ]; then
echo -e "${RED} 前端服务未运行${NC}"
fi
echo ""
echo -e "${YELLOW}启动服务: ./start_all.sh${NC}"
fi
echo -e "${CYAN}========================================${NC}"
echo ""

160
stop_all.sh
View File

@@ -1,160 +0,0 @@
#!/bin/bash
# stop_all.sh - 停止所有服务（API + 前端）
# 支持停止 Vite 开发服务器
set -e
LOG_DIR="logs"
PID_FILE_API="$LOG_DIR/api.pid"
PID_FILE_FRONTEND="$LOG_DIR/frontend.pid"
# 颜色定义
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m'
# 加载配置
if [ -f ".env" ]; then
export $(grep -v '^#' .env | grep -E '^FRONTEND_PORT' | xargs 2>/dev/null || true)
fi
FRONTEND_PORT=${FRONTEND_PORT:-3000}
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${CYAN} AI+合规智能中枢 - 停止服务${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""
# ===== 停止API服务 =====
echo -e "${YELLOW}[1/2] 停止API服务...${NC}"
if [ -f "$PID_FILE_API" ]; then
PID=$(cat $PID_FILE_API)
if ps -p $PID > /dev/null 2>&1; then
echo "正在停止API服务 (PID: $PID)..."
kill $PID
sleep 2
if ps -p $PID > /dev/null 2>&1; then
echo "进程未响应，强制终止..."
kill -9 $PID
fi
rm -f $PID_FILE_API
echo -e "${GREEN}✓ API服务已停止${NC}"
else
echo "进程已不存在，清理PID文件"
rm -f $PID_FILE_API
fi
else
echo -e "${YELLOW}API PID文件不存在${NC}"
# 尝试查找并停止所有uvicorn进程
UVICORN_PIDS=$(pgrep -f "uvicorn src.api.main" 2>/dev/null || true)
if [ -n "$UVICORN_PIDS" ]; then
echo "发现运行中的uvicorn进程: $UVICORN_PIDS"
kill $UVICORN_PIDS 2>/dev/null || true
echo -e "${GREEN}✓ 已停止uvicorn进程${NC}"
else
echo -e "${YELLOW}未发现运行中的API服务${NC}"
fi
fi
echo ""
# ===== 停止前端服务 =====
echo -e "${YELLOW}[2/2] 停止前端服务...${NC}"
STOPPED=false
# 方法1: 通过PID文件停止
if [ -f "$PID_FILE_FRONTEND" ]; then
PID=$(cat $PID_FILE_FRONTEND)
if ps -p $PID > /dev/null 2>&1; then
echo "正在停止前端服务 (PID: $PID)..."
kill $PID
sleep 2
if ps -p $PID > /dev/null 2>&1; then
echo "进程未响应，强制终止..."
kill -9 $PID
fi
rm -f $PID_FILE_FRONTEND
echo -e "${GREEN}✓ 前端服务已停止${NC}"
STOPPED=true
else
echo "进程已不存在，清理PID文件"
rm -f $PID_FILE_FRONTEND
fi
fi
# 方法2: 通过进程名查找并停止（如果PID文件方式未成功）
if [ "$STOPPED" = false ]; then
# 查找 Vite 进程
VITE_PIDS=$(pgrep -f "vite" 2>/dev/null || true)
if [ -n "$VITE_PIDS" ]; then
echo "发现运行中的Vite进程: $VITE_PIDS"
kill $VITE_PIDS 2>/dev/null || true
sleep 1
# 检查是否还有残留进程
VITE_REMAIN=$(pgrep -f "vite" 2>/dev/null || true)
if [ -n "$VITE_REMAIN" ]; then
echo "强制终止残留进程..."
kill -9 $VITE_REMAIN 2>/dev/null || true
fi
echo -e "${GREEN}✓ 已停止Vite进程${NC}"
STOPPED=true
fi
# 查找 npm/node 进程（启动Vite的父进程）
NPM_PIDS=$(pgrep -f "npm run dev" 2>/dev/null || true)
if [ -n "$NPM_PIDS" ]; then
echo "发现运行中的npm进程: $NPM_PIDS"
kill $NPM_PIDS 2>/dev/null || true
echo -e "${GREEN}✓ 已停止npm进程${NC}"
STOPPED=true
fi
# 查找 node 进程监听前端端口
NODE_PORT_PIDS=$(lsof -ti:$FRONTEND_PORT 2>/dev/null || true)
if [ -n "$NODE_PORT_PIDS" ]; then
echo "发现监听端口 $FRONTEND_PORT 的进程: $NODE_PORT_PIDS"
kill $NODE_PORT_PIDS 2>/dev/null || true
echo -e "${GREEN}✓ 已停止端口监听进程${NC}"
STOPPED=true
fi
# 查找 HTTP 服务器（备用）
HTTP_PIDS=$(pgrep -f "http.server $FRONTEND_PORT" 2>/dev/null || true)
if [ -n "$HTTP_PIDS" ]; then
echo "发现运行中的HTTP服务器: $HTTP_PIDS"
kill $HTTP_PIDS 2>/dev/null || true
echo -e "${GREEN}✓ 已停止HTTP服务${NC}"
STOPPED=true
fi
fi
if [ "$STOPPED" = false ]; then
echo -e "${YELLOW}未发现运行中的前端服务${NC}"
fi
# 清理PID文件
rm -f $PID_FILE_FRONTEND
echo ""
echo -e "${CYAN}========================================${NC}"
echo -e "${GREEN} 所有服务已停止${NC}"
echo -e "${CYAN}========================================${NC}"
echo ""

46
stop_api.sh
View File

@@ -1,46 +0,0 @@
#!/bin/bash
# stop_api.sh - 停止API服务
PID_FILE=logs/api.pid
echo "========================================"
echo "停止 AI+合规智能中枢 API服务"
echo "========================================"
echo ""
if [ -f "$PID_FILE" ]; then
PID=$(cat $PID_FILE)
if ps -p $PID > /dev/null 2>&1; then
echo "正在停止服务 (PID: $PID)..."
kill $PID
# 等待进程结束
sleep 2
if ps -p $PID > /dev/null 2>&1; then
echo "进程未响应，强制终止..."
kill -9 $PID
fi
rm -f $PID_FILE
echo "服务已停止"
else
echo "进程已不存在，清理PID文件"
rm -f $PID_FILE
fi
else
echo "PID文件不存在，服务可能未运行"
# 尝试查找并停止所有uvicorn进程
UVICORN_PIDS=$(pgrep -f "uvicorn app.main:app")
if [ -n "$UVICORN_PIDS" ]; then
echo "发现运行中的uvicorn进程: $UVICORN_PIDS"
echo "是否停止这些进程? (y/n)"
read -r answer
if [ "$answer" = "y" ]; then
kill $UVICORN_PIDS
echo "进程已停止"
fi
fi
fi

Some files were not shown because too many files have changed in this diff Show More