feat(advisor): add optimization advisor module
- rag_eval/advisor/: new package with rules engine, LLM analyzer, writer - rules.py: 7-metric diagnostic rules (warning/critical thresholds, top-3 low samples) - llm_analyzer.py: Chinese optimization report via judge_model, graceful fallback - writer.py: writes optimization_advice.md + log summary - __init__.py: run_advisor() entry point (no-op when optimization_advisor=False) - Scenario.optimization_advisor: new bool field (default False) - ScenarioModel: same field added, loader.py透传 - RunArtifactPaths.advice_md: new path field - factory.py: build_models() now public; build_metric_pipeline() accepts pre-built llm/embeddings - runner.py: lifts llm, passes to pipeline and advisor; calls run_advisor() at end - siemens online YAML: optimization_advisor: true enabled - tests: 9 rules tests + 6 writer tests, all pass - docs: advisor section added to engine-flow.md and architecture.md Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -318,6 +318,10 @@ metrics:
|
||||
- answer_relevancy
|
||||
- context_recall
|
||||
- context_precision
|
||||
# 可选:鲁棒性 / 端到端指标(需数据集含 ground_truth),完整列表见 §9.4
|
||||
# - noise_sensitivity
|
||||
# - factual_correctness
|
||||
# - semantic_similarity
|
||||
output_dir: runs/legal-assistant-offline-baseline
|
||||
runtime:
|
||||
batch_size: 4
|
||||
@@ -338,7 +342,7 @@ runtime:
|
||||
- `embedding_model`
|
||||
- 负责向量相关指标的模型
|
||||
- `metrics`
|
||||
- 本次启用的指标列表
|
||||
- 本次启用的指标列表(完整可选项与依赖见 §9.4)
|
||||
- `output_dir`
|
||||
- 本次运行结果输出目录
|
||||
- `runtime.batch_size`
|
||||
@@ -399,6 +403,32 @@ app_adapter:
|
||||
- embedding model
|
||||
- 指标实例
|
||||
|
||||
当前支持的指标(`rag_eval/metrics/registry.py` 中的 `SUPPORTED_METRICS`):
|
||||
|
||||
| 指标名 | 层面 | 依赖 |
|
||||
|---|---|---|
|
||||
| `faithfulness` | 生成 | judge model |
|
||||
| `answer_relevancy` | 生成 | judge model + embedding |
|
||||
| `context_recall` | 检索 | judge model + ground_truth |
|
||||
| `context_precision` | 检索 | judge model + ground_truth |
|
||||
| `noise_sensitivity` | 鲁棒性 | judge model + ground_truth |
|
||||
| `factual_correctness` | 端到端 | judge model + ground_truth |
|
||||
| `semantic_similarity` | 端到端 | embedding + ground_truth(无 LLM 调用) |
|
||||
|
||||
后四项以 `ground_truth`(标准答案)为参照,数据集必须提供该字段。新增指标统一在 `registry.py` / `factory.py` / `pipeline.py` 三处对齐装配。
|
||||
|
||||
**Optimization Advisor(§11 优化策略落地):**
|
||||
|
||||
评测结束后,若场景配置 `optimization_advisor: true`,则自动调用 `rag_eval/advisor/` 模块:
|
||||
- 规则引擎(`rules.py`)对 7 个指标各自设阈值,识别触发项并选取 top-3 低分样本
|
||||
- LLM 分析器(`llm_analyzer.py`)结合低分样本生成中文 Markdown 优化建议(复用 judge_model,失败自动降级为纯规则报告)
|
||||
- 写出层(`writer.py`)输出 `optimization_advice.md` 并打日志摘要
|
||||
|
||||
```yaml
|
||||
# 场景配置示例
|
||||
optimization_advisor: true
|
||||
```
|
||||
|
||||
### 9.5 并发控制
|
||||
|
||||
执行层负责并发上限,不把并发策略散落到各指标实现中。
|
||||
|
||||
Reference in New Issue
Block a user