Add Siemens CT document evaluation scenario (three-step pipeline)
- scenarios/siemens_build/siemens-pdf-build.yaml: dataset build for all 17 Siemens medical-imaging PDFs (aliyun_docmind parser, 10 questions/doc, failure_mode=skip, ~170 question total) - scenarios/offline/siemens-pdf-offline-smoke.yaml: offline evaluation using source chunks as contexts and ground_truth as answer (up to 30 samples) - scenarios/online/siemens-pdf-question-bank-online.yaml: online evaluation calling siemens_pdf_qa adapter, batch_size=4, up to 50 samples - apps/siemens_pdf_qa/adapter.py: Siemens-specific adapter with bilingual (zh/en) system prompt and strict evidence-grounding for CT domain - scripts/build_siemens_offline_smoke.py: helper to derive offline smoke CSV from completed dataset build artifacts (run after dataset build step) - docs/superpowers/specs/2026-06-15-siemens-scenario-design.md: design spec All three scenarios are automatically discovered by the web console. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
This commit is contained in:
59
docs/superpowers/specs/2026-06-15-siemens-scenario-design.md
Normal file
59
docs/superpowers/specs/2026-06-15-siemens-scenario-design.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Siemens PDF 场景设计 Spec
|
||||
|
||||
- 日期:2026-06-15
|
||||
- 状态:已确认,进入实现。
|
||||
|
||||
## 1. 目标
|
||||
|
||||
基于 `datasets/siemens-pdfs/`(17 个西门子医疗 CT 中文 PDF),跑通完整三步流水线:
|
||||
|
||||
```
|
||||
dataset_build(PDF→题库)→ offline smoke 评估 → online 评估
|
||||
```
|
||||
|
||||
完全镜像现有 `sample-pdf-*` 模式(方案 A),不改动任何现有文件。
|
||||
|
||||
## 2. 参数决策
|
||||
|
||||
| 项目 | 值 |
|
||||
|---|---|
|
||||
| 输入 PDF | `datasets/siemens-pdfs/*.pdf`(17 个) |
|
||||
| failure_mode | `skip`(单个文档解析失败不中断整批) |
|
||||
| max_questions_per_document | 10(共 ~170 题) |
|
||||
| max_source_chunks_per_question | 3 |
|
||||
| generation model | `.env` 的 `DATASET_GENERATOR_MODEL`(qwen3.6-plus) |
|
||||
| judge model | `.env` 的 `RAGAS_JUDGE_MODEL`(deepseek-v4-flash) |
|
||||
| embedding model | `.env` 的 `RAGAS_EMBEDDING_MODEL`(text-embedding-v3) |
|
||||
| online answer model | `.env` 的 `RAGAS_JUDGE_MODEL` |
|
||||
| metrics | faithfulness / answer_relevancy / context_recall / context_precision |
|
||||
|
||||
## 3. 新增文件(4 个)
|
||||
|
||||
```
|
||||
scenarios/siemens_build/siemens-pdf-build.yaml
|
||||
scenarios/offline/siemens-pdf-offline-smoke.yaml
|
||||
scenarios/online/siemens-pdf-question-bank-online.yaml
|
||||
apps/siemens_pdf_qa/__init__.py
|
||||
apps/siemens_pdf_qa/adapter.py
|
||||
```
|
||||
|
||||
加上辅助脚本:
|
||||
```
|
||||
scripts/build_siemens_offline_smoke.py ← 从 build 产物生成 offline smoke CSV
|
||||
```
|
||||
|
||||
## 4. 运行顺序
|
||||
|
||||
```
|
||||
# 步骤 1:dataset build(PDF → 题库草稿 + source_chunks.jsonl)
|
||||
python main.py --dataset-build-config scenarios/siemens_build/siemens-pdf-build.yaml
|
||||
|
||||
# 步骤 2:生成 offline smoke 数据集(一次性脚本,build 跑完后执行)
|
||||
python scripts/build_siemens_offline_smoke.py
|
||||
|
||||
# 步骤 3:offline 评估(用 source chunks 作为 contexts,ground_truth 作为 answer)
|
||||
python main.py --scenario scenarios/offline/siemens-pdf-offline-smoke.yaml
|
||||
|
||||
# 步骤 4:online 评估(实时调用 LLM 生成 answer,再评分)
|
||||
python main.py --scenario scenarios/online/siemens-pdf-question-bank-online.yaml
|
||||
```
|
||||
Reference in New Issue
Block a user