siemens_ragas/scenarios/offline/siemens-pdf-offline-smoke.yaml

scenario_name: siemens-pdf-offline-smoke
mode: offline
app_adapter: null
dataset: ../../datasets/normalized/siemens_pdf_offline_smoke.csv
judge_model: deepseek-v4-flash
embedding_model: text-embedding-v3
metrics:
  - faithfulness
  - answer_relevancy
  - context_recall
  - context_precision
  # 可选：鲁棒性 / 端到端指标（数据集已含 ground_truth，取消注释即可启用）
  # - noise_sensitivity      # 鲁棒性：对检索噪声的敏感度
  # - factual_correctness    # 端到端：事实正确性（相对标准答案）
  # - semantic_similarity    # 端到端：语义相似度（embedding，无 LLM 调用）
output_dir: ../../outputs/siemens-pdf-offline-smoke
runtime:
  batch_size: 4
  max_samples: 30
-												Add Siemens CT document evaluation scenario (three-step pipeline)

- scenarios/siemens_build/siemens-pdf-build.yaml: dataset build for all 17
  Siemens medical-imaging PDFs (aliyun_docmind parser, 10 questions/doc,
  failure_mode=skip, ~170 question total)
- scenarios/offline/siemens-pdf-offline-smoke.yaml: offline evaluation using
  source chunks as contexts and ground_truth as answer (up to 30 samples)
- scenarios/online/siemens-pdf-question-bank-online.yaml: online evaluation
  calling siemens_pdf_qa adapter, batch_size=4, up to 50 samples
- apps/siemens_pdf_qa/adapter.py: Siemens-specific adapter with bilingual
  (zh/en) system prompt and strict evidence-grounding for CT domain
- scripts/build_siemens_offline_smoke.py: helper to derive offline smoke CSV
  from completed dataset build artifacts (run after dataset build step)
- docs/superpowers/specs/2026-06-15-siemens-scenario-design.md: design spec

All three scenarios are automatically discovered by the web console.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

											
										
										
											2026-06-15 17:00:52 +08:00
+								scenario_name: siemens-pdf-offline-smoke
 								mode: offline
 								app_adapter: null
 								dataset: ../../datasets/normalized/siemens_pdf_offline_smoke.csv
 								judge_model: deepseek-v4-flash
 								embedding_model: text-embedding-v3
 								metrics:
 								  - faithfulness
 								  - answer_relevancy
 								  - context_recall
 								  - context_precision
-												更新

											
										
										
											2026-06-16 18:12:33 +08:00
+								  # 可选：鲁棒性 / 端到端指标（数据集已含 ground_truth，取消注释即可启用）
 								  # - noise_sensitivity      # 鲁棒性：对检索噪声的敏感度
 								  # - factual_correctness    # 端到端：事实正确性（相对标准答案）
 								  # - semantic_similarity    # 端到端：语义相似度（embedding，无 LLM 调用）
-												Add Siemens CT document evaluation scenario (three-step pipeline)

- scenarios/siemens_build/siemens-pdf-build.yaml: dataset build for all 17
  Siemens medical-imaging PDFs (aliyun_docmind parser, 10 questions/doc,
  failure_mode=skip, ~170 question total)
- scenarios/offline/siemens-pdf-offline-smoke.yaml: offline evaluation using
  source chunks as contexts and ground_truth as answer (up to 30 samples)
- scenarios/online/siemens-pdf-question-bank-online.yaml: online evaluation
  calling siemens_pdf_qa adapter, batch_size=4, up to 50 samples
- apps/siemens_pdf_qa/adapter.py: Siemens-specific adapter with bilingual
  (zh/en) system prompt and strict evidence-grounding for CT domain
- scripts/build_siemens_offline_smoke.py: helper to derive offline smoke CSV
  from completed dataset build artifacts (run after dataset build step)
- docs/superpowers/specs/2026-06-15-siemens-scenario-design.md: design spec

All three scenarios are automatically discovered by the web console.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

											
										
										
											2026-06-15 17:00:52 +08:00
+								output_dir: ../../outputs/siemens-pdf-offline-smoke
 								runtime:
 								  batch_size: 4
 								  max_samples: 30