Add Siemens CT document evaluation scenario (three-step pipeline)
- scenarios/siemens_build/siemens-pdf-build.yaml: dataset build for all 17 Siemens medical-imaging PDFs (aliyun_docmind parser, 10 questions/doc, failure_mode=skip, ~170 question total) - scenarios/offline/siemens-pdf-offline-smoke.yaml: offline evaluation using source chunks as contexts and ground_truth as answer (up to 30 samples) - scenarios/online/siemens-pdf-question-bank-online.yaml: online evaluation calling siemens_pdf_qa adapter, batch_size=4, up to 50 samples - apps/siemens_pdf_qa/adapter.py: Siemens-specific adapter with bilingual (zh/en) system prompt and strict evidence-grounding for CT domain - scripts/build_siemens_offline_smoke.py: helper to derive offline smoke CSV from completed dataset build artifacts (run after dataset build step) - docs/superpowers/specs/2026-06-15-siemens-scenario-design.md: design spec All three scenarios are automatically discovered by the web console. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
This commit is contained in:
6
apps/siemens_pdf_qa/__init__.py
Normal file
6
apps/siemens_pdf_qa/__init__.py
Normal file
@@ -0,0 +1,6 @@
|
||||
"""Siemens PDF question bank adapter for online evaluation.
|
||||
|
||||
Wraps the generic pdf_question_bank adapter with a Siemens-specific system
|
||||
prompt that instructs the model to answer in the same language as the question
|
||||
(Chinese for Chinese CT documentation) and to cite only the provided evidence.
|
||||
"""
|
||||
Reference in New Issue
Block a user