AIRegulation/siemens_ragas
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
45 Commits 1 Branch 0 Tags
5ced129ff7c965cd4ed6aee37acb2bac9d2de161
Commit Graph

5 Commits

Author SHA1 Message Date
wangwei
eee96eb158 docs: add Dify score API integration design spec 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 14:51:52 +08:00
wangwei
199b3af611 docs: add Linux deploy script design spec 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 14:18:14 +08:00
wangwei
ca586bf9bb docs: add metric and doc weights feature design spec 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 16:37:18 +08:00
wangwei
f5c2dce64a feat(advisor): add optimization advisor module 
- rag_eval/advisor/: new package with rules engine, LLM analyzer, writer
  - rules.py: 7-metric diagnostic rules (warning/critical thresholds, top-3 low samples)
  - llm_analyzer.py: Chinese optimization report via judge_model, graceful fallback
  - writer.py: writes optimization_advice.md + log summary
  - __init__.py: run_advisor() entry point (no-op when optimization_advisor=False)
- Scenario.optimization_advisor: new bool field (default False)
- ScenarioModel: same field added, loader.py透传
- RunArtifactPaths.advice_md: new path field
- factory.py: build_models() now public; build_metric_pipeline() accepts pre-built llm/embeddings
- runner.py: lifts llm, passes to pipeline and advisor; calls run_advisor() at end
- siemens online YAML: optimization_advisor: true enabled
- tests: 9 rules tests + 6 writer tests, all pass
- docs: advisor section added to engine-flow.md and architecture.md

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-06-16 17:06:19 +08:00
wangwei
75ae7927ad Add Siemens CT document evaluation scenario (three-step pipeline) 
- scenarios/siemens_build/siemens-pdf-build.yaml: dataset build for all 17
  Siemens medical-imaging PDFs (aliyun_docmind parser, 10 questions/doc,
  failure_mode=skip, ~170 question total)
- scenarios/offline/siemens-pdf-offline-smoke.yaml: offline evaluation using
  source chunks as contexts and ground_truth as answer (up to 30 samples)
- scenarios/online/siemens-pdf-question-bank-online.yaml: online evaluation
  calling siemens_pdf_qa adapter, batch_size=4, up to 50 samples
- apps/siemens_pdf_qa/adapter.py: Siemens-specific adapter with bilingual
  (zh/en) system prompt and strict evidence-grounding for CT domain
- scripts/build_siemens_offline_smoke.py: helper to derive offline smoke CSV
  from completed dataset build artifacts (run after dataset build step)
- docs/superpowers/specs/2026-06-15-siemens-scenario-design.md: design spec

All three scenarios are automatically discovered by the web console.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
 2026-06-15 17:00:52 +08:00