AIRegulation/siemens_ragas
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
63 Commits 1 Branch 0 Tags
4fd515d2d9b8eaa92bcd1bd6aebc39104d0217eb
Commit Graph

18 Commits

Author SHA1 Message Date
wangwei
4fd515d2d9 feat: async score jobs — POST /api/score/async + 评分记录 page 
Each async score job:
- Runs InlineScorer.score() in thread pool
- Writes standard run artifacts (metadata.json, scores.csv, summary.md)
- Runs optimization_advisor => optimization_advice.md
- Result appears in 运行列表 and 报告详情 with full report

New endpoints:
- POST /api/score/async  (202, job_id immediate)
- GET  /api/score/jobs   (list all jobs)
- GET  /api/score/jobs/{id} (single job status)

Frontend:
- 评分记录 nav page with card list
- 5s auto-polling for queued/running jobs
- 查看报告 button navigates to existing 报告详情 page

Dify: change /api/score -> /api/score/async, no response parsing needed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-24 17:24:22 +08:00
wangwei
b870ed8730 feat: make contexts optional in /api/score 
When contexts is absent, metrics that require retrieved_contexts
(faithfulness, context_recall, context_precision, noise_sensitivity)
are automatically skipped and appear in skipped_metrics.
Only answer_relevancy, factual_correctness, semantic_similarity
remain computable without contexts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-24 14:42:03 +08:00
wangwei
a03a24be4e feat: add POST /api/score endpoint for Dify real-time scoring 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 15:14:19 +08:00
wangwei
e4d4e4968b feat: add InlineScorer service with LLM client caching 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 15:03:43 +08:00
wangwei
761faf9c42 feat: add ScoreRequest/ScoreResponse models and SCORE_API_TOKEN setting 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-22 15:00:05 +08:00
wangwei
36e5506e2a feat: report_builder uses weighted means; ReportData gains weighted_score_mean 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 17:16:09 +08:00
wangwei
ce0d2291b0 feat: yaml_patcher and ProfileApplyRequest support metric_weights and doc_weights 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 17:02:21 +08:00
wangwei
480f6d66ea feat: use weighted metric means and add weighted_score row to summary.md 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 16:59:56 +08:00
wangwei
d371ef7d24 feat: add weighted_score and sample_weight columns to score rows 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 16:53:45 +08:00
wangwei
8617eaa5aa feat: add metric_weights and doc_weights to Scenario schema and dataclass 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 16:50:33 +08:00
wangwei
e0b064587f feat: add metric/doc weight computation module (weights.py) 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-18 16:47:47 +08:00
wangwei
f5c2dce64a feat(advisor): add optimization advisor module 
- rag_eval/advisor/: new package with rules engine, LLM analyzer, writer
  - rules.py: 7-metric diagnostic rules (warning/critical thresholds, top-3 low samples)
  - llm_analyzer.py: Chinese optimization report via judge_model, graceful fallback
  - writer.py: writes optimization_advice.md + log summary
  - __init__.py: run_advisor() entry point (no-op when optimization_advisor=False)
- Scenario.optimization_advisor: new bool field (default False)
- ScenarioModel: same field added, loader.py透传
- RunArtifactPaths.advice_md: new path field
- factory.py: build_models() now public; build_metric_pipeline() accepts pre-built llm/embeddings
- runner.py: lifts llm, passes to pipeline and advisor; calls run_advisor() at end
- siemens online YAML: optimization_advisor: true enabled
- tests: 9 rules tests + 6 writer tests, all pass
- docs: advisor section added to engine-flow.md and architecture.md

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-06-16 17:06:19 +08:00
wangwei
719c3b4ca4 test: ensure test package structure and all webapp tests pass 2026-06-16 16:27:54 +08:00
wangwei
e329f59139 feat: add yaml_patcher service to apply LLM profiles to scenario YAML 2026-06-16 16:21:19 +08:00
wangwei
b19054bd66 feat: add /api/llm-profiles CRUD router 2026-06-16 16:18:40 +08:00
wangwei
5d09deb420 feat: add ProfileManager service with JSON persistence 2026-06-16 16:14:31 +08:00
wangwei
b98af29449 feat: add LLMProfile pydantic models 2026-06-16 16:10:37 +08:00
Guangfei.Zhao
9cbdc1d95d first commit 2026-06-12 14:02:15 +08:00