Commit Graph
8 Commits
Author SHA1 Message Date
wangweiandCopilot 6a2bbf8239 feat(token-tracking): capture token usage in /api/score/async job runs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-02 15:03:25 +08:00
wangweiandCopilot 613d167e81 feat(token-tracking): add HTTP response hook and attach_usage_hook, wire into build_models
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-02 14:36:59 +08:00
wangweiandCopilot 24a8688a34 Add committed zh judge-prompt cache + enable in Siemens scenarios + docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-01 18:15:53 +08:00
wangwei abcd61ec8f docs: add async score jobs implementation plan 2026-06-24 17:08:01 +08:00
wangwei 9ad6ad4ebc docs: add Dify score API implementation plan 2026-06-22 14:55:43 +08:00
wangwei 078097af00 docs: add metric/doc weights implementation plan 2026-06-18 16:43:08 +08:00
wangwei 24956bbf75 更新 2026-06-16 18:12:33 +08:00
wangweiandClaude f5c2dce64a feat(advisor): add optimization advisor module
- rag_eval/advisor/: new package with rules engine, LLM analyzer, writer
  - rules.py: 7-metric diagnostic rules (warning/critical thresholds, top-3 low samples)
  - llm_analyzer.py: Chinese optimization report via judge_model, graceful fallback
  - writer.py: writes optimization_advice.md + log summary
  - __init__.py: run_advisor() entry point (no-op when optimization_advisor=False)
- Scenario.optimization_advisor: new bool field (default False)
- ScenarioModel: same field added, loader.py透传
- RunArtifactPaths.advice_md: new path field
- factory.py: build_models() now public; build_metric_pipeline() accepts pre-built llm/embeddings
- runner.py: lifts llm, passes to pipeline and advisor; calls run_advisor() at end
- siemens online YAML: optimization_advisor: true enabled
- tests: 9 rules tests + 6 writer tests, all pass
- docs: advisor section added to engine-flow.md and architecture.md

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-16 17:06:19 +08:00