wangwei and Copilot
613d167e81
feat(token-tracking): add HTTP response hook and attach_usage_hook, wire into build_models
...
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-02 14:36:59 +08:00
wangwei and Copilot
1dec4c8372
Add INFO/DEBUG logging to factory: log resolved base_url per model
...
build_models now logs: judge=<model>-><url> embedding=<model>-><url>
Makes it easy to confirm which gateway is actually used for each model.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-01 21:24:59 +08:00
wangwei and Copilot
f6e10145cd
Fix: build_models resolves separate AsyncOpenAI clients for judge and embedding models
...
Previously, only the judge_model's profile was looked up and its AsyncOpenAI
client was shared with embedding_factory. When embedding_model has a different
base_url/api_key (e.g. Qwen3-Embedding-4B on SiliconFlow vs gpt-5 on another
gateway), the embedding calls silently used the wrong URL, causing 402/404 errors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-01 21:15:09 +08:00
wangwei and Copilot
bd5658c3ac
Wire judge-prompt localization into factory and inline scorer
...
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-01 17:58:43 +08:00
wangwei and Copilot
2bb804b059
Extract shared build_metric_registry factory (DRY)
...
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-01 17:52:55 +08:00
wangwei
9828b1d44c
update
2026-06-27 14:31:45 +08:00
wangwei and Copilot
1df4010acc
fix(llm): resolve score runtime config from saved profiles
...
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-06-26 20:34:01 +08:00
wangwei and Claude
f5c2dce64a
feat(advisor): add optimization advisor module
...
- rag_eval/advisor/: new package with rules engine, LLM analyzer, writer
- rules.py: 7-metric diagnostic rules (warning/critical thresholds, top-3 low samples)
- llm_analyzer.py: Chinese optimization report via judge_model, graceful fallback
- writer.py: writes optimization_advice.md + log summary
- __init__.py: run_advisor() entry point (no-op when optimization_advisor=False)
- Scenario.optimization_advisor: new bool field (default False)
- ScenarioModel: same field added, loader.py透传
- RunArtifactPaths.advice_md: new path field
- factory.py: build_models() now public; build_metric_pipeline() accepts pre-built llm/embeddings
- runner.py: lifts llm, passes to pipeline and advisor; calls run_advisor() at end
- siemens online YAML: optimization_advisor: true enabled
- tests: 9 rules tests + 6 writer tests, all pass
- docs: advisor section added to engine-flow.md and architecture.md
Co-Authored-By: Claude <noreply@anthropic.com >
2026-06-16 17:06:19 +08:00
Guangfei.Zhao
9cbdc1d95d
first commit
2026-06-12 14:02:15 +08:00
