feat(session-async): add /api/score/session_async with incremental session report aggregation
- New POST /api/score/session_async endpoint: same session_id calls append to one shared report
- New GET /api/score/sessions/{session_id}: returns call_count, metric_means, all job records
- New GET /api/score/session/jobs/{job_id}: individual call status
- SessionScoreJobManager: deterministic run_id from session_id, per-session mutex for CSV append, advisor regenerated on every call
- SessionScoreRequest (extends ScoreRequest + session_id), SessionScoreJobResponse, SessionStatus models added
- 24 new tests, all passing
chore(weighted-score): comment out 综合加权得分 display and computation
- report.js: hide 综合加权得分 card in report detail page
- score_jobs.js: hide 综合 chip in async job list
- report_builder.py: overall_ws=None (computation disabled)
- summary.py: weighted_score summary line disabled
- evaluator.py: weighted_score/sample_weight columns no longer written to scores.csv
- score.py /api/score: weighted_score always returns null
- score_job_manager.py + session_score_manager.py: weighted=None
- Updated 3 tests to match new behaviour (6 pre-existing failures unchanged)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
<h2>优化建议怎么生成？</h2>
|
||||
<p class="subtitle">这决定了模块的核心机制与可维护性</p>
|
||||
|
||||
<div class="options">
|
||||
<div class="option" data-choice="a" onclick="toggleSelect(this)">
|
||||
<div class="letter">A</div>
|
||||
<div class="content">
|
||||
<h3>纯规则引擎</h3>
|
||||
<p>每个指标设阈值（如 faithfulness < 0.6），触发时给出预设建议文本。</p>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>零 LLM 调用，零额外成本</li>
|
||||
<li>结果可预测、可审计</li>
|
||||
<li>响应极快</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>建议固定，无法结合具体样本</li>
|
||||
<li>不能解释"为什么这批数据这个指标低"</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="b" onclick="toggleSelect(this)">
|
||||
<div class="letter">B</div>
|
||||
<div class="content">
|
||||
<h3>LLM 分析（全自动）</h3>
|
||||
<p>把评测结果（各指标均值 + 低分样本）一起交给 LLM，生成上下文感知的中文分析报告。</p>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>能结合具体低分样本给出针对性建议</li>
|
||||
<li>可用中文解释西门子场景下的问题</li>
|
||||
<li>建议质量高、内容丰富</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>每次评测多 1 次 LLM 调用</li>
|
||||
<li>依赖 judge_model 的质量</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="c" onclick="toggleSelect(this)">
|
||||
<div class="letter">C</div>
|
||||
<div class="content">
|
||||
<h3>规则定位 + LLM 解读（推荐）</h3>
|
||||
<p>规则引擎先识别哪些指标异常、触发哪条优化方向；再把"规则诊断 + 低分样本"一起给 LLM 做二次解读，生成中文建议。</p>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>规则保证诊断稳定，不依赖 LLM 自由发挥</li>
|
||||
<li>LLM 在有结构的输入下输出更准确</li>
|
||||
<li>两层可独立测试</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>实现略复杂（两个子模块）</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1,77 @@
|
||||
<h2>优化顾问模块 — 实现方案对比</h2>
|
||||
<p class="subtitle">三个方案的核心区别在于 LLM 调用边界和代码入侵程度</p>
|
||||
|
||||
<div class="options">
|
||||
<div class="option" data-choice="a" onclick="toggleSelect(this)">
|
||||
<div class="letter">A</div>
|
||||
<div class="content">
|
||||
<h3>独立后处理器（轻量集成）</h3>
|
||||
<p>新增 <code>rag_eval/advisor/</code> 包，<code>run_scenario()</code> 末尾调用一行 <code>maybe_run_advisor(result, scenario)</code>。</p>
|
||||
<p><strong>文件结构：</strong></p>
|
||||
<ul>
|
||||
<li><code>rag_eval/advisor/__init__.py</code></li>
|
||||
<li><code>rag_eval/advisor/rules.py</code> — 规则引擎，输入 score_rows，输出诊断列表</li>
|
||||
<li><code>rag_eval/advisor/llm_analyzer.py</code> — 把规则诊断 + 低分样本交给 judge_model</li>
|
||||
<li><code>rag_eval/advisor/writer.py</code> — 写 optimization_advice.md，打日志摘要</li>
|
||||
</ul>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>改动最小，runner.py 只加 3 行</li>
|
||||
<li>advisor 完全独立，可单独测试</li>
|
||||
<li>与现有分层架构完全吻合</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>无法拿到 per-metric 的原始 NaN 率（需从 score_rows 重新算）</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="b" onclick="toggleSelect(this)">
|
||||
<div class="letter">B</div>
|
||||
<div class="content">
|
||||
<h3>嵌入 reporting 层（复用写出基础设施）</h3>
|
||||
<p>把 advisor 作为 <code>rag_eval/reporting/</code> 的一部分，<code>write_run_artifacts()</code> 内部判断是否写 advice。</p>
|
||||
<p><strong>文件结构：</strong></p>
|
||||
<ul>
|
||||
<li><code>rag_eval/reporting/advisor.py</code> — 规则 + LLM + 写出三合一</li>
|
||||
<li><code>write_run_artifacts()</code> 里追加 <code>if scenario.optimization_advisor: write_advice(...)</code></li>
|
||||
</ul>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>artifacts 路径管理统一，advice 自然进 run 目录</li>
|
||||
<li>文件更少</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>reporting 层本是"无副作用写文件"，混入 LLM 调用破坏这一约定</li>
|
||||
<li>advisor 逻辑和写出逻辑耦合，难以单独测试规则引擎</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="c" onclick="toggleSelect(this)">
|
||||
<div class="letter">C</div>
|
||||
<div class="content">
|
||||
<h3>方案 A 变体：advisor 有独立 settings（推荐）</h3>
|
||||
<p>与方案 A 相同的文件结构，但 LLM 调用使用 <strong>scenario 已有的 judge_model</strong>，不新增任何模型配置——advisor 复用 <code>build_models()</code> 已构建好的 llm 实例。</p>
|
||||
<ul>
|
||||
<li><code>rag_eval/advisor/rules.py</code> — 纯函数，7 条指标诊断规则</li>
|
||||
<li><code>rag_eval/advisor/llm_analyzer.py</code> — 接收已有 llm 实例，不重新建 client</li>
|
||||
<li><code>rag_eval/advisor/writer.py</code> — 写 md + 日志</li>
|
||||
<li><code>rag_eval/advisor/__init__.py</code> — 暴露 <code>run_advisor()</code></li>
|
||||
</ul>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>不重复创建 LLM client（节省资源）</li>
|
||||
<li>advisor 阈值可通过 YAML 的 optimization_advisor 块扩展配置</li>
|
||||
<li>独立包边界清晰，易于单测</li>
|
||||
<li>runner.py 改动最小</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>需把 llm 实例从 runner 传入 advisor（多传一个参数）</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1,53 @@
|
||||
<h2>优化顾问模块 — 整体架构与数据流</h2>
|
||||
<p class="subtitle">新增 rag_eval/advisor/ 包，插入 run_scenario() 末尾，复用已有 llm 实例</p>
|
||||
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">执行链路（变更前 → 变更后）</div>
|
||||
<div class="mockup-body" style="font-family:monospace;font-size:13px;line-height:2">
|
||||
<span style="color:#94a3b8">run_scenario()</span><br>
|
||||
→ load_scenario() <span style="color:#94a3b8"># 读 YAML，解析 Scenario + optimization_advisor 字段</span><br>
|
||||
→ build_models() <span style="color:#94a3b8"># 已有：创建 llm, embeddings</span><br>
|
||||
→ build_metric_pipeline() <span style="color:#94a3b8"># 已有</span><br>
|
||||
→ Evaluator.evaluate() <span style="color:#94a3b8"># 已有：打分 → EvaluationResult</span><br>
|
||||
→ write_run_artifacts() <span style="color:#94a3b8"># 已有：scores.csv / summary.md / ...</span><br>
|
||||
<span style="color:#4ade80;font-weight:bold">→ run_advisor(result, scenario, llm) # 新增 3 行</span><br>
|
||||
<span style="color:#4ade80"> → rules.diagnose(score_rows) # 规则引擎：识别异常指标 + 方向</span><br>
|
||||
<span style="color:#4ade80"> → llm_analyzer.analyze(diag, samples) # LLM：结合低分样本生成中文建议</span><br>
|
||||
<span style="color:#4ade80"> → writer.write(advice, paths) # 写 optimization_advice.md + 日志</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h3>新增文件一览</h3>
|
||||
<div class="mockup">
|
||||
<div class="mockup-body" style="font-family:monospace;font-size:13px;line-height:1.9">
|
||||
rag_eval/advisor/<br>
|
||||
__init__.py <span style="color:#94a3b8">← 暴露 run_advisor()，是外部唯一入口</span><br>
|
||||
rules.py <span style="color:#94a3b8">← 纯函数，无 LLM，可单独单测</span><br>
|
||||
llm_analyzer.py <span style="color:#94a3b8">← 接收 llm 实例 + 诊断结构 → 中文 Markdown</span><br>
|
||||
writer.py <span style="color:#94a3b8">← 写 optimization_advice.md，打日志摘要</span><br>
|
||||
<br>
|
||||
rag_eval/shared/models.py <span style="color:#fbbf24">← 修改：Scenario 加 optimization_advisor 字段</span><br>
|
||||
rag_eval/config/schema.py <span style="color:#fbbf24">← 修改：ScenarioModel 加字段</span><br>
|
||||
rag_eval/execution/runner.py <span style="color:#fbbf24">← 修改：末尾加 3 行调用</span><br>
|
||||
rag_eval/reporting/artifacts.py <span style="color:#fbbf24">← 修改：RunArtifactPaths 加 advice_md 路径</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h3>输出产物</h3>
|
||||
<div class="mockup">
|
||||
<div class="mockup-body" style="font-family:monospace;font-size:13px;line-height:1.9">
|
||||
outputs/online/siemens-pdf-question-bank/<run_id>/<br>
|
||||
scenario.snapshot.yaml<br>
|
||||
scores.csv<br>
|
||||
invalid.csv<br>
|
||||
summary.md<br>
|
||||
metadata.json<br>
|
||||
<span style="color:#4ade80;font-weight:bold">optimization_advice.md ← 新增</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p style="margin-top:1rem;color:#94a3b8;font-size:13px">整体看起来 OK 吗？这是新模块与现有链路的接入方式。</p>
|
||||
@@ -0,0 +1,68 @@
|
||||
<h2>优化顾问在什么情况下运行？</h2>
|
||||
<p class="subtitle">这决定了模块与现有评测流程的集成方式</p>
|
||||
|
||||
<div class="options">
|
||||
<div class="option" data-choice="a" onclick="toggleSelect(this)">
|
||||
<div class="letter">A</div>
|
||||
<div class="content">
|
||||
<h3>每次评测自动运行</h3>
|
||||
<p>run_scenario() 结束后自动调用，无需任何额外配置。</p>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>零感知，开箱即用</li>
|
||||
<li>每次跑完都有建议报告</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>每次都多一次 LLM 调用，不管是否需要</li>
|
||||
<li>无法关闭</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="b" onclick="toggleSelect(this)">
|
||||
<div class="letter">B</div>
|
||||
<div class="content">
|
||||
<h3>YAML 场景中显式开启（推荐）</h3>
|
||||
<p>在 scenario YAML 里加一行 <code>optimization_advisor: true</code>，默认关闭。</p>
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">siemens-pdf-question-bank-online.yaml</div>
|
||||
<div class="mockup-body" style="font-family:monospace;font-size:13px;line-height:1.8">
|
||||
metrics:<br>
|
||||
- faithfulness<br>
|
||||
- noise_sensitivity<br>
|
||||
...<br>
|
||||
<span style="color:#4ade80;font-weight:bold">optimization_advisor: true # 新增</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>显式可见，按需开启</li>
|
||||
<li>与现有 YAML 驱动风格一致</li>
|
||||
<li>可为不同场景独立配置</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>需要手动在 YAML 里加一行</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="option" data-choice="c" onclick="toggleSelect(this)">
|
||||
<div class="letter">C</div>
|
||||
<div class="content">
|
||||
<h3>阈值触发（任一指标低于警戒线时自动激活）</h3>
|
||||
<p>规则引擎先算，若发现有指标低于阈值则自动启动 LLM 分析；一切正常则跳过。</p>
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>优点</h4><ul>
|
||||
<li>"有问题才报警"，符合直觉</li>
|
||||
<li>高分场景无额外成本</li>
|
||||
</ul></div>
|
||||
<div class="cons"><h4>缺点</h4><ul>
|
||||
<li>阈值需要维护，不同场景可能不同</li>
|
||||
<li>正常分数时无建议，但用户可能仍想看优化空间</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1,3 @@
|
||||
<div style="display:flex;align-items:center;justify-content:center;min-height:60vh">
|
||||
<p class="subtitle">Writing spec & moving to implementation...</p>
|
||||
</div>
|
||||
@@ -0,0 +1,3 @@
|
||||
<div style="display:flex;align-items:center;justify-content:center;min-height:60vh">
|
||||
<p class="subtitle">Continuing in terminal — 正在设计方案...</p>
|
||||
</div>
|
||||
@@ -0,0 +1 @@
|
||||
{"reason":"idle timeout","timestamp":1781598635371}
|
||||
1
.superpowers/brainstorm/1625-1781595805/state/server.pid
Normal file
1
.superpowers/brainstorm/1625-1781595805/state/server.pid
Normal file
@@ -0,0 +1 @@
|
||||
1625
|
||||
Reference in New Issue
Block a user