feat(advisor-comparison): add 相比上次运行 panel to report detail page

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

README.md

@@ -148,7 +148,50 @@ def run(question: str, **kwargs) -> dict:
 
 这意味着后续你比较不同模型、不同 prompt、不同检索策略时，不需要再靠手工记参数。
 
-## 7. PDF 题库构建
+## 7. 中文评判 Prompt 适配
+
+RAGAS 评判 prompt 默认为英文，对中文语料启用中文适配可提升判定一致性（架构设计 §10.5）。
+
+### 配置方式
+
+**场景 YAML**（离线/在线评估场景）：
+```yaml
+judge_language: zh   # 启用中文评判（默认 en）
+```
+
+**Score API**（`/api/score`、`/api/score/async`、`/api/score/session_async`）：
+```json
+{ "judge_language": "zh", ... }
+```
+
+**全局默认**（`.env`）：
+```env
+RAGAS_JUDGE_LANGUAGE=zh
+```
+
+优先级：请求/场景显式值 > `RAGAS_JUDGE_LANGUAGE`（默认 `en`）。
+
+### 支持的指标
+
+| 指标 | 中文适配 |
+|---|---|
+| `faithfulness` | ✅ |
+| `answer_relevancy` | ✅ |
+| `context_recall` | ✅ |
+| `context_precision` | ✅ |
+| `factual_correctness` | ✅ |
+| `noise_sensitivity` | ⚠️ 使用内置函数式 prompt，不支持 adapt，保持英文 |
+| `semantic_similarity` | — 无 LLM 调用，不涉及 |
+
+### 缓存机制
+
+中文 prompt 以 JSON 缓存提交在 `configs/judge_prompts/zh/`，运行时零延迟加载。RAGAS 升级后若检测到 prompt 变化（`stale cache` 日志告警），请重新运行：
+
+```bash
+python -m scripts.build_judge_prompt_cache --language zh
+```
+
+
 
 仓库现在额外支持把 PDF 文档解析成可人工复核的在线题库草稿。最推荐的阅读顺序是：
 

configs/judge_prompts/zh/answer_relevancy__prompt.json

@@ -0,0 +1,37 @@
+{
+  "metric": "answer_relevancy",
+  "prompt_attr": "prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "f6f3a804bf3542363e3ce438ca62e1545de2736058e77900820bacc86be05ac3",
+  "instruction": "根据给定的回答生成对应的问题，并判断该回答是否为回避性回答（noncommittal）。\n如果回答是回避性的（含糊、模糊或有歧义），则 noncommittal 为 1；如果回答具有实质内容，则为 0。\n回避性回答示例：「我不知道」「我不确定」「这取决于具体情况」。",
+  "examples": [
+    {
+      "input": {
+        "response": "阿尔伯特·爱因斯坦出生于德国。"
+      },
+      "output": {
+        "question": "阿尔伯特·爱因斯坦出生在哪里？",
+        "noncommittal": 0
+      }
+    },
+    {
+      "input": {
+        "response": "法国的首都是巴黎，这座城市以其建筑和文化而闻名。"
+      },
+      "output": {
+        "question": "法国的首都是哪里？",
+        "noncommittal": 0
+      }
+    },
+    {
+      "input": {
+        "response": "关于2023年发明的智能手机的突破性功能，我不了解，因为我没有2022年以后的信息。"
+      },
+      "output": {
+        "question": "2023年发明的智能手机有哪些突破性功能？",
+        "noncommittal": 1
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/context_precision__prompt.json

@@ -0,0 +1,43 @@
+{
+  "metric": "context_precision",
+  "prompt_attr": "prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "948aa1ff0d12e1903926ddadbdf4420554b0c0bded6d01bf4eabd46b69697511",
+  "instruction": "给定问题、回答和上下文，验证该上下文在得出给定回答时是否有用。若有用则输出 verdict 为 1，若无用则输出 0，并以 JSON 格式返回结果。",
+  "examples": [
+    {
+      "input": {
+        "question": "请介绍一下阿尔伯特·爱因斯坦。",
+        "context": "阿尔伯特·爱因斯坦（1879年3月14日—1955年4月18日）是一位出生于德国的理论物理学家，被广泛认为是有史以来最伟大、最具影响力的科学家之一。以创立相对论著称，同时对量子力学做出了重要贡献。他的质能等价公式 E = mc² 被称为「世界上最著名的方程」。他因「对理论物理学的贡献，特别是发现光电效应定律」而荣获1921年诺贝尔物理学奖。",
+        "answer": "阿尔伯特·爱因斯坦生于1879年3月14日，是一位出生于德国的理论物理学家，被广泛认为是有史以来最伟大、最具影响力的科学家之一。他因对理论物理学的贡献荣获1921年诺贝尔物理学奖。"
+      },
+      "output": {
+        "reason": "提供的上下文对得出给定回答确实有帮助。上下文包含了关于爱因斯坦生平和贡献的关键信息，这些内容都体现在回答中。",
+        "verdict": 1
+      }
+    },
+    {
+      "input": {
+        "question": "2020年ICC世界杯谁获胜？",
+        "context": "2022年ICC男子T20世界杯于2022年10月16日至11月13日在澳大利亚举行，是该赛事第八届。原定于2020年举办，因新冠疫情推迟。英格兰队在决赛中以五个击球门差击败巴基斯坦队，赢得了第二个ICC男子T20世界杯冠军。",
+        "answer": "英格兰。"
+      },
+      "output": {
+        "reason": "该上下文有助于澄清2020年ICC世界杯的情况，并表明英格兰是原定于2020年、实际于2022年举办的锦标赛的获胜者。",
+        "verdict": 1
+      }
+    },
+    {
+      "input": {
+        "question": "世界上最高的山是什么？",
+        "context": "安第斯山脉是世界上最长的大陆山脉，位于南美洲，横跨七个国家，拥有西半球许多最高峰。该山脉以其多样的生态系统而闻名，包括高海拔的安第斯高原和亚马逊雨林。",
+        "answer": "珠穆朗玛峰。"
+      },
+      "output": {
+        "reason": "提供的上下文讨论安第斯山脉，虽然令人印象深刻，但不包含珠穆朗玛峰，也与世界最高山峰的问题无直接关联。",
+        "verdict": 0
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/context_recall__prompt.json

@@ -0,0 +1,73 @@
+{
+  "metric": "context_recall",
+  "prompt_attr": "prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "51995670010acd56920fa3cd9959b4cdb4a82a692e5cd2e9cfd545b3dfc68a22",
+  "instruction": "给定一段上下文和一个回答，分析回答中每一条陈述，判断该陈述是否可以归因于给定的上下文。\n仅使用二分类：如果陈述可以从上下文中归因，则为 1；否则为 0。\n请为每条分类提供详细的推理说明。",
+  "examples": [
+    {
+      "input": {
+        "question": "请介绍一下阿尔伯特·爱因斯坦。",
+        "context": "阿尔伯特·爱因斯坦（1879年3月14日—1955年4月18日）是一位出生于德国的理论物理学家，被广泛认为是有史以来最伟大、最具影响力的科学家之一。他以创立相对论而闻名，同时对量子力学也做出了重要贡献，是20世纪初物理学革命的核心人物。他的质能等价公式 E = mc²被称为「世界上最著名的方程」。他因「对理论物理学的贡献，特别是发现光电效应定律」而荣获1921年诺贝尔物理学奖。",
+        "answer": "阿尔伯特·爱因斯坦生于1879年3月14日，是一位出生于德国的理论物理学家，被广泛认为是有史以来最伟大、最具影响力的科学家之一。他因对理论物理学的贡献荣获1921年诺贝尔物理学奖。他于1905年发表了4篇论文。爱因斯坦于1895年移居瑞士。"
+      },
+      "output": {
+        "classifications": [
+          {
+            "statement": "阿尔伯特·爱因斯坦生于1879年3月14日，是一位出生于德国的理论物理学家，被广泛认为是有史以来最伟大、最具影响力的科学家之一。",
+            "reason": "上下文明确提到了爱因斯坦的出生日期及其理论物理学家身份。",
+            "attributed": 1
+          },
+          {
+            "statement": "他因对理论物理学的贡献荣获1921年诺贝尔物理学奖。",
+            "reason": "上下文中有完全一致的表述。",
+            "attributed": 1
+          },
+          {
+            "statement": "他于1905年发表了4篇论文。",
+            "reason": "上下文中没有提到他发表论文的信息。",
+            "attributed": 0
+          },
+          {
+            "statement": "爱因斯坦于1895年移居瑞士。",
+            "reason": "上下文中没有支持该陈述的依据。",
+            "attributed": 0
+          }
+        ]
+      }
+    },
+    {
+      "input": {
+        "question": "2020年ICC世界杯谁获胜？",
+        "context": "2022年ICC男子T20世界杯于2022年10月16日至11月13日在澳大利亚举行，这是该赛事的第八届。原定于2020年举办，因新冠疫情推迟。英格兰队在决赛中以五个击球门差击败巴基斯坦队，赢得了第二个ICC男子T20世界杯冠军。",
+        "answer": "英格兰。"
+      },
+      "output": {
+        "classifications": [
+          {
+            "statement": "英格兰。",
+            "reason": "上下文说明英格兰赢得了2022年届（原定2020年）ICC世界杯冠军。",
+            "attributed": 1
+          }
+        ]
+      }
+    },
+    {
+      "input": {
+        "question": "世界上最高的山是什么？",
+        "context": "安第斯山脉是世界上最长的大陆山脉，位于南美洲，横跨七个国家，拥有西半球许多最高峰。该山脉以其多样的生态系统而闻名，包括高海拔的安第斯高原和亚马逊雨林。",
+        "answer": "珠穆朗玛峰。"
+      },
+      "output": {
+        "classifications": [
+          {
+            "statement": "珠穆朗玛峰。",
+            "reason": "上下文讨论的是安第斯山脉，与世界最高峰珠穆朗玛峰无关。",
+            "attributed": 0
+          }
+        ]
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/factual_correctness__nli_prompt.json

@@ -0,0 +1,45 @@
+{
+  "metric": "factual_correctness",
+  "prompt_attr": "nli_prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "28e28e3c3cd02af21729a850633791390f2173deeaddfb006bba11e8569e7af5",
+  "instruction": "你的任务是根据给定的上下文，判断一系列陈述的真实性（忠实度）。对于每条陈述，如果该陈述可以直接从上下文中推断出来，则返回 verdict 为 1；如果不能从上下文中直接推断，则返回 verdict 为 0。",
+  "examples": [
+    {
+      "input": {
+        "context": "张明是XYZ大学的一名学生，主修计算机科学。本学期他选修了数据结构、算法和数据库管理等几门课程。张明是一名勤奋的学生，花大量时间学习和完成作业。他经常在图书馆里工作到很晚，完成他的项目。",
+        "statements": [
+          "张明主修生物学。",
+          "张明正在修读人工智能课程。",
+          "张明是一名努力认真的学生。",
+          "张明有一份兼职工作。"
+        ]
+      },
+      "output": {
+        "statements": [
+          {
+            "statement": "张明主修生物学。",
+            "reason": "上下文明确指出张明的专业是计算机科学，而非生物学。",
+            "verdict": 0
+          },
+          {
+            "statement": "张明正在修读人工智能课程。",
+            "reason": "上下文提到的课程包括数据结构、算法和数据库管理，未提及人工智能课程。",
+            "verdict": 0
+          },
+          {
+            "statement": "张明是一名努力认真的学生。",
+            "reason": "上下文描述张明是一名勤奋的学生，花大量时间学习和完成作业，与该陈述相符。",
+            "verdict": 1
+          },
+          {
+            "statement": "张明有一份兼职工作。",
+            "reason": "上下文中没有关于张明兼职工作的任何信息。",
+            "verdict": 0
+          }
+        ]
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/factual_correctness__prompt.json

@@ -0,0 +1,35 @@
+{
+  "metric": "factual_correctness",
+  "prompt_attr": "prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "bacda198cc6f32b2f450e3aa63a492d461fc63a60222e91a3ac9d22013ab6113",
+  "instruction": "将每个输入句子分解为一条或多条独立陈述。每条陈述应是一个可独立验证的独立论断。\n请按照示例中展示的原子性（atomicity）和覆盖度（coverage）水平进行分解。",
+  "examples": [
+    {
+      "input": {
+        "response": "查尔斯·巴贝奇是一位法国数学家、哲学家和美食评论家。",
+        "atomicity": "low",
+        "coverage": "low"
+      },
+      "output": {
+        "claims": [
+          "查尔斯·巴贝奇是一位数学家和哲学家。"
+        ]
+      }
+    },
+    {
+      "input": {
+        "response": "阿尔伯特·爱因斯坦是一位德国理论物理学家。他创立了相对论，并对量子力学的发展做出了贡献。",
+        "atomicity": "low",
+        "coverage": "low"
+      },
+      "output": {
+        "claims": [
+          "阿尔伯特·爱因斯坦是一位德国物理学家。",
+          "阿尔伯特·爱因斯坦创立了相对论并对量子力学做出了贡献。"
+        ]
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/faithfulness__nli_statement_prompt.json

@@ -0,0 +1,45 @@
+{
+  "metric": "faithfulness",
+  "prompt_attr": "nli_statement_prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "28e28e3c3cd02af21729a850633791390f2173deeaddfb006bba11e8569e7af5",
+  "instruction": "你的任务是根据给定的上下文，判断一系列陈述的真实性（忠实度）。对于每条陈述，如果该陈述可以直接从上下文中推断出来，则返回 verdict 为 1；如果不能从上下文中直接推断，则返回 verdict 为 0。",
+  "examples": [
+    {
+      "input": {
+        "context": "张明是XYZ大学的一名学生，主修计算机科学。本学期他选修了数据结构、算法和数据库管理等几门课程。张明是一名勤奋的学生，花大量时间学习和完成作业。他经常在图书馆里工作到很晚，完成他的项目。",
+        "statements": [
+          "张明主修生物学。",
+          "张明正在修读人工智能课程。",
+          "张明是一名努力认真的学生。",
+          "张明有一份兼职工作。"
+        ]
+      },
+      "output": {
+        "statements": [
+          {
+            "statement": "张明主修生物学。",
+            "reason": "上下文明确指出张明的专业是计算机科学，而非生物学。",
+            "verdict": 0
+          },
+          {
+            "statement": "张明正在修读人工智能课程。",
+            "reason": "上下文提到的课程包括数据结构、算法和数据库管理，未提及人工智能课程。",
+            "verdict": 0
+          },
+          {
+            "statement": "张明是一名努力认真的学生。",
+            "reason": "上下文描述张明是一名勤奋的学生，花大量时间学习和完成作业，与该陈述相符。",
+            "verdict": 1
+          },
+          {
+            "statement": "张明有一份兼职工作。",
+            "reason": "上下文中没有关于张明兼职工作的任何信息。",
+            "verdict": 0
+          }
+        ]
+      }
+    }
+  ]
+}

configs/judge_prompts/zh/faithfulness__statement_generator_prompt.json

@@ -0,0 +1,24 @@
+{
+  "metric": "faithfulness",
+  "prompt_attr": "statement_generator_prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "7950551a460f88534523f1f8b7d5819395edb6eb90bb6d246b71a51dab85dfaf",
+  "instruction": "给定一个问题和对应的回答，分析回答中每个句子的复杂性，将每个句子拆解为一条或多条完整、可独立理解的陈述。每条陈述中不得使用代词，必须明确指代对象。",
+  "examples": [
+    {
+      "input": {
+        "question": "阿尔伯特·爱因斯坦是谁？他最著名的成就是什么？",
+        "answer": "他是一位出生于德国的理论物理学家，被公认为有史以来最伟大、最具影响力的物理学家之一。他最著名的成就是创立相对论，同时也对量子力学的发展做出了重要贡献。"
+      },
+      "output": {
+        "statements": [
+          "阿尔伯特·爱因斯坦是一位出生于德国的理论物理学家。",
+          "阿尔伯特·爱因斯坦被公认为有史以来最伟大、最具影响力的物理学家之一。",
+          "阿尔伯特·爱因斯坦最著名的成就是创立相对论。",
+          "阿尔伯特·爱因斯坦对量子力学的发展做出了重要贡献。"
+        ]
+      }
+    }
+  ]
+}

docs/superpowers/specs/2026-07-01-chinese-judge-prompt-design.md

@@ -0,0 +1,189 @@
+# 中文评判 Prompt 适配设计（judge_language: zh）
+
+**日期**: 2026-07-01
+**状态**: 已批准，待实现
+**范围**: 让 RAGAS 的 LLM 评判 prompt 从默认英文切换为中文，提升对中文语料的判定一致性（架构设计 §10.5）。通过场景 YAML 字段 `judge_language: zh` 和 score API 可选字段 `judge_language` 控制，默认 `en` 保持完全向后兼容。
+
+---
+
+## 1. 背景与目标
+
+架构设计 §10.5 明确要求：
+
+> 语料为中文，评判模型选多语言能力充分者即可。RAGAS 评判 prompt 默认英文，对中文语料应启用其语言适配，将评判 prompt 切换为中文以提升判定一致性。
+
+当前 `rag_eval/metrics/factory.py` 与 `webapp/services/inline_scorer.py` 构建的 RAGAS `ragas.metrics.collections` 指标全部使用默认英文 `instruction` + `examples`。本设计新增中文评判能力：
+
+- 场景 YAML 新增 `judge_language: zh`（默认 `en`）。
+- score 系列 API（`/api/score`、`/api/score/async`、`/api/score/session_async`）新增可选 `judge_language` 字段。
+- 采用 RAGAS 原生 `BasePrompt.adapt()` + 提交到仓库的持久化缓存文件，生产运行完全确定、零额外延迟。
+
+**成功标准**：
+- `judge_language: zh` 时，6 个 LLM 评判指标使用中文 prompt；`semantic_similarity`（纯 embedding）不受影响。
+- 默认 `en` 时，所有既有场景与 API 行为逐字节不变。
+- 单变量对比可行：同一数据集分别以 `en` / `zh` 各评一次，比较指标均值与方差。
+
+---
+
+## 2. 需要本地化的指标与 prompt（已从 RAGAS 0.4.3 源码核实）
+
+每个 LLM 评判指标把 prompt 存为**实例属性**，可在构建后覆盖：
+
+| 指标 | prompt 实例属性 |
+|---|---|
+| `faithfulness` | `statement_generator_prompt`, `nli_statement_prompt` |
+| `answer_relevancy` | `prompt` |
+| `context_recall` | `prompt` |
+| `context_precision` | `prompt` |
+| `noise_sensitivity` | 使用函数式 `to_string()`（无 `instruction`/`examples`），无法通过 `adapt()` 本地化，**跳过** |
+| `factual_correctness` | `prompt`, `nli_prompt` |
+| `semantic_similarity` | 无（纯 embedding，跳过） |
+
+权威映射（写入 `judge_prompts.py`）：
+
+```python
+METRIC_PROMPT_ATTRS = {
+    "faithfulness": ("statement_generator_prompt", "nli_statement_prompt"),
+    "answer_relevancy": ("prompt",),
+    "context_recall": ("prompt",),
+    "context_precision": ("prompt",),
+    "noise_sensitivity": ("statement_prompt", "faithfulness_prompt"),
+    "factual_correctness": ("prompt", "nli_prompt"),
+}
+```
+
+`ragas.prompt.metrics.base_prompt.BasePrompt` 提供原生 `async adapt(target_language, llm, adapt_instruction=False)`：翻译 `examples`，`adapt_instruction=True` 时同时翻译 `instruction`，返回新 prompt 实例，并设置 `language` 属性。
+
+---
+
+## 3. 架构与数据流
+
+```
+[一次性] scripts/build_judge_prompt_cache.py --language zh
+    对每个指标的每个 prompt 调 adapt("chinese", llm, adapt_instruction=True)
+    → 写出 configs/judge_prompts/zh/<metric>__<attr>.json（提交入库）
+
+[运行时·YAML 场景路径]
+    runner → factory.build_metric_pipeline(scenario)
+        若 scenario.judge_language == "zh":
+            localize_pipeline_prompts(registry, "zh")
+
+[运行时·score API 路径]
+    /api/score, /api/score/async, /api/score/session_async
+        → inline_scorer.score(..., judge_language=...)
+            → _build_metric_instances(...)
+                若 language == "zh": localize_pipeline_prompts(registry, "zh")
+```
+
+两条路径共用同一个本地化器，保持 DRY。
+
+---
+
+## 4. 组件
+
+### 4.1 一次性引导脚本 `scripts/build_judge_prompt_cache.py`
+
+- 参数：`--language zh`、`--judge-model <model>`（默认取 settings）。
+- 对 `METRIC_PROMPT_ATTRS` 每个 (metric, attr)：取活体 prompt 实例，`await prompt.adapt("chinese", llm, adapt_instruction=True)`，序列化为 JSON。
+- `adapt()` 失败重试若干次；仍失败则整体中止、不写半成品。
+- 产物提交入库，评审后即为生产资产。
+
+### 4.2 缓存文件 `configs/judge_prompts/zh/<metric>__<attr>.json`
+
+按 `(metric, prompt_attr)` 命名，1:1 对应覆盖目标，零碰撞（多个指标各自独立生成，即使 prompt 类同名也互不影响）。
+
+```json
+{
+  "metric": "faithfulness",
+  "prompt_attr": "nli_statement_prompt",
+  "language": "chinese",
+  "ragas_version": "0.4.3",
+  "source_hash": "<英文 instruction + 序列化 examples 的 sha256>",
+  "instruction": "……中文 instruction……",
+  "examples": [ { "input": { }, "output": { } } ]
+}
+```
+
+### 4.3 运行时本地化器 `rag_eval/metrics/judge_prompts.py`
+
+公开函数：`localize_pipeline_prompts(registry: dict[str, Any], language: str) -> LocalizationReport`
+
+- `language == "en"` 或空 → 直接返回，不做任何事。
+- 对 registry 中每个已知指标的每个 prompt 属性：
+  1. 从缓存目录读取对应 JSON（**内存缓存已解析结果**，按 language 键，首次读盘后复用，保证 `/api/score` 高频调用零额外磁盘 IO）。
+  2. 用**活体 prompt 实例的** `input_model` / `output_model` 从 JSON 重建 `examples`，覆盖 `prompt.instruction`、`prompt.examples`、`prompt.language`。
+- 返回 `LocalizationReport`（已应用 / 跳过 / 告警计数），供日志与测试断言。
+
+### 4.4 配置面
+
+- `rag_eval/settings.py`：新增 `ragas_judge_language`（env `RAGAS_JUDGE_LANGUAGE`，默认 `en`）作为唯一全局默认。
+- `rag_eval/shared/models.py`：`Scenario`（`@dataclass`）新增 `judge_language: str = "en"`，存放**已解析**的最终值。
+- **YAML 场景路径的解析在配置加载器**（`rag_eval/config`）完成：`judge_language = YAML 显式值 or settings.ragas_judge_language`；同时在此处校验值域 `{en, zh}`（与既有 metrics 校验同层，dataclass 本身不做校验）。因此 `Scenario.judge_language` 对 factory 而言是权威值。
+- `webapp/models.py`：`ScoreRequest` 新增 `judge_language: str | None = None`（`SessionScoreRequest` 继承自动获得）；score API 路径解析 `request.judge_language or settings.ragas_judge_language`。
+- **统一优先级（两条路径一致）**：显式值（YAML 字段 / 请求字段）> `settings.ragas_judge_language`（默认 `en`）。
+
+### 4.5 集成点改动
+
+- `rag_eval/metrics/factory.py`：`build_metric_pipeline()` 构建 registry 后，若 `scenario.judge_language == "zh"`，调 `localize_pipeline_prompts(registry, "zh")`。
+- `webapp/services/inline_scorer.py`：`score()` 与 `_build_metric_instances()` 增加 `judge_language` 参数；构建指标后按需本地化。`(judge_model, embedding_model)` 缓存仅缓存 llm/embeddings，指标每次重建，故每次正确应用本地化，缓存键无需变动。
+- `webapp/api/score.py`、`webapp/services/score_job_manager.py`、`webapp/services/session_score_manager.py`：把 `request.judge_language`（回退到 settings 默认）透传给 `inline_scorer.score()`。
+
+---
+
+## 5. 错误处理与漂移检测
+
+- **运行时缺文件 / JSON 损坏 / examples 按当前 schema 重建失败** → 记 WARNING，保留该 prompt 的英文默认，**绝不中断评分**（优雅降级）。
+- **`judge_language: zh` 但缓存整体缺失** → 告警一次，按英文继续。
+- **漂移检测**：加载时用当前英文源计算 `source_hash` 与缓存值比对：
+  - 不一致但 schema 仍有效 → 应用中文 + 记 loud WARNING「缓存已过期，请重新运行 build_judge_prompt_cache.py」。
+  - schema 已失效（重建抛错）→ 自动回退英文。
+- **引导脚本** adapt 失败 → 重试后中止，不写半成品。
+
+---
+
+## 6. 测试（确定性，Mock LLM，遵循 AGENTS.md）
+
+新增 `tests/test_judge_prompt_localizer.py`：
+
+- fixture 缓存文件正确覆盖 `metric.<attr>.instruction` 与 `.examples`（examples 按活体 model 重建成功）。
+- 缺文件 → 保留英文默认、不抛异常。
+- `source_hash` 不匹配 → 记告警仍应用中文；schema 失效 → 回退英文。
+- `language == "en"` → registry 完全不变。
+- 内存缓存：同一 language 第二次调用不再读盘（用可计数的假 loader 或 monkeypatch 断言）。
+
+扩展/新增：
+
+- `Scenario` schema：`judge_language` 解析、默认 `en`、非法值报错。
+- `ScoreRequest`：`judge_language` 可选、默认 `None`。
+- 引导脚本：mock LLM（`adapt` 返回预置中文）→ 断言写出预期 JSON 结构与字段。
+
+**可选人工验证（非 CI）**：固定小样本集分别以 `en` / `zh` 各评一次，比较均值/方差，并复核已知正确样本（如 “MAGNETOM Vida 设备总重 7370 kg”）。
+
+---
+
+## 7. 改动文件清单
+
+**新增**：
+- `rag_eval/metrics/judge_prompts.py` — 本地化器（加载/覆盖/hash/内存缓存）
+- `scripts/build_judge_prompt_cache.py` — 一次性引导脚本
+- `configs/judge_prompts/zh/*.json` — 提交入库的中文 prompt 缓存
+- `tests/test_judge_prompt_localizer.py` — 测试
+
+**修改**：
+- `rag_eval/shared/models.py` — `Scenario.judge_language` 字段（存已解析值）
+- `rag_eval/config`（YAML 加载器）— 解析 `judge_language` = 显式值 or settings 默认，并校验值域
+- `rag_eval/settings.py` — `ragas_judge_language` 默认
+- `rag_eval/metrics/factory.py` — `build_metric_pipeline` 接入本地化
+- `webapp/models.py` — `ScoreRequest.judge_language`
+- `webapp/services/inline_scorer.py` — `score()`/`_build_metric_instances()` +参数+本地化
+- `webapp/api/score.py`、`webapp/services/score_job_manager.py`、`webapp/services/session_score_manager.py` — 透传 `judge_language`
+- 示例场景 YAML（如 `scenarios/siemens_build/*`、一个 offline 示例）— 增加 `judge_language: zh`
+- `README.md` — 简述机制与「如何重新生成缓存」
+
+---
+
+## 8. 兼容性与影响
+
+- 三个 score 接口默认（不传 `judge_language`）行为**逐字节不变**，现有 Dify Tool 调用零影响。
+- YAML 场景默认 `judge_language: en`，既有 run 结果口径不变。
+- 仅当显式 `zh` 时切换中文 prompt；可随时通过重跑脚本刷新缓存。

docs/superpowers/specs/2026-07-02-advisor-comparison-design.md

@@ -0,0 +1,159 @@
+# 优化建议历史对比设计（Advisor Comparison）
+
+**日期**: 2026-07-02
+**状态**: 已批准，待实现
+**范围**: 报告详情页「优化建议」区域新增"相比上次运行"的自动对比摘要，落地架构设计 §11"单变量变更、回归复测"方法论。仅涉及 webapp 读取/展示层，不改动 `rag_eval/advisor/` 的任何写入逻辑。
+
+---
+
+## 1. 背景与目标
+
+架构设计 §11 明确优化方法论：
+
+> 优化由评估结果驱动，遵循**单变量变更、回归复测**，避免一次性引入多项变更导致归因困难……每批经回归复测确认后再推进下一批。
+
+当前平台已有两类"跨 run 对比"能力：
+- **Dashboard**（`dashboard.js`）：多 run 指标均值趋势线图 + 阈值柱状图，用户手动勾选 run 对比
+- **问题历史**（`question_history.py`）：同一问题在不同 run 中的分数对比，展示在最低分样本详情里
+
+但**顾问诊断层面**没有对比能力：`Diagnosis`（metric/severity/threshold）目前只以自由文本嵌入 `optimization_advice.md`，从未结构化保留，无法程序化判断"上次标记的问题这次是否真的解决了"。
+
+**成功标准**：
+- 打开报告详情页时，若存在同 `scenario_name` 的上一次运行，优化建议区域自动显示一个精简的"相比上次运行"摘要，只列出有变化或仍有问题的指标
+- 找不到上一次运行（首次评测、或 `/api/score/async`/`session_async` 这类每次 scenario_name 唯一的场景）时，静默不显示，不影响现有优化建议区域
+- 不新增任何持久化文件；对比结果始终基于当前代码里的最新阈值规则重新计算
+- 现有 `rag_eval/advisor/` 三个文件（rules.py / llm_analyzer.py / writer.py）与三条写入入口（runner.py / score_job_manager.py / session_score_manager.py）零改动
+
+---
+
+## 2. 架构
+
+新增的对比逻辑完全是 **webapp 层的只读附加计算**，对齐 `question_history.py` 的既有模式：
+
+```
+report_builder.build_report(run_dir, metrics)
+  ├── 现有逻辑：读取 scores.csv、构建 metric_means / distributions / groupings / lowest_samples
+  ├── 现有逻辑：question_history.build_question_history_index(...)
+  └── 新增：advisor_comparison.build_advisor_comparison(run_dir, scenario_name, metrics)
+              ├── find_previous_run(...) → 复用 run_reader.list_run_summaries()
+              ├── 读取上一次 run 的 scores.csv + metrics（复用 run_reader 既有 helper）
+              ├── 对两次分别调用 rag_eval.advisor.diagnose()（已公开，零修改）
+              └── 按指标名分类差异状态，返回 AdvisorComparison | None
+```
+
+不引入新的持久化文件；每次报告详情页请求都会现场重新计算，天然保证"始终用最新阈值规则重新评估历史数据"。
+
+---
+
+## 3. 数据模型
+
+### 3.1 `webapp/services/advisor_comparison.py`（新文件）
+
+```python
+def find_previous_run(
+    scenario_name: str,
+    current_run_id: str,
+    current_finished_at: str,
+) -> RunSummary | None:
+    """在所有 run 中找到同 scenario_name、时间上最近的前一次运行。"""
+
+def build_advisor_comparison(
+    run_dir: Path,
+    scenario_name: str,
+    metrics: list[str],
+) -> AdvisorComparison | None:
+    """构建当前 run 相对上一次同名 run 的顾问诊断差异，找不到上一次 run 时返回 None。"""
+```
+
+内部指标状态分类（4 种，两次都健康的指标不生成条目）：
+
+| 状态 | 含义 |
+|---|---|
+| `resolved` | 上次触发诊断，本次不再触发 |
+| `regressed` | 上次未触发（该指标当时也被评测），本次新触发 |
+| `still_triggered` | 两次都触发，展示分数与严重度变化 |
+| `new_metric` | 上次运行未评测该指标（无基线），本次触发——不算回归，只是新指标 |
+
+### 3.2 `webapp/models.py` 新增
+
+```python
+class AdvisorComparisonEntry(BaseModel):
+    metric: str
+    status: Literal["resolved", "regressed", "still_triggered", "new_metric"]
+    previous_score: float | None = None
+    previous_severity: str | None = None  # "critical" | "warning" | "low" | None
+    current_score: float | None = None
+    current_severity: str | None = None
+
+
+class AdvisorComparison(BaseModel):
+    previous_run_id: str
+    previous_finished_at: str
+    previous_judge_model: str = ""
+    current_judge_model: str = ""
+    judge_model_changed: bool = False
+    entries: list[AdvisorComparisonEntry] = Field(default_factory=list)
+```
+
+`ReportData` 新增字段：
+
+```python
+advisor_comparison: AdvisorComparison | None = Field(
+    default=None,
+    description="相比同场景上一次运行的顾问诊断差异；无可比对象时为 None。",
+)
+```
+
+---
+
+## 4. 数据流
+
+1. 用户打开报告详情页，run R（`scenario_name=S`、`run_id=R_id`、`finished_at=T`、`metrics=M`）
+2. `build_report()` 调用 `find_previous_run(S, R_id, T)`：
+   - 复用现有 `run_reader.list_run_summaries()`
+   - 过滤 `scenario_name == S 且 run_id != R_id 且 finished_at < T`
+   - 按 `finished_at` 倒序，取第一个（时间上最近的前一次）
+   - 无匹配 → 返回 `None`
+3. 若找到上一次 run P：
+   - 读取 P 的 `scores.csv`（`run_reader.read_scores_frame`）与其 metrics 列表（快照或推断，复用现有 helper）
+   - 对 P 和 R 分别调用 `rag_eval.advisor.diagnose(score_rows, metrics)`（现有函数，零修改）
+   - 取两次触发指标名的并集，按 §3.1 表格分类状态，跳过两次都未触发的指标
+   - 记录两次的 `judge_model`/`embedding_model`（来自各自的 `metadata.json`），设置 `judge_model_changed`
+4. 返回 `AdvisorComparison`（或 `None`），挂到 `ReportData.advisor_comparison`
+5. 前端 `report.js` 新增 `renderAdvisorComparison(report)`：
+   - `advisor_comparison` 为 `None` 或 `entries` 为空 → 不渲染任何内容
+   - 否则在「⑤ 优化建议」区域上方渲染"相比上次运行（run_id，时间）"精简列表，每条用 `MetricPresenter.deltaInfo()` 计算涨跌箭头，复用 `.delta-good`/`.delta-bad` 样式；`new_metric`/`regressed` 状态额外给一个醒目标记
+   - `judge_model_changed=true` 时追加一行小字提示"judge_model 不同（X → Y），对比仅供参考"
+
+---
+
+## 5. 错误处理
+
+| 情况 | 处理 |
+|---|---|
+| 找不到同 scenario_name 的历史 run（含 `/api/score/async`、`session_async` 这类 scenario_name 唯一的场景） | `find_previous_run` 返回 `None`，整个对比区域不渲染 |
+| 上一次 run 的 `scores.csv` 缺失/损坏/无法解析 | 捕获异常，记录日志，`build_advisor_comparison` 返回 `None`（不影响报告详情页其余部分） |
+| 两次指标完全不重合、且都没有 `new_metric` 触发 | `entries` 为空列表，前端按"无内容"处理，不显示对比区域 |
+| `diagnose()` 内部抛出任何异常 | 在 `build_advisor_comparison` 顶层 try/except 兜底，返回 `None`，绝不让报告详情页 500 |
+
+设计原则与 `run_advisor()` 一致：辅助性/可观测性功能失败必须静默降级，不能影响核心报告渲染。
+
+---
+
+## 6. 测试策略
+
+- `tests/test_advisor_comparison.py`（新文件）：
+  - `find_previous_run`：同名匹配、排除自身、按时间取最近一次、无匹配返回 `None`、多个候选按时间正确排序
+  - `build_advisor_comparison`：用 `tmp_path` 构造两个假 run 目录（`scores.csv` + `metadata.json` + `scenario.snapshot.yaml`，复用 `tests/test_webapp_report_builder.py` 里已有的构造模式），覆盖 4 种状态分类、`judge_model_changed` 检测、无上一次 run 时返回 `None`、两次都健康时 `entries` 为空
+- 扩展 `tests/test_webapp_report_builder.py`：验证 `build_report()` 正确挂载 `advisor_comparison` 字段（有上一次 run / 无上一次 run 两种情形）
+- 不修改任何现有 `test_advisor_*.py`（rules/llm_analyzer/writer 行为完全不变，零回归风险）
+
+---
+
+## 7. 非目标（本轮明确不做）
+
+- 不支持手动选择任意两个 run 做对比（只做"自动对比同场景上一次"）
+- 不持久化结构化诊断数据（`diagnoses.json` 或类似文件）——每次现场用 `diagnose()` 重新计算
+- 不改动 `rag_eval/advisor/` 内部规则、LLM 分析 prompt 或写入逻辑
+- 不支持跨 scenario_name 的对比（不同场景之间的指标不具可比性）
+- 不做多跳历史链（只对比"最近一次"，不做完整时间序列的顾问诊断趋势图——这块需求已由 Dashboard 的原始指标趋势线覆盖）

docs/superpowers/specs/2026-07-02-token-usage-tracking-design.md

@@ -0,0 +1,216 @@
+# Judge LLM Token 用量追踪设计
+
+**日期**: 2026-07-02
+**状态**: 已批准，待实现
+**范围**: 按 LLM 模型累计记录评测过程中判官（judge）与向量化（embedding）模型的原始 token 用量（不换算金额），持久化到 run 产物并在报告详情页展示（架构设计 §10.5）。
+
+---
+
+## 1. 背景与目标
+
+架构设计 §10.5 明确要求：
+
+> 此类 LLM 评判指标对每条样本产生多次 LLM 调用，规模化运行的 token 开销需预先评估——一次数百条的回归，叠加每条多次评判调用，量级不小。
+
+当前全链路（`MetricScore` / `EvaluationResult` / `metadata.json` / `summary.md` / Web 报告）**没有任何 token 用量记录**，规模化跑评测时无法评估实际消耗。本设计新增能力：
+
+- 按**模型名**累计 input/output token 数与调用次数（不做金额换算，YAGNI）。
+- 覆盖 RAGAS 评分调用（judge model + embedding model）与优化顾问 LLM 分析调用（`advisor/llm_analyzer.py`）。
+- 只统计**落盘成 run 目录**的路径：CLI `main.py --scenario`、`/api/score/async`、`/api/score/session_async`；不落盘的同步 `/api/score` 不纳入范围。
+- 持久化到 `metadata.json`，摘要写入 `summary.md`，报告详情页新增"Token 用量"面板。
+
+**成功标准**：
+- 跑完一次评测（任一入口），`metadata.json` 里能看到按模型分组的 `{input_tokens, output_tokens, calls}`。
+- session_async 多次调用同一 session，token 用量随调用**累加**，不是只保留最后一次。
+- 网关不返回 `usage` 字段时静默跳过，不影响评分主流程、不抛异常。
+- 现有评分行为、既有测试逐字节不变（这是纯增量的可观测性功能）。
+
+---
+
+## 2. 现状确认（已核实的技术事实）
+
+1. RAGAS 0.4.3 的 `ragas/cost.py`（`TokenUsageParser` / `CostCallbackHandler`）只服务旧版 langchain `evaluate()` 回调链路，**新版 `ragas.metrics.collections` + instructor 路径完全不走这条链路**，无法复用。
+2. `ragas.llms.base.llm_factory()` 内部用 `instructor.from_openai(client, mode=instructor.Mode.JSON)` 包装我们传入的 `AsyncOpenAI`，`InstructorLLM.generate/agenerate` 只在返回前触发 RAGAS 自己的匿名遥测事件（`track(LLMUsageEvent(...))`），**不是真实 token 计数**，也不能拿到。
+3. `embedding_factory(provider="openai", ...)` 走的是普通 `AsyncOpenAI.embeddings.create()`，不经过 instructor，跟 chat completions 是两条不同的调用路径 —— 说明"在 instructor 层挂钩子"这条路无法同时覆盖 judge 和 embedding 两种调用，必须在更底层（HTTP 层）统一拦截。
+4. `webapp/services/inline_scorer.py` 的 `InlineScorer` 用 `(judge_model, embedding_model)` 做 key **跨请求缓存** `(llm, embeddings)` 客户端对象；`/api/score`、`/api/score/async`（`score_job_manager.py`）、`/api/score/session_async`（`session_score_manager.py`）三个接口都调用同一个 `inline_scorer` 单例。**因此统计器不能绑定在客户端构造时刻，必须按每次调用动态切换**。
+5. 三条落盘路径最终都调用同一个 `rag_eval/reporting/writers.py::write_run_artifacts(EvaluationResult)` 生成 `metadata.json`，这是唯一需要改动的持久化落点（DRY）。
+
+---
+
+## 3. 架构与数据流
+
+### 3.1 核心组件：`rag_eval/metrics/token_tracker.py`（新文件）
+
+```python
+@dataclass
+class TokenUsageTracker:
+    """按模型名累计一次 run/一次请求范围内的 token 用量。"""
+    _totals: dict[str, dict[str, int]] = field(default_factory=dict)
+
+    def record(self, model: str, input_tokens: int, output_tokens: int) -> None:
+        """累加一次 API 调用的用量（未知模型名用 "unknown"）。"""
+
+    def as_dict(self) -> dict[str, dict[str, int]]:
+        """返回 {model: {"input_tokens": int, "output_tokens": int, "calls": int}}。"""
+
+    def merge_into(self, existing: dict[str, dict[str, int]]) -> dict[str, dict[str, int]]:
+        """把当前统计合并进已有汇总（用于 session 累加），返回合并后的新 dict。"""
+```
+
+一个模块级 `contextvars.ContextVar[TokenUsageTracker | None]`（默认 `None`）保存"当前活跃统计器"。配套：
+
+```python
+@contextmanager
+def track_token_usage() -> Iterator[TokenUsageTracker]:
+    """创建并激活一个新 tracker，退出时还原上下文，返回 tracker 供调用方读取汇总。"""
+
+def get_current_tracker() -> TokenUsageTracker | None:
+    """供 HTTP 钩子读取；未激活时返回 None（即不统计，用于 /api/score 等排除路径）。"""
+```
+
+`contextvars.ContextVar` 天然按 async task / 线程隔离：`session_score_manager` 的线程池并发场景、`MetricPipeline.score_samples()` 内部 `asyncio.gather` 并发场景都能正确工作 —— 同一次 run 内的并发样本共享同一个 tracker（这正是我们想要的：整个 run 的用量汇总到一起），不同 run/请求之间互不干扰。
+
+### 3.2 HTTP 层拦截：`attach_usage_hook(client: AsyncOpenAI) -> None`
+
+在 `rag_eval/metrics/factory.py` 新增辅助函数，对**已构造好**的 `AsyncOpenAI` 客户端做构造后挂载（不改变现有 `AsyncOpenAI(**kwargs)` 调用点的参数，只在其后追加一行）：
+
+```python
+async def _usage_response_hook(response: httpx.Response) -> None:
+    tracker = get_current_tracker()
+    if tracker is None:
+        return
+    try:
+        await response.aread()
+        data = response.json()
+        usage = data.get("usage") or {}
+        model = data.get("model") or "unknown"
+        tracker.record(
+            model,
+            int(usage.get("prompt_tokens", 0)),
+            int(usage.get("completion_tokens", 0)),
+        )
+    except Exception:
+        pass  # 可观测性钩子绝不能影响评分主流程
+
+
+def attach_usage_hook(client: AsyncOpenAI) -> None:
+    """给已构造的 AsyncOpenAI 客户端追加 token 用量响应钩子（幂等，可重复调用）。"""
+    httpx_client = client._client  # openai SDK 内部持有的 httpx.AsyncClient
+    hooks = httpx_client.event_hooks.setdefault("response", [])
+    if _usage_response_hook not in hooks:
+        hooks.append(_usage_response_hook)
+```
+
+调用方式：在 `AsyncOpenAI(**kwargs)` 构造完成后紧跟一行 `attach_usage_hook(llm_client)`。已在当前锁定版本 `openai==1.102.0` 上验证：`AsyncOpenAI()._client` 是 `AsyncHttpxClientWrapper`（`httpx.AsyncClient` 子类），且自带 `event_hooks = {"request": [], "response": []}` 可直接追加，无需额外适配。
+
+- 用响应体里网关自己回填的 `model` 字段分组（chat completions 和 embeddings 响应体都带 `model` + `usage`），而不是我们请求时传的模型名 —— 这样即使 judge/embedding 共享同一个 client（`build_models()` 中 base_url/api_key 相同时的既有优化），也能按请求本身的 `model` 字段正确拆开统计。
+- 网关不返回 `usage`/`model`、返回非 JSON、或任何解析异常，一律静默跳过（`except Exception: pass`），钩子失败不能让评分调用报错。
+
+### 3.3 挂载点
+
+- `rag_eval/metrics/factory.py::build_models()`：构造 `llm_client`、`emb_client` 时都调用 `attach_usage_hook`（对共享 client 场景只需挂一次）。
+- `rag_eval/advisor/llm_analyzer.py::analyze()`：构造自己的 `AsyncOpenAI` 时同样调用 `attach_usage_hook`（从 `rag_eval.metrics.factory` 导入复用，不重复实现)。
+
+---
+
+## 4. 三条入口路径的接入方式
+
+### 4.1 CLI `main.py --scenario`（`rag_eval/execution/evaluator.py`）
+
+评测主流程包一层：
+
+```python
+with track_token_usage() as tracker:
+    # ...现有的 build_metric_pipeline + score_samples 流程...
+result.token_usage = tracker.as_dict()
+```
+
+`EvaluationResult` 新增字段：
+
+```python
+token_usage: dict[str, dict[str, int]] = field(default_factory=dict)
+```
+
+### 4.2 `/api/score/async`（`webapp/services/score_job_manager.py`）
+
+`_run()` 里调用 `inline_scorer.score(...)` 的地方外面包 `track_token_usage()`，把汇总结果塞进它构造的 `EvaluationResult.token_usage` 再调 `write_run_artifacts()`。单样本、单次落盘，语义等同 CLI。
+
+### 4.3 `/api/score/session_async`（`webapp/services/session_score_manager.py`）—— 累加语义
+
+session 场景每次调用都会重写整个 `metadata.json`（现有逻辑：重新汇总全部累计样本行）。Token 用量需要**跨调用累加**而不是只保留最后一次：
+
+1. 调用前：`existing = _read_json(metadata_path).get("token_usage", {})`。
+2. 本次调用用 `track_token_usage()` 统计。
+3. 写回前：`merged = tracker.merge_into(existing)`，赋给 `EvaluationResult.token_usage = merged`。
+
+这与 `scores.csv` 逐次追加行的既有累加模式保持一致。
+
+### 4.4 `write_run_artifacts()`（`rag_eval/reporting/writers.py`）
+
+`metadata` dict 新增一个键：
+
+```python
+"token_usage": result.token_usage,  # {} 表示未统计或统计为空
+```
+
+`summary.md`（`rag_eval/reporting/summary.py::build_summary_markdown`）追加一段：
+
+```
+## Token 用量
+
+| 模型 | input_tokens | output_tokens | 调用次数 |
+|---|---|---|---|
+| gpt-5 | 12450 | 3200 | 60 |
+| Qwen/Qwen3-Embedding-4B | 45000 | 0 | 30 |
+```
+
+`token_usage` 为空时输出 `未记录 token 用量。`（保持既有"空数据降级文案"风格，参考 `_table_from_frame` 对空 DataFrame 的处理）。
+
+---
+
+## 5. 报告层与 Web UI
+
+- `webapp/models.py::ReportData` 新增字段：
+  ```python
+  token_usage: dict[str, dict[str, int]] = Field(default_factory=dict)
+  ```
+- `webapp/services/report_builder.py::build_report()`：从 `run_reader._read_json(run_dir / "metadata.json")` 读出的 `metadata` 里取 `token_usage` 塞进 `ReportData`（此函数已经在读 `metadata.json` 取 `run_id`，顺手多取一个字段即可，无需新增 IO）。
+- `webapp/static/js/report.js`：新增 `renderTokenUsage(report)`，仿照 `renderGroupings` 的表格渲染方式，在报告详情页新增一个"Token 用量"面板：按模型一行，列为 input/output/调用次数；`token_usage` 为空时显示"暂无 token 用量数据"。对应在 `index.html` 加一个面板容器 `token-usage-wrap`，在 `Report.render()` 里追加调用。
+
+---
+
+## 6. 错误处理与边界情况
+
+| 情况 | 处理 |
+|---|---|
+| 网关响应无 `usage` 字段 | 钩子静默跳过，不记录、不报错 |
+| 网关响应非 JSON / 读取异常 | `except Exception: pass`，钩子绝不向上抛异常 |
+| 未激活 tracker（如未来有调用方忘记包 `track_token_usage()`，或 `/api/score` 这类明确排除的路径） | `get_current_tracker()` 返回 `None`，钩子直接返回，等同于关闭统计 |
+| `judge_model == embedding_model` 或共享同一个 `AsyncOpenAI` 客户端（`build_models()` 现有优化） | 用响应体 `model` 字段区分，两个模型各自正确累加，无需为共享客户端特殊处理 |
+| 旧的 `metadata.json`（无 `token_usage` 键，历史 run） | `report_builder` 用 `metadata.get("token_usage", {})`，前端空数据降级展示，不报错 |
+| session_async 并发多个不同 session | 各自线程内 `contextvars` 独立，互不干扰；同一 session 的 `metadata.json` 读-改-写已有per-session锁（`threading.Lock`）保护，新增的 `token_usage` 合并逻辑复用这把锁 |
+
+---
+
+## 7. 测试策略
+
+新增/修改测试文件：
+
+- `tests/test_token_tracker.py`：`TokenUsageTracker.record/as_dict/merge_into` 纯函数单测；`track_token_usage()` 上下文管理器激活/还原行为；未激活时 `get_current_tracker()` 返回 `None`。
+- `tests/test_token_usage_hook.py`：用假的 `httpx.Response`（或轻量 mock transport）验证 `_usage_response_hook` 在有/无 `usage` 字段、非 JSON 响应时的行为；验证 `attach_usage_hook` 正确挂载到 `AsyncOpenAI` 客户端。
+- `tests/test_evaluator.py` 或现有 `test_offline_eval.py` / `test_online_eval.py`：扩展断言 `EvaluationResult.token_usage` 字段存在且结构正确（mock LLM 客户端返回带 `usage` 的响应）。
+- `tests/webapp/test_score_jobs_api.py`：断言 `/api/score/async` 产出的 `metadata.json` 含 `token_usage`。
+- `tests/webapp/test_session_score_jobs_api.py`：断言连续两次调用同一 session，`token_usage` 按模型**累加**而非覆盖。
+- `tests/test_webapp_report_builder.py`：断言 `ReportData.token_usage` 从 `metadata.json` 正确透传；`metadata.json` 缺失该键时默认空 dict。
+
+全部使用 mock LLM 客户端 / 假 httpx 响应，不依赖真实网络调用（遵循仓库现有测试约定）。
+
+---
+
+## 8. 非目标（本轮明确不做）
+
+- 不做金额（$/¥）换算，不改 LLM Profile 增加单价字段（用户已确认后续再说）。
+- 不统计不落盘的同步 `/api/score` 接口。
+- 不统计 `dataset_builder` 题库生成路径的 token 用量（范围已与用户确认排除）。
+- 不做事前成本预估/dry-run 采样估算（用户已确认只要事后按模型累计的原始计数）。
+- 不改变现有评分逻辑、指标计算或既有 API 响应结构（这是纯增量的可观测性能力）。

rag_eval/advisor/__init__.py

@@ -20,7 +20,10 @@ __all__ = ["run_advisor", "Diagnosis", "diagnose"]
 def run_advisor(
     result: EvaluationResult,
     scenario: Scenario,
-    llm: Any,
+    llm: Any = None,
+    *,
+    settings: Any | None = None,
+    chat_client: Any | None = None,
 ) -> None:
     """Run the full optimization advisor pipeline after an evaluation completes.
 
@@ -30,7 +33,10 @@ def run_advisor(
     Args:
         result: Completed EvaluationResult from Evaluator.evaluate().
         scenario: The resolved Scenario (provides metrics, judge_model, output_dir).
-        llm: Pre-built RAGAS LLM instance (from build_models()) for LLM analysis.
+        llm: Deprecated/unused — kept for backward-compatible call sites. The
+            advisor now issues its own direct LLM call resolved from judge_model.
+        settings: Optional EvaluationSettings; defaults to EvaluationSettings().
+        chat_client: Optional pre-built chat client (used by tests to avoid network).
     """
     if not scenario.optimization_advisor:
         return
@@ -38,6 +44,10 @@ def run_advisor(
     logger.info("[advisor] starting optimization analysis  scenario=%s", scenario.scenario_name)
 
     try:
+        if settings is None:
+            from rag_eval.settings import EvaluationSettings
+            settings = EvaluationSettings()
+
         artifact_paths = build_artifact_paths(scenario.output_dir, result.run_id)
         if artifact_paths.advice_md is None:
             logger.warning("[advisor] advice_md path not set in RunArtifactPaths — skipping")
@@ -47,7 +57,15 @@ def run_advisor(
         logger.info("[advisor] rule diagnosis complete: %d metric(s) triggered", len(diagnoses))
 
         if diagnoses:
-            llm_markdown = asyncio.run(analyze(diagnoses, llm, scenario.scenario_name))
+            llm_markdown = asyncio.run(
+                analyze(
+                    diagnoses,
+                    scenario.scenario_name,
+                    scenario.judge_model,
+                    settings,
+                    chat_client=chat_client,
+                )
+            )
         else:
             llm_markdown = ""
 

rag_eval/advisor/llm_analyzer.py

@@ -1,4 +1,11 @@
-"""LLM-powered analysis of rule diagnostics and low-score samples."""
+"""LLM-powered analysis of rule diagnostics and low-score samples.
+
+The analyzer issues a single, direct OpenAI-compatible chat call (no langchain,
+no instructor structured output) to produce a detailed Chinese optimization
+report. It deliberately uses plain text generation so it works with any
+OpenAI-compatible gateway and never trips the structured-output response-shape
+issues seen with the scoring path.
+"""
 from __future__ import annotations
 
 import logging
@@ -8,27 +15,41 @@ from .rules import Diagnosis
 
 logger = logging.getLogger("rag_eval.advisor")
 
+# Worked-example oriented prompt: forces per-question decomposition (为什么低 →
+# 拆解 → 怎么改) plus cross-metric causal reasoning, so the report is concrete
+# instead of a generic restatement of the static rule templates.
 _PROMPT_TEMPLATE = """\
-你是一个 RAG 系统优化专家，正在分析西门子医疗 CT 文档问答系统的评测结果。
+你是一位资深的 RAG（检索增强生成）系统优化专家，正在分析西门子医疗 CT 文档问答系统的 RAGAS 评测结果。
-请用中文撰写一份优化建议报告，格式为 Markdown。
+请基于以下诊断数据与低分样本，用中文撰写一份**详细、具体、可落地**的优化建议报告（Markdown 格式）。
 
 ## 评测诊断摘要
 
 {diagnosis_summary}
 
-## 低分样本示例
+## 低分样本明细（含检索片段 contexts，用于定位问题出在检索还是生成环节）
 
 {low_sample_text}
 
-## 报告要求
+## 撰写要求
 
-1. 按指标分节（## 指标名  [严重程度]），先解释"为什么低"（结合低分样本具体分析），再给出"具体怎么改"
+1. **按指标分节**：每个指标一个 `## 指标名 [严重程度]` 小节。
-2. 严重程度说明：critical=严重（<阈值50%），warning=警告（<阈值70%），low=待优化（低于0.85，有提升空间）
+2. **每节必须包含「举例拆解」**：从该指标的低分样本中挑 1-2 个最典型的，逐条按如下结构拆解：
-3. "具体怎么改"要结合低分样本的实际内容，而不只是泛泛建议
+   - **问题**：简述该样本的 question
-4. 最后写一节 **## 优先优化次序**，按性价比排序（不增加 LLM 调用次数的优化优先），critical 和 warning 项优先于 low 项
+   - **当前得分**：该样本在此指标上的分数
-5. 语言简洁，面向工程师，不要废话，不要重复列表内容
+   - **为什么低**：结合该样本的 answer / 检索片段 contexts / 标准答案 ground_truth，**具体指出**问题所在
+     （例如：答案里哪句话没有被检索片段支持；检索片段里缺了哪个关键信息；答案偏离了问题的哪个点）
+   - **拆解定位**：判断问题出在「检索环节」「生成环节」还是「两者兼有」
+   - **优化动作**：针对这个具体样本，给出 1-3 条**可操作**的改法，不要泛泛而谈
+3. **跨指标关联分析**：若多个指标同时偏低，分析其因果关系。例如：
+   - faithfulness 低且 context_recall 低 → 多半是检索缺失关键信息，导致模型臆造（幻觉）
+   - context_recall 正常但 context_precision 低 → 检索引入噪声，稀释了有效信息
+   - faithfulness 低但 context_recall 高 → 生成环节 grounding 不足，需收紧生成 prompt
+4. 最后写一节 `## 优先优化次序`：按性价比排序（不增加 LLM 调用次数的优化优先；critical/warning 优先于 low）。
+5. 语言简洁，面向工程师，重点是「具体、可操作」。不要复述本提示词，不要无意义的客套。
 
-只输出 Markdown 报告正文，不要任何前置说明。
+严重程度说明：critical=严重（远低于阈值），warning=警告（低于阈值），low=待优化（达标但低于 0.85，仍有提升空间）。
+
+只输出 Markdown 报告正文，不要任何前置说明或代码块包裹。
 """
 
 
@@ -38,8 +59,15 @@ _SEVERITY_LABEL_ZH: dict[str, str] = {
     "low": "待优化",
 }
 
+# Per-sample text limits keep the prompt bounded regardless of sample size.
+_ANSWER_LIMIT = 400
+_GT_LIMIT = 300
+_CONTEXT_LIMIT = 600
+_CONTEXT_SEPARATOR = " |||| "
+
 
 def _build_diagnosis_summary(diagnoses: list[Diagnosis]) -> str:
+    """Render the per-metric diagnosis block fed to the LLM."""
     lines = []
     for d in diagnoses:
         direction = "（越低越好）" if d.metric == "noise_sensitivity" else ""
@@ -53,7 +81,22 @@ def _build_diagnosis_summary(diagnoses: list[Diagnosis]) -> str:
     return "\n".join(lines)
 
 
+def _format_contexts(raw: Any) -> str:
+    """Render a sample's contexts (stored as a joined string) as a short list."""
+    text = str(raw or "").strip()
+    if not text:
+        return "（无检索片段）"
+    parts = [p.strip() for p in text.split(_CONTEXT_SEPARATOR) if p.strip()]
+    if not parts:
+        return "（无检索片段）"
+    rendered = "; ".join(f"[{i + 1}] {p}" for i, p in enumerate(parts))
+    if len(rendered) > _CONTEXT_LIMIT:
+        rendered = rendered[:_CONTEXT_LIMIT] + "…"
+    return rendered
+
+
 def _build_low_sample_text(diagnoses: list[Diagnosis]) -> str:
+    """Render low-score samples (now including contexts) for grounding analysis."""
     lines = []
     for d in diagnoses:
         if not d.low_samples:
@@ -61,24 +104,63 @@ def _build_low_sample_text(diagnoses: list[Diagnosis]) -> str:
         lines.append(f"### {d.metric} 低分样本（最多 3 条）")
         for i, s in enumerate(d.low_samples, 1):
             score = s.get(d.metric, "N/A")
-            lines.append(f"\n**样本 {i}**（分数={score}）")
+            lines.append(f"\n**样本 {i}**（{d.metric}={score}）")
-            lines.append(f"- 问题：{s.get('question', '')}")
+            lines.append(f"- 问题 question：{s.get('question', '')}")
-            lines.append(f"- 回答：{s.get('answer', '')[:300]}")
+            lines.append(f"- 生成答案 answer：{str(s.get('answer', ''))[:_ANSWER_LIMIT]}")
-            lines.append(f"- 标准答案：{s.get('ground_truth', '')[:200]}")
+            lines.append(f"- 检索片段 contexts：{_format_contexts(s.get('contexts'))}")
+            lines.append(f"- 标准答案 ground_truth：{str(s.get('ground_truth', ''))[:_GT_LIMIT]}")
     return "\n".join(lines)
 
 
+def _is_reasoning_model(model: str) -> bool:
+    """Return True for OpenAI reasoning models (gpt-5+/o-series/codex-mini).
+
+    These require `max_completion_tokens` instead of `max_tokens` and do not
+    accept a custom temperature. Mirrors RAGAS's own detection logic so advice
+    generation stays consistent with the scoring path.
+    """
+    m = (model or "").lower()
+    # O-series: o1..o9 optionally followed by - or _
+    if len(m) >= 2 and m[0] == "o" and m[1] in "123456789":
+        if len(m) == 2 or m[2] in ("-", "_"):
+            return True
+    # GPT-5 through GPT-19 generation
+    if m.startswith("gpt-"):
+        version_str = m[4:].split("-")[0].split("_")[0].split(".")[0]
+        try:
+            if 5 <= int(version_str) <= 19:
+                return True
+        except ValueError:
+            pass
+    if m == "codex-mini":
+        return True
+    return False
+
+
+def _chat_token_kwargs(model: str, max_tokens: int) -> dict[str, Any]:
+    """Return the completion-budget kwargs appropriate for the model family."""
+    if _is_reasoning_model(model):
+        # Reasoning models: only max_completion_tokens; temperature must stay default (1).
+        return {"max_completion_tokens": max_tokens}
+    return {"max_tokens": max_tokens, "temperature": 0.2}
+
+
 async def analyze(
     diagnoses: list[Diagnosis],
-    llm: Any,
     scenario_name: str,
+    judge_model: str,
+    settings: Any,
+    *,
+    chat_client: Any | None = None,
 ) -> str:
-    """Call the judge LLM to generate a Chinese optimization report.
+    """Call the judge LLM directly to generate a Chinese optimization report.
 
     Args:
         diagnoses: Non-empty list of Diagnosis from rules.diagnose().
-        llm: RAGAS LLM wrapper (has .agenerate() method).
         scenario_name: Used only for logging.
+        judge_model: Model name to call (also selects the matching LLM profile).
+        settings: EvaluationSettings (supplies client kwargs + token budget).
+        chat_client: Optional pre-built AsyncOpenAI-compatible client (for tests).
 
     Returns:
         LLM-generated Markdown string, or "" on failure (triggers writer fallback).
@@ -86,22 +168,48 @@ async def analyze(
     if not diagnoses:
         return ""
 
-    diagnosis_summary = _build_diagnosis_summary(diagnoses)
-    low_sample_text = _build_low_sample_text(diagnoses)
     prompt = _PROMPT_TEMPLATE.format(
-        diagnosis_summary=diagnosis_summary,
+        diagnosis_summary=_build_diagnosis_summary(diagnoses),
-        low_sample_text=low_sample_text,
+        low_sample_text=_build_low_sample_text(diagnoses),
     )
 
     try:
         logger.info("[advisor] calling LLM for optimization analysis  scenario=%s", scenario_name)
-        from langchain_core.messages import HumanMessage
+        client = chat_client
-        # Use the underlying langchain chat model directly (RAGAS LangchainLLMWrapper wraps BaseChatModel)
+        owns_client = False
-        response = await llm.langchain_llm.ainvoke([HumanMessage(content=prompt)])
+        if client is None:
-        text = response.content.strip()
+            from openai import AsyncOpenAI
+
+            from rag_eval.metrics.factory import attach_usage_hook, resolve_openai_client_kwargs
+
+            client = AsyncOpenAI(**resolve_openai_client_kwargs(judge_model, settings))
+            attach_usage_hook(client)
+            owns_client = True
+
+        try:
+            # Advice is a longer document than per-metric scoring; give it headroom.
+            max_tokens = max(2048, int(getattr(settings, "ragas_llm_max_tokens", 4096) or 4096))
+            token_kwargs = _chat_token_kwargs(judge_model, max_tokens)
+
+            response = await client.chat.completions.create(
+                model=judge_model,
+                messages=[{"role": "user", "content": prompt}],
+                **token_kwargs,
+            )
+            text = (response.choices[0].message.content or "").strip()
             logger.info("[advisor] LLM analysis complete  chars=%d", len(text))
             return text
-    except Exception as exc:
+        finally:
+            # Close the httpx connection pool inside THIS event loop. run_advisor
+            # drives analyze() via asyncio.run(), which closes the loop on return;
+            # a later GC-time aclose() on the dead loop would otherwise log
+            # "RuntimeError: Event loop is closed". Only close clients we created.
+            if owns_client:
+                try:
+                    await client.close()
+                except Exception:  # noqa: BLE001
+                    pass
+    except Exception as exc:  # noqa: BLE001
         logger.warning(
             "[advisor] LLM analysis failed (%s: %s) — falling back to rule report",
             type(exc).__name__, exc,

rag_eval/advisor/rules.py

@@ -159,7 +159,8 @@ def _select_low_samples(
     valid = [r for r in rows if metric in r and not math.isnan(float(r[metric]))]
     sorted_rows = sorted(valid, key=lambda r: float(r[metric]), reverse=not higher_is_better)
     worst = sorted_rows[:top_n]
-    keep_keys = {"sample_id", "question", "answer", "ground_truth", metric}
+    # contexts is included so the LLM/fallback can judge grounding (retrieval vs generation).
+    keep_keys = {"sample_id", "question", "answer", "ground_truth", "contexts", metric}
     return [{k: v for k, v in row.items() if k in keep_keys} for row in worst]
 
 

rag_eval/advisor/writer.py

@@ -29,10 +29,15 @@ def _format_log_summary(diagnoses: list[Diagnosis], advice_path: Path) -> str:
 
 
 def _build_fallback_report(diagnoses: list[Diagnosis]) -> str:
-    """Build a rules-only report when LLM analysis is unavailable."""
+    """Build a rules-only report when LLM analysis is unavailable.
+
+    Even without the LLM, embed each metric's worst sample(s) — question,
+    answer, ground truth — so the advice still references concrete problems
+    instead of reading as a purely generic template.
+    """
     if not diagnoses:
         return ""
-    lines = ["## 规则诊断（LLM 分析不可用）\n"]
+    lines = ["## 规则诊断（LLM 分析不可用，以下为规则引擎输出）\n"]
     for d in diagnoses:
         label = _SEVERITY_LABEL.get(d.severity, d.severity)
         lines.append(f"### {d.metric}  [{label}]  均值={d.mean_score:.4f}")
@@ -42,6 +47,22 @@ def _build_fallback_report(diagnoses: list[Diagnosis]) -> str:
         lines.append("\n**建议动作：**")
         for action in d.suggested_actions:
             lines.append(f"- {action}")
+        if d.low_samples:
+            lines.append("\n**低分样本举例拆解：**")
+            for i, sample in enumerate(d.low_samples, 1):
+                score = sample.get(d.metric, "N/A")
+                question = str(sample.get("question", "")).strip() or "（无问题文本）"
+                lines.append(f"\n- **样本 {i}**（{d.metric}={score}）问题：{question}")
+                answer = str(sample.get("answer", "")).strip()
+                if answer:
+                    lines.append(f"  - 生成答案：{answer[:200]}")
+                ground_truth = str(sample.get("ground_truth", "")).strip()
+                if ground_truth:
+                    lines.append(f"  - 标准答案：{ground_truth[:160]}")
+                lines.append(
+                    f"  - 拆解：该样本 {d.metric} 偏低，请对照上述「可能原因 / 建议动作」"
+                    f"重点排查本问题的检索片段与生成答案。"
+                )
         lines.append("")
     return "\n".join(lines)
 

rag_eval/config/loader.py

@@ -64,6 +64,7 @@ def load_scenario(path: str | Path) -> Scenario:
         optimization_advisor=model.optimization_advisor,
         metric_weights=dict(model.metric_weights),
         doc_weights=dict(model.doc_weights),
+        judge_language=model.judge_language,
     )
     # Run cross-field checks after all relative paths have been resolved.
     validate_scenario(scenario)

rag_eval/config/schema.py

@@ -57,6 +57,7 @@ class ScenarioModel(BaseModel):
     optimization_advisor: bool = False
     metric_weights: dict[str, float] = Field(default_factory=dict)
     doc_weights: dict[str, float] = Field(default_factory=dict)
+    judge_language: Literal["en", "zh"] | None = None
 
     @field_validator("metrics")
     @classmethod

rag_eval/execution/evaluator.py

@@ -12,6 +12,7 @@ from rag_eval.datasets.loader import load_dataset_records
 from rag_eval.datasets.normalizers import normalize_records
 from rag_eval.execution.concurrency import gather_with_limit
 from rag_eval.metrics.pipeline import MetricPipeline
+from rag_eval.metrics.token_tracker import track_token_usage
 from rag_eval.metrics.weights import compute_weighted_score, resolve_weight
 from rag_eval.shared.models import EvaluationResult, InvalidSample, NormalizedSample, Scenario
 from rag_eval.shared.utils import utc_now_iso
@@ -67,6 +68,7 @@ class Evaluator:
 
         logger.info("[eval] scoring %d samples with metric pipeline ...", len(samples))
         t0 = time.monotonic()
+        with track_token_usage() as usage_tracker:
             metric_scores = asyncio.run(
                 self.metric_pipeline.score_samples(
                     samples,
@@ -75,6 +77,7 @@ class Evaluator:
             )
         elapsed = time.monotonic() - t0
         logger.info("[eval] metric scoring done  elapsed=%.1fs", elapsed)
+        logger.info("[eval] token_usage=%s", usage_tracker.as_dict())
 
         finished_at = utc_now_iso()
         score_rows = [self._merge_score(sample, score) for sample, score in zip(samples, metric_scores)]
@@ -99,6 +102,7 @@ class Evaluator:
             valid_samples=samples,
             invalid_samples=invalid_samples,
             score_rows=score_rows,
+            token_usage=usage_tracker.as_dict(),
         )
 
     async def _enrich_online_samples(

rag_eval/execution/runner.py

@@ -79,6 +79,6 @@ def run_scenario(
     logger.info("[runner] artifacts written for run_id=%s", result.run_id)
 
     # Optimization advisor — runs only if scenario.optimization_advisor is True.
-    run_advisor(result, scenario, llm)
+    run_advisor(result, scenario, settings=settings)
 
     return result

rag_eval/metrics/factory.py

@@ -2,14 +2,18 @@
 
 from __future__ import annotations
 
+import logging
 from typing import Any
 
+import httpx
 from openai import AsyncOpenAI
 
 from rag_eval.compat import ensure_ragas_import_compat
 from rag_eval.settings import EvaluationSettings
 from rag_eval.shared.models import Scenario
 
+from .token_tracker import get_current_tracker
+
 ensure_ragas_import_compat()
 
 from ragas.embeddings.base import embedding_factory
@@ -24,17 +28,63 @@ from ragas.metrics.collections import (
     SemanticSimilarity,
 )
 
+from .judge_prompts import localize_pipeline_prompts
 from .pipeline import MetricPipeline
 
 
+logger = logging.getLogger("rag_eval.metrics.factory")
+
+
+async def _usage_response_hook(response: httpx.Response) -> None:
+    """Record token usage from an OpenAI-compatible HTTP response, if a tracker is active.
+
+    Applies to both chat-completions and embeddings responses since both
+    return top-level `model` and `usage` fields in OpenAI-compatible APIs.
+    Never raises — a broken/incompatible gateway response must not affect scoring.
+    """
+    tracker = get_current_tracker()
+    if tracker is None:
+        return
+    try:
+        await response.aread()
+        data = response.json()
+        usage = data.get("usage")
+        if not usage:
+            # Gateway did not report usage at all — skip rather than record a
+            # misleading 0/0 call.
+            return
+        model = data.get("model") or "unknown"
+        tracker.record(
+            model,
+            int(usage.get("prompt_tokens", 0) or 0),
+            int(usage.get("completion_tokens", 0) or 0),
+        )
+    except Exception:  # noqa: BLE001
+        logger.debug("[factory] usage hook failed to parse response", exc_info=True)
+
+
+def attach_usage_hook(client: AsyncOpenAI) -> None:
+    """Attach the token-usage response hook to an AsyncOpenAI client (idempotent).
+
+    Safe to call multiple times on the same client (e.g. when judge and
+    embedding models share one client) — the hook is only appended once.
+    """
+    httpx_client = getattr(client, "_client", None)
+    if httpx_client is None or not hasattr(httpx_client, "event_hooks"):
+        return
+    hooks = httpx_client.event_hooks.setdefault("response", [])
+    if _usage_response_hook not in hooks:
+        hooks.append(_usage_response_hook)
+
+
 def _resolve_openai_client_kwargs(
-    judge_model: str,
+    model: str,
     settings: EvaluationSettings,
 ) -> dict[str, Any]:
     """Return AsyncOpenAI kwargs, preferring a matching LLM Profile over .env settings.
 
     Lookup order:
-      1. LLM Profile whose model name equals judge_model (exact match)
+      1. LLM Profile whose model name equals `model` (exact match)
       2. Fall back to EvaluationSettings (.env)
     """
     try:
@@ -42,19 +92,61 @@ def _resolve_openai_client_kwargs(
         from webapp.services.profile_manager import profile_manager
         profiles = profile_manager.list_all()
         for profile in profiles:
-            if profile.model == judge_model:
+            if profile.model == model:
                 kwargs: dict[str, Any] = {
                     "api_key": profile.api_key or "sk-placeholder",
                     "timeout": float(profile.timeout_seconds or 30),
                 }
                 if profile.base_url and profile.base_url.strip():
                     kwargs["base_url"] = profile.base_url.strip()
+                logger.debug(
+                    "[factory] model=%s  source=profile  base_url=%s",
+                    model, kwargs.get("base_url", "(not set, using default)")
+                )
                 return kwargs
-    except Exception:  # noqa: BLE001
+    except Exception as exc:  # noqa: BLE001
         # If profile lookup fails for any reason, fall through to .env settings.
-        pass
+        logger.warning("[factory] profile lookup failed for model=%s: %s", model, exc)
 
-    return settings.openai_client_kwargs
+    fallback = settings.openai_client_kwargs
+    logger.debug(
+        "[factory] model=%s  source=env  base_url=%s",
+        model, fallback.get("base_url", "(not set)")
+    )
+    return fallback
+
+
+def resolve_openai_client_kwargs(
+    judge_model: str,
+    settings: EvaluationSettings,
+) -> dict[str, Any]:
+    """Public accessor for profile-aware AsyncOpenAI kwargs (matched by model name).
+
+    Exposed so other components (e.g. the optimization advisor's direct LLM call)
+    can build a client that honors the same saved-profile/.env resolution used by
+    the scoring pipeline, instead of duplicating the lookup logic.
+    """
+    return _resolve_openai_client_kwargs(judge_model, settings)
+
+
+def build_metric_registry(llm: Any, embeddings: Any) -> dict[str, Any]:
+    """Instantiate the full set of supported RAGAS metrics keyed by canonical name.
+
+    Shared by the scenario pipeline, the inline scorer, and the prompt-cache
+    bootstrap so the metric set is defined in exactly one place.
+    """
+    return {
+        "faithfulness": Faithfulness(llm=llm),
+        "answer_relevancy": AnswerRelevancy(llm=llm, embeddings=embeddings),
+        "context_recall": ContextRecall(llm=llm),
+        "context_precision": ContextPrecision(llm=llm),
+        # NoiseSensitivity mode='relevant': sensitivity to noise from relevant contexts.
+        "noise_sensitivity": NoiseSensitivity(llm=llm),
+        # FactualCorrectness mode='f1': balances claim precision and recall vs. ground truth.
+        "factual_correctness": FactualCorrectness(llm=llm),
+        # SemanticSimilarity: embedding cosine between answer and ground truth (no LLM call).
+        "semantic_similarity": SemanticSimilarity(embeddings=embeddings),
+    }
 
 
 def build_models(
@@ -64,19 +156,34 @@ def build_models(
 ) -> tuple[Any, Any]:
     """Create the LLM and embedding clients required by the selected RAGAS metrics.
 
-    Dynamically resolves connection settings from the stored LLM Profiles first
+    Resolves connection settings independently for the judge LLM and the embedding
-    (matched by model name), falling back to .env settings when no profile matches.
+    model by looking up each in the stored LLM Profiles (matched by model name).
+    This allows judge_model and embedding_model to use different gateways / API keys.
+    Falls back to .env settings when no matching profile is found.
     """
-    client_kwargs = _resolve_openai_client_kwargs(judge_model, settings)
+    llm_kwargs = _resolve_openai_client_kwargs(judge_model, settings)
-    client = AsyncOpenAI(**client_kwargs)
+    emb_kwargs = _resolve_openai_client_kwargs(embedding_model, settings)
+
+    logger.info(
+        "[factory] build_models  judge=%s→%s  embedding=%s→%s",
+        judge_model, llm_kwargs.get("base_url", "(env default)"),
+        embedding_model, emb_kwargs.get("base_url", "(env default)"),
+    )
+
+    llm_client = AsyncOpenAI(**llm_kwargs)
+    attach_usage_hook(llm_client)
+    # Only allocate a second client when the embedding model needs different settings.
+    emb_client = AsyncOpenAI(**emb_kwargs) if emb_kwargs != llm_kwargs else llm_client
+    attach_usage_hook(emb_client)
+
     # RAGAS structured-output judge calls can be truncated by the upstream default
     # 1024 completion budget, especially for faithfulness and GPT-5 family models.
     llm = llm_factory(
         judge_model,
-        client=client,
+        client=llm_client,
         max_tokens=max(1, int(settings.ragas_llm_max_tokens)),
     )
-    embeddings = embedding_factory(provider="openai", model=embedding_model, client=client)
+    embeddings = embedding_factory(provider="openai", model=embedding_model, client=emb_client)
     return llm, embeddings
 
 
@@ -98,21 +205,13 @@ def build_metric_pipeline(
             settings,
         )
 
-    # Build the full registry once, then slice it by configured metric names.
+    # Build the full registry once using the shared factory, then slice by requested metrics.
-    registry: dict[str, Any] = {
+    registry = build_metric_registry(llm, embeddings)
-        "faithfulness": Faithfulness(llm=llm),
+    selected = {name: registry[name] for name in scenario.metrics}
-        "answer_relevancy": AnswerRelevancy(llm=llm, embeddings=embeddings),
+    # Apply language-specific judge prompt localization (no-op for 'en').
-        "context_recall": ContextRecall(llm=llm),
+    language = scenario.judge_language or settings.ragas_judge_language or "en"
-        "context_precision": ContextPrecision(llm=llm),
+    localize_pipeline_prompts(selected, language)
-        # Robustness / end-to-end metrics (架构设计 §10.2).
-        # NoiseSensitivity mode='relevant': sensitivity to noise from relevant contexts.
-        "noise_sensitivity": NoiseSensitivity(llm=llm),
-        # FactualCorrectness mode='f1': balances claim precision and recall vs. ground truth.
-        "factual_correctness": FactualCorrectness(llm=llm),
-        # SemanticSimilarity: embedding cosine between answer and ground truth (no LLM call).
-        "semantic_similarity": SemanticSimilarity(embeddings=embeddings),
-    }
     return MetricPipeline(
-        metrics={name: registry[name] for name in scenario.metrics},
+        metrics=selected,
         metric_timeout_seconds=settings.ragas_metric_timeout_seconds,
     )

rag_eval/metrics/judge_prompts.py

@@ -0,0 +1,154 @@
+"""Localize RAGAS collections judge prompts to a target language (e.g. Chinese).
+
+Loads committed, pre-translated prompt cache files from
+configs/judge_prompts/<language>/<metric>__<attr>.json and overrides each
+metric's prompt instance attributes in place. Missing, corrupt, or schema-drifted
+cache entries degrade gracefully to the built-in English prompt so scoring never
+breaks.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+logger = logging.getLogger("rag_eval.metrics.judge_prompts")
+
+_REPO_ROOT = Path(__file__).resolve().parents[2]
+CACHE_ROOT = _REPO_ROOT / "configs" / "judge_prompts"
+
+# Metric name -> prompt instance attribute names holding a BasePrompt.
+# Verified against RAGAS 0.4.3 collections source; semantic_similarity has none.
+# NOTE: noise_sensitivity's statement_prompt and faithfulness_prompt use function-based
+# to_string() (ragas.prompt.metrics.common) without instruction/examples attributes,
+# so they cannot be adapted with adapt() and are intentionally excluded here.
+METRIC_PROMPT_ATTRS: dict[str, tuple[str, ...]] = {
+    "faithfulness": ("statement_generator_prompt", "nli_statement_prompt"),
+    "answer_relevancy": ("prompt",),
+    "context_recall": ("prompt",),
+    "context_precision": ("prompt",),
+    "factual_correctness": ("prompt", "nli_prompt"),
+}
+
+# In-memory memoization of parsed cache files, keyed by (language, metric, attr).
+_MEMO: dict[tuple[str, str, str], dict | None] = {}
+
+
+@dataclass
+class LocalizationReport:
+    """Outcome of a localize_pipeline_prompts() call, for logging and tests."""
+
+    language: str
+    applied: list[str] = field(default_factory=list)
+    skipped: list[str] = field(default_factory=list)
+    warnings: list[str] = field(default_factory=list)
+
+
+def reset_cache() -> None:
+    """Clear the in-memory parsed-cache memo (used by tests)."""
+    _MEMO.clear()
+
+
+def prompt_source_hash(prompt: Any) -> str:
+    """Return a stable sha256 of a prompt's English instruction + examples."""
+    examples = [
+        {"input": inp.model_dump(), "output": out.model_dump()}
+        for inp, out in getattr(prompt, "examples", [])
+    ]
+    payload = json.dumps(
+        {"instruction": getattr(prompt, "instruction", ""), "examples": examples},
+        ensure_ascii=False,
+        sort_keys=True,
+    )
+    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def _cache_path(language: str, metric: str, attr: str) -> Path:
+    """Resolve the cache file path for one (language, metric, attr) triple."""
+    return CACHE_ROOT / language / f"{metric}__{attr}.json"
+
+
+def _load_cache_file(language: str, metric: str, attr: str) -> dict | None:
+    """Load and memoize a cache file; return None if absent or unreadable."""
+    key = (language, metric, attr)
+    if key in _MEMO:
+        return _MEMO[key]
+    path = _cache_path(language, metric, attr)
+    data: dict | None
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except FileNotFoundError:
+        data = None
+    except (OSError, json.JSONDecodeError) as exc:  # corrupt file: degrade to English
+        logger.warning("[judge_prompts] cache read failed %s: %s", path, exc)
+        data = None
+    _MEMO[key] = data
+    return data
+
+
+def apply_localized_prompt(prompt: Any, data: dict) -> None:
+    """Override a live prompt's instruction/examples/language from a cache dict.
+
+    Examples are rebuilt using the live prompt's input/output models, so an
+    upstream schema change raises here and is caught by the caller (which then
+    keeps the English prompt).
+    """
+    examples = [
+        (prompt.input_model(**ex["input"]), prompt.output_model(**ex["output"]))
+        for ex in data.get("examples", [])
+    ]
+    prompt.instruction = data["instruction"]
+    prompt.examples = examples
+    prompt.language = data.get("language", "chinese")
+
+
+def localize_pipeline_prompts(registry: dict[str, Any], language: str) -> LocalizationReport:
+    """Override judge prompts in `registry` with cached `language` translations.
+
+    `registry` maps metric name -> RAGAS metric instance. Only metrics in
+    METRIC_PROMPT_ATTRS are touched; unknown metrics and semantic_similarity are
+    left untouched. English ("en"/"english"/empty) is a no-op.
+    """
+    report = LocalizationReport(language=language)
+    normalized = (language or "en").strip().lower()
+    if normalized in ("", "en", "english"):
+        return report
+
+    for metric_name, attrs in METRIC_PROMPT_ATTRS.items():
+        metric = registry.get(metric_name)
+        if metric is None:
+            continue
+        for attr in attrs:
+            tag = f"{metric_name}.{attr}"
+            prompt = getattr(metric, attr, None)
+            if prompt is None:
+                report.skipped.append(tag)
+                continue
+            data = _load_cache_file(normalized, metric_name, attr)
+            if data is None:
+                report.skipped.append(tag)
+                report.warnings.append(f"missing cache for {tag}")
+                continue
+            # Drift detection: warn if the English source changed since caching.
+            if data.get("source_hash") and data["source_hash"] != prompt_source_hash(prompt):
+                report.warnings.append(f"stale cache for {tag} (regenerate)")
+            try:
+                apply_localized_prompt(prompt, data)
+                report.applied.append(tag)
+            except Exception as exc:  # noqa: BLE001  schema drift -> keep English
+                report.warnings.append(f"apply failed for {tag}: {exc}; kept english")
+
+    if report.warnings:
+        logger.warning(
+            "[judge_prompts] language=%s applied=%d skipped=%d warnings=%s",
+            normalized, len(report.applied), len(report.skipped), report.warnings,
+        )
+    else:
+        logger.info(
+            "[judge_prompts] language=%s applied=%d", normalized, len(report.applied)
+        )
+    return report

rag_eval/metrics/token_tracker.py

@@ -0,0 +1,82 @@
+"""Per-run token usage accumulation, keyed by model name.
+
+RAGAS 0.4.3's `ragas.metrics.collections` + instructor code path does not
+expose real token counts (`ragas/cost.py` only serves the legacy langchain
+`evaluate()` path). This module provides a context-scoped accumulator that
+the HTTP response hook in `rag_eval.metrics.factory` feeds into, so token
+counts survive across the AsyncOpenAI client caching used by InlineScorer
+(see docs/superpowers/specs/2026-07-02-token-usage-tracking-design.md).
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from typing import Iterator
+
+
+@dataclass
+class TokenUsageTracker:
+    """Accumulates input/output token counts and call counts, grouped by model name."""
+
+    _totals: dict[str, dict[str, int]] = field(default_factory=dict)
+
+    def record(self, model: str, input_tokens: int, output_tokens: int) -> None:
+        """Add one API call's usage to the running total for `model`."""
+        key = model or "unknown"
+        bucket = self._totals.setdefault(
+            key, {"input_tokens": 0, "output_tokens": 0, "calls": 0}
+        )
+        bucket["input_tokens"] += int(input_tokens)
+        bucket["output_tokens"] += int(output_tokens)
+        bucket["calls"] += 1
+
+    def as_dict(self) -> dict[str, dict[str, int]]:
+        """Return a plain-dict snapshot: {model: {input_tokens, output_tokens, calls}}."""
+        return {model: dict(usage) for model, usage in self._totals.items()}
+
+    def merge_into(self, existing: dict[str, dict[str, int]]) -> dict[str, dict[str, int]]:
+        """Return a new dict combining `existing` accumulated totals with this tracker's totals.
+
+        Used by session-scoped scoring (one call at a time) to keep a running
+        total across multiple calls instead of overwriting with just the latest call.
+        Does not mutate `existing`.
+        """
+        merged: dict[str, dict[str, int]] = {
+            model: dict(usage) for model, usage in existing.items()
+        }
+        for model, usage in self.as_dict().items():
+            bucket = merged.setdefault(
+                model, {"input_tokens": 0, "output_tokens": 0, "calls": 0}
+            )
+            bucket["input_tokens"] += usage["input_tokens"]
+            bucket["output_tokens"] += usage["output_tokens"]
+            bucket["calls"] += usage["calls"]
+        return merged
+
+
+_current_tracker: ContextVar[TokenUsageTracker | None] = ContextVar(
+    "_current_tracker", default=None
+)
+
+
+@contextmanager
+def track_token_usage() -> Iterator[TokenUsageTracker]:
+    """Activate a fresh TokenUsageTracker for the duration of the `with` block.
+
+    Any AsyncOpenAI client with `attach_usage_hook()` applied that makes a
+    call while this context is active will have its usage recorded here.
+    Safe to nest; the innermost tracker is active within its own block.
+    """
+    tracker = TokenUsageTracker()
+    token = _current_tracker.set(tracker)
+    try:
+        yield tracker
+    finally:
+        _current_tracker.reset(token)
+
+
+def get_current_tracker() -> TokenUsageTracker | None:
+    """Return the currently active tracker, or None if no `track_token_usage()` block is active."""
+    return _current_tracker.get()

rag_eval/reporting/summary.py

@@ -35,6 +35,23 @@ def _table_from_frame(frame: pd.DataFrame) -> str:
     return "\n".join([header, separator, *body])
 
 
+def _token_usage_section(token_usage: dict[str, dict[str, int]]) -> list[str]:
+    """Render the '## Token 用量' section as a list of markdown lines."""
+    lines = ["", "## Token 用量", ""]
+    if not token_usage:
+        lines.append("未记录 token 用量。")
+        return lines
+    lines.append("| 模型 | input_tokens | output_tokens | 调用次数 |")
+    lines.append("|---|---|---|---|")
+    for model in sorted(token_usage):
+        usage = token_usage[model]
+        lines.append(
+            f"| {model} | {usage.get('input_tokens', 0)} "
+            f"| {usage.get('output_tokens', 0)} | {usage.get('calls', 0)} |"
+        )
+    return lines
+
+
 def build_summary_markdown(result: EvaluationResult) -> str:
     """Build the human-readable markdown summary written for each evaluation run."""
     total = len(result.valid_samples) + len(result.invalid_samples)
@@ -57,6 +74,7 @@ def build_summary_markdown(result: EvaluationResult) -> str:
 
     if scores.empty:
         lines.append("No valid samples were scored.")
+        lines.extend(_token_usage_section(result.token_usage))
         return "\n".join(lines) + "\n"
 
     score_rows_list = scores.to_dict(orient="records")
@@ -97,4 +115,5 @@ def build_summary_markdown(result: EvaluationResult) -> str:
         _table_from_frame(detail),
         "```",
     ])
+    lines.extend(_token_usage_section(result.token_usage))
     return "\n".join(lines) + "\n"

rag_eval/reporting/writers.py

@@ -45,6 +45,7 @@ def write_run_artifacts(result: EvaluationResult) -> None:
         "dataset": result.scenario.dataset.path.as_posix(),
         "valid_samples": len(result.valid_samples),
         "invalid_samples": len(result.invalid_samples),
+        "token_usage": result.token_usage,
     }
     artifact_paths.metadata_json.write_text(
         json.dumps(metadata, ensure_ascii=False, indent=2),

rag_eval/settings.py

@@ -22,6 +22,7 @@ class EvaluationSettings(BaseSettings):
     openai_api_key: str | None = Field(default=None, alias="OPENAI_API_KEY")
     openai_base_url: str = Field(default="http://6.86.80.4:30080/v1", alias="OPENAI_BASE_URL")
     ragas_judge_model: str = Field(default="gpt-5", alias="RAGAS_JUDGE_MODEL")
+    ragas_judge_language: str = Field(default="en", alias="RAGAS_JUDGE_LANGUAGE")
     ragas_embedding_model: str = Field(
         default="text-embedding-3-small",
         alias="RAGAS_EMBEDDING_MODEL",

rag_eval/shared/models.py

@@ -79,6 +79,9 @@ class Scenario:
     optimization_advisor: bool = False
     metric_weights: dict[str, float] = field(default_factory=dict)
     doc_weights: dict[str, float] = field(default_factory=dict)
+    # Language used for judge prompts: 'en' (default, English) or 'zh' (Chinese).
+    # None means "use the global settings default".
+    judge_language: str | None = None
 
     def snapshot(self) -> dict[str, Any]:
         """Serialize the scenario into a reporting-friendly dictionary snapshot."""
@@ -150,6 +153,9 @@ class EvaluationResult:
     valid_samples: list[NormalizedSample]
     invalid_samples: list[InvalidSample]
     score_rows: list[dict[str, Any]]
+    # Token usage grouped by model name: {model: {input_tokens, output_tokens, calls}}.
+    # Populated by callers via rag_eval.metrics.token_tracker.track_token_usage().
+    token_usage: dict[str, dict[str, int]] = field(default_factory=dict)
 
 
 @dataclass(slots=True)

scenarios/offline/siemens-pdf-offline-smoke.yaml

@@ -4,6 +4,8 @@ app_adapter: null
 dataset: ../../datasets/normalized/siemens_pdf_offline_smoke.csv
 judge_model: deepseek-v4-flash
 embedding_model: text-embedding-v3
+# 使用中文评判 prompt 以提升对中文语料的判定一致性（架构设计 §10.5）
+judge_language: zh
 metrics:
   - faithfulness
   - answer_relevancy

scenarios/online/siemens-pdf-question-bank-online.yaml

@@ -3,6 +3,8 @@ mode: online
 dataset: ../../datasets/raw/generated/siemens-pdf-question-bank.csv
 judge_model: deepseek-v4-flash
 embedding_model: text-embedding-v3
+# 使用中文评判 prompt 以提升对中文语料的判定一致性（架构设计 §10.5）
+judge_language: zh
 optimization_advisor: true
 metrics:
 - faithfulness

scripts/__init__.py

@@ -0,0 +1 @@
+# scripts package — exposes bootstrap and maintenance scripts as importable modules.

scripts/build_judge_prompt_cache.py

@@ -0,0 +1,119 @@
+"""One-off bootstrap: generate committed Chinese judge-prompt cache via RAGAS adapt().
+
+Usage:
+    python -m scripts.build_judge_prompt_cache --language zh [--judge-model <model>]
+
+Runs BasePrompt.adapt("chinese", llm, adapt_instruction=True) for every judge
+prompt of every LLM-scored metric and writes the result to
+configs/judge_prompts/<language>/<metric>__<attr>.json.
+
+Re-run after a RAGAS upgrade to refresh the cache (the localizer will warn
+via 'stale cache' log messages when the source hash changes).
+Requires a working judge LLM configured in settings / LLM profiles.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import logging
+from pathlib import Path
+from typing import Any
+
+from rag_eval.compat import ensure_ragas_import_compat
+from rag_eval.metrics.factory import build_metric_registry, build_models
+from rag_eval.metrics.judge_prompts import CACHE_ROOT, METRIC_PROMPT_ATTRS, prompt_source_hash
+from rag_eval.settings import EvaluationSettings
+
+ensure_ragas_import_compat()
+
+logger = logging.getLogger("scripts.build_judge_prompt_cache")
+
+# Map from judge_language short code to the natural-language name used by RAGAS adapt().
+_ADAPT_LANGUAGE: dict[str, str] = {
+    "zh": "chinese",
+    "en": "english",
+}
+
+
+def serialize_prompt(
+    adapted: Any,
+    source_hash: str,
+    metric: str,
+    attr: str,
+    language: str,
+    ragas_version: str,
+) -> dict:
+    """Serialize an adapted prompt instance into the committed cache JSON schema."""
+    return {
+        "metric": metric,
+        "prompt_attr": attr,
+        "language": getattr(adapted, "language", language),
+        "ragas_version": ragas_version,
+        "source_hash": source_hash,
+        "instruction": adapted.instruction,
+        "examples": [
+            {"input": inp.model_dump(), "output": out.model_dump()}
+            for inp, out in adapted.examples
+        ],
+    }
+
+
+async def build_cache(
+    language: str,
+    judge_model: str,
+    embedding_model: str,
+    settings: Any,
+) -> list[Path]:
+    """Generate and write the full prompt cache for one language; return written paths.
+
+    Each prompt is adapted once, serialized, and written atomically (temp file rename
+    is not used — if the script crashes mid-way, re-running overwrites partial files).
+    """
+    import ragas
+
+    target = _ADAPT_LANGUAGE.get(language, language)
+    llm, _embeddings = build_models(judge_model, embedding_model, settings)
+    registry = build_metric_registry(llm, _embeddings)
+    out_dir = CACHE_ROOT / language
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    written: list[Path] = []
+    for metric_name, attrs in METRIC_PROMPT_ATTRS.items():
+        for attr in attrs:
+            metric = registry[metric_name]
+            prompt = getattr(metric, attr)
+            src_hash = prompt_source_hash(prompt)
+            logger.info("adapting %s.%s …", metric_name, attr)
+            adapted = await prompt.adapt(target, llm, adapt_instruction=True)
+            data = serialize_prompt(adapted, src_hash, metric_name, attr, language, ragas.__version__)
+            path = out_dir / f"{metric_name}__{attr}.json"
+            path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
+            written.append(path)
+            logger.info("  written → %s", path.name)
+
+    return written
+
+
+def main() -> None:
+    """CLI entry point: parse args, resolve models, and build the cache."""
+    parser = argparse.ArgumentParser(
+        description="Build the RAGAS judge-prompt translation cache for a target language."
+    )
+    parser.add_argument("--language", default="zh", help="Target language code (default: zh)")
+    parser.add_argument("--judge-model", default=None, help="Override judge model from settings")
+    parser.add_argument("--embedding-model", default=None, help="Override embedding model")
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s  %(message)s")
+    settings = EvaluationSettings()
+    judge_model = args.judge_model or settings.ragas_judge_model
+    embedding_model = args.embedding_model or settings.ragas_embedding_model
+
+    paths = asyncio.run(build_cache(args.language, judge_model, embedding_model, settings))
+    logger.info("Done: %d cache files written to configs/judge_prompts/%s/", len(paths), args.language)
+
+
+if __name__ == "__main__":
+    main()

tests/test_advisor_comparison.py

@@ -0,0 +1,306 @@
+"""Tests for webapp.services.advisor_comparison: same-scenario run comparison."""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pandas as pd
+
+from webapp.services.advisor_comparison import build_advisor_comparison, find_previous_run
+
+
+def _write_fake_run(
+    root: Path,
+    run_id: str,
+    scenario_name: str,
+    finished_at: str,
+    judge_model: str,
+    rows: list[dict],
+) -> Path:
+    """Write a minimal run directory discoverable by run_reader.list_run_summaries()."""
+    run_dir = root / run_id
+    run_dir.mkdir(parents=True, exist_ok=True)
+    pd.DataFrame(rows).to_csv(run_dir / "scores.csv", index=False)
+    metadata = {
+        "run_id": run_id,
+        "scenario_name": scenario_name,
+        "judge_model": judge_model,
+        "embedding_model": "embed-model",
+        "finished_at": finished_at,
+        "started_at": finished_at,
+        "valid_samples": len(rows),
+        "invalid_samples": 0,
+    }
+    (run_dir / "metadata.json").write_text(json.dumps(metadata), encoding="utf-8")
+    return run_dir
+
+
+class TestFindPreviousRun:
+    def test_finds_most_recent_prior_run_with_same_scenario(self, tmp_path: Path) -> None:
+        _write_fake_run(tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.5}])
+        _write_fake_run(tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.6}])
+        _write_fake_run(tmp_path, "r3", "scn-a", "2026-01-03T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.9}])
+
+        previous = find_previous_run(
+            "scn-a", "r3", "2026-01-03T00:00:00+00:00", extra_roots=[tmp_path]
+        )
+
+        assert previous is not None
+        assert previous.run_id == "r2"
+
+    def test_excludes_runs_with_different_scenario_name(self, tmp_path: Path) -> None:
+        _write_fake_run(tmp_path, "r1", "scn-other", "2026-01-01T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.5}])
+        _write_fake_run(tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.6}])
+
+        previous = find_previous_run(
+            "scn-a", "r2", "2026-01-02T00:00:00+00:00", extra_roots=[tmp_path]
+        )
+
+        assert previous is None
+
+    def test_returns_none_when_no_history(self, tmp_path: Path) -> None:
+        _write_fake_run(tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.5}])
+
+        previous = find_previous_run(
+            "scn-a", "r1", "2026-01-01T00:00:00+00:00", extra_roots=[tmp_path]
+        )
+
+        assert previous is None
+
+    def test_ignores_runs_at_or_after_current_time(self, tmp_path: Path) -> None:
+        _write_fake_run(tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.5}])
+        _write_fake_run(tmp_path, "r2", "scn-a", "2026-01-05T00:00:00+00:00", "gpt-5",
+                         [{"sample_id": "s1", "faithfulness": 0.6}])
+
+        # Current run finished at 2026-01-02, i.e. AFTER r1 but BEFORE r2.
+        previous = find_previous_run(
+            "scn-a", "r-current", "2026-01-02T00:00:00+00:00", extra_roots=[tmp_path]
+        )
+
+        assert previous is not None
+        assert previous.run_id == "r1"
+
+
+class TestBuildAdvisorComparison:
+    def test_resolved_status_when_previously_triggered_now_healthy(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}, {"sample_id": "s2", "faithfulness": 0.5}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.95}, {"sample_id": "s2", "faithfulness": 0.95}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        assert len(comparison.entries) == 1
+        entry = comparison.entries[0]
+        assert entry.metric == "faithfulness"
+        assert entry.status == "resolved"
+        assert entry.previous_score == 0.5
+        assert entry.current_score is None  # not triggered now → no Diagnosis on current side
+
+    def test_regressed_status_when_previously_healthy_now_triggered(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.95}, {"sample_id": "s2", "faithfulness": 0.95}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}, {"sample_id": "s2", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        assert len(comparison.entries) == 1
+        entry = comparison.entries[0]
+        assert entry.status == "regressed"
+        assert entry.previous_score is None
+        assert entry.current_score == 0.5
+
+    def test_still_triggered_status_shows_score_and_severity_change(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.55}, {"sample_id": "s2", "faithfulness": 0.55}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.45}, {"sample_id": "s2", "faithfulness": 0.45}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        entry = comparison.entries[0]
+        assert entry.status == "still_triggered"
+        assert entry.previous_severity == "warning"
+        assert entry.current_severity == "critical"
+        assert entry.previous_score == 0.55
+        assert entry.current_score == 0.45
+
+    def test_new_metric_status_when_metric_not_measured_before(self, tmp_path: Path) -> None:
+        # Previous run only measured context_recall (healthy); faithfulness wasn't tracked at all.
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "context_recall": 0.95}, {"sample_id": "s2", "context_recall": 0.95}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}, {"sample_id": "s2", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        assert len(comparison.entries) == 1
+        entry = comparison.entries[0]
+        assert entry.metric == "faithfulness"
+        assert entry.status == "new_metric"
+        assert entry.previous_score is None
+        assert entry.current_score == 0.5
+
+    def test_metrics_healthy_in_both_are_omitted(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.95}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.96}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is None  # nothing to show → overall None
+
+    def test_metric_dropped_from_current_scope_is_not_marked_resolved(self, tmp_path: Path) -> None:
+        # Previous run triggered on context_precision, but current run doesn't
+        # evaluate that metric at all — must NOT claim "resolved" without a
+        # fair current-side measurement.
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "context_precision": 0.3}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.95}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is None
+
+    def test_returns_none_when_no_previous_run(self, tmp_path: Path) -> None:
+        current_dir = _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is None
+
+    def test_judge_model_changed_flag_set_when_models_differ(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-4o",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        assert comparison.judge_model_changed is True
+        assert comparison.previous_judge_model == "gpt-4o"
+        assert comparison.current_judge_model == "gpt-5"
+
+    def test_judge_model_changed_false_when_same(self, tmp_path: Path) -> None:
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        assert comparison.judge_model_changed is False
+
+    def test_gracefully_returns_none_on_corrupt_previous_scores_csv(self, tmp_path: Path) -> None:
+        previous_dir = _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+        # Corrupt the previous run's scores.csv with invalid-encoding bytes so
+        # pd.read_csv raises UnicodeDecodeError (a ValueError subclass caught
+        # by run_reader.read_scores_frame, which then returns an empty frame).
+        (previous_dir / "scores.csv").write_bytes(b"\xff\xfe\x00\x01broken binary data \x00\x00")
+
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [{"sample_id": "s1", "faithfulness": 0.5}],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is None
+
+    def test_worse_statuses_sorted_before_resolved(self, tmp_path: Path) -> None:
+        """regressed/still_triggered/new_metric surface above resolved for visibility."""
+        _write_fake_run(
+            tmp_path, "r1", "scn-a", "2026-01-01T00:00:00+00:00", "gpt-5",
+            [
+                {"sample_id": "s1", "faithfulness": 0.5, "context_recall": 0.95},
+            ],
+        )
+        current_dir = _write_fake_run(
+            tmp_path, "r2", "scn-a", "2026-01-02T00:00:00+00:00", "gpt-5",
+            [
+                {"sample_id": "s1", "faithfulness": 0.95, "context_recall": 0.5},
+            ],
+        )
+
+        comparison = build_advisor_comparison(
+            current_dir, "scn-a", ["faithfulness", "context_recall"], extra_roots=[tmp_path]
+        )
+
+        assert comparison is not None
+        statuses = [entry.status for entry in comparison.entries]
+        # context_recall regressed → must appear before faithfulness resolved.
+        assert statuses.index("regressed") < statuses.index("resolved")

tests/test_advisor_comparison_models.py

@@ -0,0 +1,50 @@
+"""Tests for the AdvisorComparison Pydantic models and their default wiring."""
+from __future__ import annotations
+
+from webapp.models import AdvisorComparison, AdvisorComparisonEntry, ReportData
+
+
+def test_advisor_comparison_entry_accepts_all_statuses():
+    for status in ("resolved", "regressed", "still_triggered", "new_metric"):
+        entry = AdvisorComparisonEntry(
+            metric="faithfulness",
+            status=status,
+            previous_score=0.5,
+            previous_severity="warning",
+            current_score=0.9,
+            current_severity=None,
+        )
+        assert entry.status == status
+
+
+def test_advisor_comparison_entry_rejects_unknown_status():
+    import pydantic
+
+    try:
+        AdvisorComparisonEntry(metric="faithfulness", status="unknown_status")
+        raised = False
+    except pydantic.ValidationError:
+        raised = True
+    assert raised
+
+
+def test_advisor_comparison_defaults():
+    comparison = AdvisorComparison(
+        previous_run_id="run-1",
+        previous_finished_at="2026-01-01T00:00:00+00:00",
+    )
+    assert comparison.previous_judge_model == ""
+    assert comparison.current_judge_model == ""
+    assert comparison.judge_model_changed is False
+    assert comparison.entries == []
+
+
+def test_report_data_defaults_advisor_comparison_to_none():
+    report = ReportData()
+    assert report.advisor_comparison is None
+
+
+def test_report_data_accepts_advisor_comparison():
+    comparison = AdvisorComparison(previous_run_id="run-1", previous_finished_at="t")
+    report = ReportData(advisor_comparison=comparison)
+    assert report.advisor_comparison == comparison

tests/test_advisor_llm_analyzer.py

@@ -0,0 +1,174 @@
+"""Tests for the optimization advisor's direct-LLM analyzer.
+
+These tests inject a fake async chat client so no network call is made. They
+verify that analyze() uses a plain chat.completions call (not the removed
+langchain path), returns the text from choices[0].message.content, embeds the
+worked-example instructions and the low-sample contexts in the prompt, and
+selects the correct token parameter for reasoning vs. legacy models.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+from rag_eval.advisor.llm_analyzer import analyze, _is_reasoning_model
+from rag_eval.advisor.rules import Diagnosis
+
+
+class _FakeMessage:
+    def __init__(self, content: str) -> None:
+        self.content = content
+
+
+class _FakeChoice:
+    def __init__(self, content: str) -> None:
+        self.message = _FakeMessage(content)
+
+
+class _FakeResponse:
+    def __init__(self, content: str) -> None:
+        self.choices = [_FakeChoice(content)]
+
+
+class _FakeCompletions:
+    def __init__(self, captured: dict) -> None:
+        self._captured = captured
+
+    async def create(self, **kwargs):
+        self._captured.update(kwargs)
+        return _FakeResponse("## faithfulness [警告]\n\n针对该问题的具体优化建议")
+
+
+class _FakeChat:
+    def __init__(self, captured: dict) -> None:
+        self.completions = _FakeCompletions(captured)
+
+
+class _FakeClient:
+    def __init__(self, captured: dict) -> None:
+        self.chat = _FakeChat(captured)
+        self.closed = False
+
+    async def close(self) -> None:
+        self.closed = True
+
+
+class _Settings:
+    ragas_llm_max_tokens = 4096
+
+
+def _diagnosis() -> Diagnosis:
+    return Diagnosis(
+        metric="faithfulness",
+        mean_score=0.55,
+        threshold=0.7,
+        severity="warning",
+        root_causes=["生成未严格 grounding"],
+        suggested_actions=["强化 grounding 约束"],
+        low_samples=[
+            {
+                "sample_id": "s1",
+                "question": "球管寿命如何评估？",
+                "answer": "球管寿命约 3 年。",
+                "ground_truth": "球管寿命取决于使用强度。",
+                "contexts": "球管寿命与扫描负载相关 |||| 高负载会缩短寿命",
+                "faithfulness": 0.4,
+            }
+        ],
+    )
+
+
+def test_analyze_uses_direct_chat_and_returns_content() -> None:
+    captured: dict = {}
+    text = asyncio.run(
+        analyze([_diagnosis()], "scn", "gpt-4o", _Settings(), chat_client=_FakeClient(captured))
+    )
+    assert "优化建议" in text
+    assert captured["model"] == "gpt-4o"
+    prompt = captured["messages"][0]["content"]
+    assert "举例拆解" in prompt          # worked-example instruction present
+    assert "球管寿命与扫描负载相关" in prompt  # low-sample contexts embedded
+    assert "max_tokens" in captured       # legacy model uses max_tokens
+    assert "max_completion_tokens" not in captured
+
+
+def test_analyze_reasoning_model_uses_max_completion_tokens() -> None:
+    captured: dict = {}
+    asyncio.run(
+        analyze([_diagnosis()], "scn", "gpt-5", _Settings(), chat_client=_FakeClient(captured))
+    )
+    assert "max_completion_tokens" in captured
+    assert "max_tokens" not in captured
+
+
+def test_analyze_empty_diagnoses_returns_empty() -> None:
+    assert asyncio.run(analyze([], "scn", "gpt-4o", _Settings())) == ""
+
+
+def test_analyze_closes_client_it_creates(monkeypatch) -> None:
+    """A self-created client is closed in-loop to avoid 'Event loop is closed'."""
+    captured: dict = {}
+    fake = _FakeClient(captured)
+
+    import openai
+    import rag_eval.metrics.factory as factory_mod
+
+    monkeypatch.setattr(openai, "AsyncOpenAI", lambda **kwargs: fake)
+    monkeypatch.setattr(
+        factory_mod, "resolve_openai_client_kwargs", lambda *a, **k: {"api_key": "x"}
+    )
+
+    # No chat_client passed → analyze() builds (and must close) its own client.
+    text = asyncio.run(analyze([_diagnosis()], "scn", "gpt-4o", _Settings()))
+
+    assert "优化建议" in text
+    assert fake.closed is True
+
+
+def test_analyze_does_not_close_injected_client() -> None:
+    """An injected client is owned by the caller and must not be closed."""
+    fake = _FakeClient({})
+    asyncio.run(analyze([_diagnosis()], "scn", "gpt-4o", _Settings(), chat_client=fake))
+    assert fake.closed is False
+
+
+def test_analyze_attaches_usage_hook_to_self_created_client(monkeypatch) -> None:
+    """A self-created client gets the token-usage hook attached (not the injected-client path)."""
+    captured: dict = {}
+    fake = _FakeClient(captured)
+    hook_calls = []
+
+    import openai
+    import rag_eval.metrics.factory as factory_mod
+
+    monkeypatch.setattr(openai, "AsyncOpenAI", lambda **kwargs: fake)
+    monkeypatch.setattr(
+        factory_mod, "resolve_openai_client_kwargs", lambda *a, **k: {"api_key": "x"}
+    )
+    monkeypatch.setattr(factory_mod, "attach_usage_hook", lambda c: hook_calls.append(c))
+
+    asyncio.run(analyze([_diagnosis()], "scn", "gpt-4o", _Settings()))
+
+    assert hook_calls == [fake]
+
+
+def test_analyze_does_not_attach_hook_for_injected_client() -> None:
+    """An injected chat_client is assumed to already have the hook attached by its owner."""
+    hook_calls = []
+    import rag_eval.metrics.factory as factory_mod
+    import unittest.mock as mock
+
+    with mock.patch.object(factory_mod, "attach_usage_hook", lambda c: hook_calls.append(c)):
+        fake = _FakeClient({})
+        asyncio.run(analyze([_diagnosis()], "scn", "gpt-4o", _Settings(), chat_client=fake))
+
+    assert hook_calls == []
+
+
+def test_is_reasoning_model_detection() -> None:
+    assert _is_reasoning_model("gpt-5")
+    assert _is_reasoning_model("gpt-5.5")
+    assert _is_reasoning_model("o1-mini")
+    assert _is_reasoning_model("o3")
+    assert not _is_reasoning_model("gpt-4o")
+    assert not _is_reasoning_model("deepseek-v4-flash")

tests/test_build_models_separate_clients.py

@@ -0,0 +1,134 @@
+"""Tests that build_models resolves separate clients for judge and embedding models."""
+
+from unittest.mock import MagicMock, patch, call
+from rag_eval.metrics.factory import build_models
+from rag_eval.settings import EvaluationSettings
+
+
+def _settings():
+    return EvaluationSettings(_env_file=None)
+
+
+def _patch_profiles(profiles_by_model: dict):
+    """Patch profile_manager.list_all() to return fake profiles."""
+    fake_profiles = []
+    for model, base_url in profiles_by_model.items():
+        p = MagicMock()
+        p.model = model
+        p.base_url = base_url
+        p.api_key = "sk-test"
+        p.timeout_seconds = 30
+        fake_profiles.append(p)
+    return fake_profiles
+
+
+def test_build_models_uses_separate_clients_for_different_profiles(monkeypatch):
+    """When judge and embedding models have different profiles, two AsyncOpenAI clients are created."""
+    created_clients = []
+
+    class _FakeClient:
+        def __init__(self, **kwargs):
+            created_clients.append(kwargs.copy())
+
+    fake_profiles = _patch_profiles({
+        "gpt-5": "http://llm-gateway/v1",
+        "Qwen3-Embedding-4B": "https://api.siliconflow.cn/v1",
+    })
+
+    import webapp.services.profile_manager as pm_mod
+    monkeypatch.setattr(pm_mod.profile_manager, "list_all", lambda: fake_profiles)
+
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    with patch("rag_eval.metrics.factory.AsyncOpenAI", side_effect=_FakeClient), \
+         patch("rag_eval.metrics.factory.llm_factory", return_value=MagicMock(spec=InstructorBaseRagasLLM)), \
+         patch("rag_eval.metrics.factory.embedding_factory", return_value=MagicMock(spec=BaseRagasEmbedding)):
+        build_models("gpt-5", "Qwen3-Embedding-4B", _settings())
+
+    # Two distinct clients should have been created with different base_urls.
+    assert len(created_clients) == 2
+    base_urls = {c["base_url"] for c in created_clients}
+    assert "http://llm-gateway/v1" in base_urls
+    assert "https://api.siliconflow.cn/v1" in base_urls
+
+
+def test_build_models_shares_client_when_same_profile(monkeypatch):
+    """When judge and embedding models resolve to the same settings, only one client is created."""
+    created_clients = []
+
+    class _FakeClient:
+        def __init__(self, **kwargs):
+            created_clients.append(kwargs.copy())
+
+    # Both models map to the same gateway.
+    fake_profiles = _patch_profiles({
+        "gpt-5": "http://same-gateway/v1",
+        "text-embedding-3-small": "http://same-gateway/v1",
+    })
+
+    import webapp.services.profile_manager as pm_mod
+    monkeypatch.setattr(pm_mod.profile_manager, "list_all", lambda: fake_profiles)
+
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    with patch("rag_eval.metrics.factory.AsyncOpenAI", side_effect=_FakeClient), \
+         patch("rag_eval.metrics.factory.llm_factory", return_value=MagicMock(spec=InstructorBaseRagasLLM)), \
+         patch("rag_eval.metrics.factory.embedding_factory", return_value=MagicMock(spec=BaseRagasEmbedding)):
+        build_models("gpt-5", "text-embedding-3-small", _settings())
+
+    # Same settings → only one client needed.
+    assert len(created_clients) == 1
+
+
+def test_embedding_factory_receives_embedding_client(monkeypatch):
+    """embedding_factory is called with the client resolved from the embedding model's profile."""
+    emb_client_ref = []
+    llm_client_ref = []
+
+    class _LLMClient:
+        label = "llm"
+        def __init__(self, **kwargs):
+            llm_client_ref.append(self)
+
+    class _EmbClient:
+        label = "emb"
+        def __init__(self, **kwargs):
+            emb_client_ref.append(self)
+
+    fake_profiles = _patch_profiles({
+        "gpt-5": "http://llm-gw/v1",
+        "Qwen3-Embedding-4B": "https://api.siliconflow.cn/v1",
+    })
+
+    import webapp.services.profile_manager as pm_mod
+    monkeypatch.setattr(pm_mod.profile_manager, "list_all", lambda: fake_profiles)
+
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    emb_factory_calls = []
+
+    def _fake_emb_factory(provider, model, client):
+        emb_factory_calls.append({"model": model, "client": client})
+        return MagicMock(spec=BaseRagasEmbedding)
+
+    call_count = [0]
+
+    def _client_factory(**kwargs):
+        call_count[0] += 1
+        if "llm-gw" in kwargs.get("base_url", ""):
+            return _LLMClient(**kwargs)
+        return _EmbClient(**kwargs)
+
+    with patch("rag_eval.metrics.factory.AsyncOpenAI", side_effect=_client_factory), \
+         patch("rag_eval.metrics.factory.llm_factory", return_value=MagicMock(spec=InstructorBaseRagasLLM)), \
+         patch("rag_eval.metrics.factory.embedding_factory", side_effect=_fake_emb_factory):
+        build_models("gpt-5", "Qwen3-Embedding-4B", _settings())
+
+    assert len(emb_factory_calls) == 1
+    # The client passed to embedding_factory must be the siliconflow client, not the LLM client.
+    assert isinstance(emb_factory_calls[0]["client"], _EmbClient), (
+        "embedding_factory should receive the embedding-model client, not the judge-model client"
+    )

tests/test_dashboard.py

@@ -0,0 +1,197 @@
+"""Tests for the Dashboard module's pure data functions and MetricPresenter additions.
+
+The pure data functions (_buildTrendDatasets, _buildComparisonData) and the new
+MetricPresenter helpers (passThreshold, meetsTarget) are tested via Node.js so no
+browser or network is required.
+"""
+from __future__ import annotations
+
+import subprocess
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+
+
+def _run_node(script: str) -> str:
+    """Execute a Node.js script and return stdout."""
+    completed = subprocess.run(
+        ["node", "-e", script],
+        cwd=REPO_ROOT,
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        check=True,
+    )
+    return completed.stdout.strip()
+
+
+def _load_js() -> str:
+    """Return a Node-runnable bootstrap that loads MetricPresenter + Dashboard."""
+    presenter_path = (REPO_ROOT / "webapp" / "static" / "js" / "metric_presenter.js").as_posix()
+    dashboard_path = (REPO_ROOT / "webapp" / "static" / "js" / "dashboard.js").as_posix()
+    return f"""
+const fs = require("fs");
+const vm = require("vm");
+
+// Shared sandbox (window object shared by both scripts)
+const sandbox = {{ window: {{}}, console }};
+
+// MetricPresenter
+vm.runInNewContext(fs.readFileSync("{presenter_path}", "utf8"), sandbox);
+const MetricPresenter = sandbox.window.MetricPresenter;
+
+// Minimal App stub required by dashboard.js
+sandbox.App = {{
+  escape: (s) => String(s == null ? "" : s),
+  shortMetric: (m) => m,
+  shortTime: (t) => (t || "").slice(0, 16),
+  scoreClass: (m, v) => MetricPresenter.scoreClass(m, v),
+}};
+sandbox.MetricPresenter = MetricPresenter;
+// Stub Chart.js (not exercised by pure functions)
+sandbox.Chart = function() {{ this.destroy = () => {{}}; }};
+
+// Dashboard (attaches itself to sandbox.window.Dashboard)
+vm.runInNewContext(fs.readFileSync("{dashboard_path}", "utf8"), sandbox);
+const Dashboard = sandbox.window.Dashboard;
+"""
+
+
+def test_pass_threshold_higher_better() -> None:
+    """All higher-better metrics should have passThreshold 0.85."""
+    script = _load_js() + """
+const result = {
+  faith: MetricPresenter.passThreshold("faithfulness"),
+  ans: MetricPresenter.passThreshold("answer_relevancy"),
+  recall: MetricPresenter.passThreshold("context_recall"),
+  prec: MetricPresenter.passThreshold("context_precision"),
+  fact: MetricPresenter.passThreshold("factual_correctness"),
+  sem: MetricPresenter.passThreshold("semantic_similarity"),
+};
+console.log(JSON.stringify(result));
+"""
+    out = _run_node(script)
+    assert '"faith":0.85' in out
+    assert '"ans":0.85' in out
+    assert '"recall":0.85' in out
+    assert '"prec":0.85' in out
+    assert '"fact":0.85' in out
+    assert '"sem":0.85' in out
+
+
+def test_pass_threshold_noise_sensitivity_lower_better() -> None:
+    """noise_sensitivity (lower-better) should have passThreshold 0.15."""
+    script = _load_js() + """
+console.log(JSON.stringify(MetricPresenter.passThreshold("noise_sensitivity")));
+"""
+    out = _run_node(script)
+    assert out.strip() == "0.15"
+
+
+def test_meets_target_higher_better() -> None:
+    """meetsTarget should return true only at/above 0.85 for higher-better metrics."""
+    script = _load_js() + """
+const result = {
+  at085: MetricPresenter.meetsTarget("faithfulness", 0.85),
+  above: MetricPresenter.meetsTarget("faithfulness", 0.90),
+  below: MetricPresenter.meetsTarget("faithfulness", 0.84),
+  zero: MetricPresenter.meetsTarget("faithfulness", 0),
+  nul: MetricPresenter.meetsTarget("faithfulness", null),
+};
+console.log(JSON.stringify(result));
+"""
+    out = _run_node(script)
+    assert '"at085":true' in out
+    assert '"above":true' in out
+    assert '"below":false' in out
+    assert '"zero":false' in out
+    assert '"nul":false' in out
+
+
+def test_meets_target_noise_sensitivity() -> None:
+    """meetsTarget for noise_sensitivity: true only at/below 0.15."""
+    script = _load_js() + """
+const result = {
+  at015: MetricPresenter.meetsTarget("noise_sensitivity", 0.15),
+  below: MetricPresenter.meetsTarget("noise_sensitivity", 0.10),
+  above: MetricPresenter.meetsTarget("noise_sensitivity", 0.16),
+};
+console.log(JSON.stringify(result));
+"""
+    out = _run_node(script)
+    assert '"at015":true' in out
+    assert '"below":true' in out
+    assert '"above":false' in out
+
+
+def test_build_trend_datasets_time_order_and_null_gap() -> None:
+    """_buildTrendDatasets returns metrics in appearance order, null for missing values."""
+    script = (
+        _load_js()
+        + """
+const runs = [
+  { run_id: "r1", scenario_name: "scn", finished_at: "2026-01-01T00:00:00",
+    metrics: ["faithfulness"], metric_means: { faithfulness: 0.60 } },
+  { run_id: "r2", scenario_name: "scn", finished_at: "2026-02-01T00:00:00",
+    metrics: ["faithfulness", "noise_sensitivity"],
+    metric_means: { faithfulness: 0.80, noise_sensitivity: 0.20 } },
+  { run_id: "r3", scenario_name: "scn", finished_at: "2026-03-01T00:00:00",
+    metrics: ["faithfulness"], metric_means: { faithfulness: 0.90 } },
+];
+const { labels, datasets } = Dashboard._buildTrendDatasets(runs);
+console.log(JSON.stringify({ labels, datasets }));
+"""
+    )
+    import json
+
+    out = json.loads(_run_node(script))
+    assert len(out["labels"]) == 3
+    # faithfulness dataset: all 3 points
+    faith_ds = next(d for d in out["datasets"] if "faithfulness" in d["label"])
+    assert faith_ds["data"] == [0.60, 0.80, 0.90]
+    # noise_sensitivity only present in r2 → null in r1 and r3
+    noise_ds = next(d for d in out["datasets"] if "noise_sensitivity" in d["label"])
+    assert noise_ds["data"] == [None, 0.20, None]
+    # noise label should note lower-is-better
+    assert "越低越好" in noise_ds["label"]
+
+
+def test_build_comparison_data_structure() -> None:
+    """_buildComparisonData returns correct labels, actuals, thresholds and targetMet."""
+    script = (
+        _load_js()
+        + """
+const run = {
+  run_id: "r1", scenario_name: "scn",
+  metrics: ["faithfulness", "noise_sensitivity"],
+  metric_means: { faithfulness: 0.90, noise_sensitivity: 0.10 },
+};
+const result = Dashboard._buildComparisonData(run);
+console.log(JSON.stringify(result));
+"""
+    )
+    import json
+
+    out = json.loads(_run_node(script))
+    assert out["labels"] == ["faithfulness", "noise_sensitivity"]
+    assert out["actual"] == [0.90, 0.10]
+    assert out["thresholds"] == [0.85, 0.15]
+    assert out["targetMet"] == [True, True]  # 0.90 >= 0.85 ✓; 0.10 <= 0.15 ✓
+
+
+def test_build_comparison_data_unmet_targets() -> None:
+    """targetMet is False when metrics are below threshold."""
+    script = (
+        _load_js()
+        + """
+const run = {
+  run_id: "r1", scenario_name: "scn",
+  metrics: ["faithfulness", "noise_sensitivity"],
+  metric_means: { faithfulness: 0.60, noise_sensitivity: 0.40 },
+};
+const result = Dashboard._buildComparisonData(run);
+console.log(JSON.stringify(result.targetMet));
+"""
+    )
+    out = _run_node(script)
+    assert out.strip() == "[false,false]"

tests/test_evaluator_token_usage.py

@@ -0,0 +1,129 @@
+"""Tests verifying the CLI evaluation flow captures token usage from metric scoring."""
+from __future__ import annotations
+
+import shutil
+import unittest
+from pathlib import Path
+
+import pandas as pd
+
+from rag_eval.execution.evaluator import Evaluator
+from rag_eval.metrics.pipeline import MetricPipeline
+from rag_eval.metrics.token_tracker import get_current_tracker
+from rag_eval.shared.models import DatasetConfig, RuntimeConfig, Scenario
+
+
+class FakeMetricWithUsage:
+    """Fake RAGAS metric that records token usage like a real HTTP-hooked call would."""
+
+    def __init__(self, value: float, model: str, input_tokens: int, output_tokens: int):
+        self.value = value
+        self.model = model
+        self.input_tokens = input_tokens
+        self.output_tokens = output_tokens
+
+    async def ascore(self, **kwargs):
+        tracker = get_current_tracker()
+        if tracker is not None:
+            tracker.record(self.model, self.input_tokens, self.output_tokens)
+
+        class Result:
+            def __init__(self, value: float):
+                self.value = value
+
+        return Result(self.value)
+
+
+class PlainFakeMetric:
+    """Fake metric that never records usage (simulates a hook that captured nothing)."""
+
+    async def ascore(self, **kwargs):
+        class Result:
+            value = 0.9
+
+        return Result()
+
+
+class EvaluatorTokenUsageTests(unittest.TestCase):
+    def setUp(self) -> None:
+        root = Path("tests/.tmp").resolve()
+        root.mkdir(parents=True, exist_ok=True)
+        self.temp_dir = root / self._testMethodName
+        shutil.rmtree(self.temp_dir, ignore_errors=True)
+        self.temp_dir.mkdir(parents=True, exist_ok=True)
+
+    def tearDown(self) -> None:
+        shutil.rmtree(self.temp_dir, ignore_errors=True)
+
+    def _write_offline_dataset(self, path: Path, rows: list[dict]) -> None:
+        pd.DataFrame(rows).to_csv(path, index=False)
+
+    def test_evaluate_populates_token_usage_from_metric_calls(self) -> None:
+        dataset_path = self.temp_dir / "offline.csv"
+        self._write_offline_dataset(dataset_path, [
+            {
+                "sample_id": "sample-1",
+                "question": "What is the policy scope?",
+                "answer": "It covers all employees.",
+                "contexts": '["Context A"]',
+                "ground_truth": "It covers all employees.",
+            },
+            {
+                "sample_id": "sample-2",
+                "question": "What about contractors?",
+                "answer": "Contractors are excluded.",
+                "contexts": '["Context B"]',
+                "ground_truth": "Contractors are excluded.",
+            },
+        ])
+
+        scenario = Scenario(
+            scenario_name="token-usage-test",
+            mode="offline",
+            dataset=DatasetConfig(path=dataset_path),
+            judge_model="gpt-5",
+            embedding_model="embedding-model",
+            metrics=["faithfulness"],
+            output_dir=self.temp_dir / "outputs",
+            runtime=RuntimeConfig(batch_size=1),
+        )
+        pipeline = MetricPipeline(
+            metrics={"faithfulness": FakeMetricWithUsage(0.8, "gpt-5", 100, 40)}
+        )
+        evaluator = Evaluator(scenario=scenario, metric_pipeline=pipeline)
+
+        result = evaluator.evaluate()
+
+        # Two samples each recorded one call → totals sum across both.
+        self.assertEqual(
+            result.token_usage,
+            {"gpt-5": {"input_tokens": 200, "output_tokens": 80, "calls": 2}},
+        )
+
+    def test_evaluate_defaults_to_empty_token_usage_when_nothing_recorded(self) -> None:
+        dataset_path = self.temp_dir / "offline.csv"
+        self._write_offline_dataset(dataset_path, [
+            {
+                "sample_id": "sample-1",
+                "question": "What is the policy scope?",
+                "answer": "It covers all employees.",
+                "contexts": '["Context A"]',
+                "ground_truth": "It covers all employees.",
+            },
+        ])
+
+        scenario = Scenario(
+            scenario_name="token-usage-empty-test",
+            mode="offline",
+            dataset=DatasetConfig(path=dataset_path),
+            judge_model="gpt-5",
+            embedding_model="embedding-model",
+            metrics=["faithfulness"],
+            output_dir=self.temp_dir / "outputs",
+            runtime=RuntimeConfig(batch_size=1),
+        )
+        pipeline = MetricPipeline(metrics={"faithfulness": PlainFakeMetric()})
+        evaluator = Evaluator(scenario=scenario, metric_pipeline=pipeline)
+
+        result = evaluator.evaluate()
+        self.assertEqual(result.token_usage, {})

tests/test_judge_language_config.py

@@ -0,0 +1,72 @@
+"""Tests for judge_language plumbing across settings, scenario schema, and loader."""
+
+from pathlib import Path
+
+import pytest
+
+from rag_eval.settings import EvaluationSettings
+from rag_eval.config.loader import load_scenario
+
+
+def test_settings_default_judge_language_is_en():
+    """ragas_judge_language defaults to 'en' when the env var is absent."""
+    settings = EvaluationSettings(_env_file=None)
+    assert settings.ragas_judge_language == "en"
+
+
+def _write_scenario(tmp_path: Path, extra: str) -> Path:
+    """Write a minimal valid offline scenario YAML plus the given extra line(s)."""
+    dataset = tmp_path / "data.csv"
+    dataset.write_text("sample_id,question,answer,contexts,ground_truth\n", encoding="utf-8")
+    text = (
+        "scenario_name: t\n"
+        "mode: offline\n"
+        f"dataset: {dataset.name}\n"
+        "judge_model: gpt-5\n"
+        "embedding_model: text-embedding-3-small\n"
+        "metrics: [faithfulness]\n"
+        "output_dir: out\n"
+        f"{extra}"
+    )
+    path = tmp_path / "s.yaml"
+    path.write_text(text, encoding="utf-8")
+    return path
+
+
+def test_scenario_loads_judge_language_zh(tmp_path):
+    """A scenario may declare judge_language: zh and it lands on the dataclass."""
+    path = _write_scenario(tmp_path, "judge_language: zh\n")
+    scenario = load_scenario(path)
+    assert scenario.judge_language == "zh"
+
+
+def test_scenario_defaults_judge_language_none(tmp_path):
+    """Omitting judge_language leaves it None so the factory can apply the settings default."""
+    path = _write_scenario(tmp_path, "")
+    scenario = load_scenario(path)
+    assert scenario.judge_language is None
+
+
+def test_scenario_rejects_invalid_judge_language(tmp_path):
+    """An unsupported judge_language value is rejected at schema validation."""
+    path = _write_scenario(tmp_path, "judge_language: fr\n")
+    with pytest.raises(Exception):
+        load_scenario(path)
+
+
+def test_score_request_judge_language_defaults_none():
+    """ScoreRequest exposes an optional judge_language defaulting to None."""
+    from webapp.models import ScoreRequest
+
+    req = ScoreRequest(question="q", answer="a")
+    assert req.judge_language is None
+    req_zh = ScoreRequest(question="q", answer="a", judge_language="zh")
+    assert req_zh.judge_language == "zh"
+
+
+def test_session_score_request_inherits_judge_language():
+    """SessionScoreRequest inherits the judge_language field from ScoreRequest."""
+    from webapp.models import SessionScoreRequest
+
+    req = SessionScoreRequest(session_id="s1", question="q", answer="a", judge_language="zh")
+    assert req.judge_language == "zh"

tests/test_judge_language_wiring.py

@@ -0,0 +1,119 @@
+"""Tests that the factory and inline scorer invoke the localizer per language."""
+
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import rag_eval.metrics.factory as factory_mod
+import webapp.services.inline_scorer as inline_mod
+
+
+def test_build_pipeline_localizes_when_zh(monkeypatch):
+    """build_metric_pipeline calls localize_pipeline_prompts with 'zh'."""
+    calls = []
+    monkeypatch.setattr(factory_mod, "localize_pipeline_prompts",
+                        lambda registry, language: calls.append(language))
+
+    from rag_eval.shared.models import DatasetConfig, Scenario
+    from rag_eval.settings import EvaluationSettings
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    scenario = Scenario(
+        scenario_name="t", mode="offline",
+        dataset=DatasetConfig(path=Path("x.csv")),
+        judge_model="gpt-5", embedding_model="text-embedding-3-small",
+        metrics=["faithfulness"], output_dir=Path("out"),
+        judge_language="zh",
+    )
+    factory_mod.build_metric_pipeline(
+        scenario, EvaluationSettings(_env_file=None),
+        llm=MagicMock(spec=InstructorBaseRagasLLM),
+        embeddings=MagicMock(spec=BaseRagasEmbedding),
+    )
+    assert calls == ["zh"]
+
+
+def test_build_pipeline_falls_back_to_settings_default(monkeypatch):
+    """When scenario.judge_language is None, settings.ragas_judge_language is used."""
+    calls = []
+    monkeypatch.setattr(factory_mod, "localize_pipeline_prompts",
+                        lambda registry, language: calls.append(language))
+
+    from rag_eval.shared.models import DatasetConfig, Scenario
+    from rag_eval.settings import EvaluationSettings
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    scenario = Scenario(
+        scenario_name="t", mode="offline",
+        dataset=DatasetConfig(path=Path("x.csv")),
+        judge_model="gpt-5", embedding_model="text-embedding-3-small",
+        metrics=["faithfulness"], output_dir=Path("out"),
+        judge_language=None,
+    )
+    settings = EvaluationSettings(_env_file=None)
+    settings.ragas_judge_language = "zh"
+    factory_mod.build_metric_pipeline(
+        scenario, settings,
+        llm=MagicMock(spec=InstructorBaseRagasLLM),
+        embeddings=MagicMock(spec=BaseRagasEmbedding),
+    )
+    assert calls == ["zh"]
+
+
+def test_build_pipeline_en_is_noop(monkeypatch):
+    """Default language 'en' still calls localize (which is a no-op inside)."""
+    calls = []
+    monkeypatch.setattr(factory_mod, "localize_pipeline_prompts",
+                        lambda registry, language: calls.append(language))
+
+    from rag_eval.shared.models import DatasetConfig, Scenario
+    from rag_eval.settings import EvaluationSettings
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+
+    scenario = Scenario(
+        scenario_name="t", mode="offline",
+        dataset=DatasetConfig(path=Path("x.csv")),
+        judge_model="gpt-5", embedding_model="text-embedding-3-small",
+        metrics=["faithfulness"], output_dir=Path("out"),
+    )
+    factory_mod.build_metric_pipeline(
+        scenario, EvaluationSettings(_env_file=None),
+        llm=MagicMock(spec=InstructorBaseRagasLLM),
+        embeddings=MagicMock(spec=BaseRagasEmbedding),
+    )
+    assert calls == ["en"]
+
+
+def test_inline_score_threads_judge_language(monkeypatch):
+    """InlineScorer.score forwards judge_language to _build_metric_instances."""
+    seen: dict = {}
+
+    def fake_build_instances(metrics, llm, embeddings, judge_language="en"):
+        seen["lang"] = judge_language
+        return {}
+
+    monkeypatch.setattr(inline_mod, "_build_metric_instances", fake_build_instances)
+    monkeypatch.setattr(inline_mod.InlineScorer, "_get_models",
+                        lambda self, j, e, s: (object(), object()))
+
+    class _FakePipeline:
+        def __init__(self, *a, **k):
+            pass
+
+        async def score_sample(self, sample):
+            return _FakeResult()
+
+    class _FakeResult:
+        metrics: dict = {}
+
+    import asyncio
+    monkeypatch.setattr(inline_mod, "MetricPipeline", _FakePipeline)
+    monkeypatch.setattr(asyncio, "run", lambda coro: _FakeResult())
+
+    scorer = inline_mod.InlineScorer()
+    scorer.score(question="q", answer="a", contexts=[], ground_truth=None,
+                 metrics=["faithfulness"], judge_model="gpt-5",
+                 embedding_model="e", settings=MagicMock(), judge_language="zh")
+    assert seen.get("lang") == "zh"

tests/test_judge_prompt_cache_builder.py

@@ -0,0 +1,80 @@
+"""Tests for the prompt-cache bootstrap script (mocked adapt/LLM)."""
+
+import asyncio
+import json
+
+from pydantic import BaseModel
+
+import scripts.build_judge_prompt_cache as builder
+
+
+class _In(BaseModel):
+    question: str
+
+
+class _Out(BaseModel):
+    statements: list[str]
+
+
+class _FakeAdapted:
+    """Fake adapted prompt returned by prompt.adapt()."""
+
+    def __init__(self):
+        self.instruction = "中文指令。"
+        self.language = "chinese"
+        self.examples = [(_In(question="问题"), _Out(statements=["陈述"]))]
+
+
+class _FakePrompt:
+    """Fake prompt with async adapt() that returns _FakeAdapted."""
+
+    def __init__(self):
+        self.input_model = _In
+        self.output_model = _Out
+        self.instruction = "English."
+        self.examples = [(_In(question="q"), _Out(statements=["s"]))]
+        self.language = "english"
+
+    async def adapt(self, target_language, llm, adapt_instruction=False):
+        assert target_language == "chinese"
+        assert adapt_instruction is True
+        return _FakeAdapted()
+
+
+def test_serialize_prompt_shape():
+    """serialize_prompt emits the expected committed cache schema."""
+    data = builder.serialize_prompt(
+        _FakeAdapted(), "hash123", "context_recall", "prompt", "zh", "0.4.3"
+    )
+    assert data["metric"] == "context_recall"
+    assert data["prompt_attr"] == "prompt"
+    assert data["language"] == "chinese"
+    assert data["source_hash"] == "hash123"
+    assert data["instruction"] == "中文指令。"
+    assert data["examples"] == [
+        {"input": {"question": "问题"}, "output": {"statements": ["陈述"]}}
+    ]
+
+
+def test_build_cache_writes_all_files(tmp_path, monkeypatch):
+    """build_cache writes one JSON per (metric, attr) under CACHE_ROOT/<language>/."""
+    # Build a fake registry where every prompt attr is a _FakePrompt.
+    fake_registry = {name: type("M", (), {})() for name in builder.METRIC_PROMPT_ATTRS}
+    for name, attrs in builder.METRIC_PROMPT_ATTRS.items():
+        for attr in attrs:
+            setattr(fake_registry[name], attr, _FakePrompt())
+
+    monkeypatch.setattr(builder, "build_models", lambda j, e, s: (object(), object()))
+    monkeypatch.setattr(builder, "build_metric_registry", lambda llm, emb: fake_registry)
+    monkeypatch.setattr(builder, "CACHE_ROOT", tmp_path)
+
+    written = asyncio.run(builder.build_cache("zh", "gpt-5", "emb", object()))
+
+    expected_count = sum(len(a) for a in builder.METRIC_PROMPT_ATTRS.values())
+    assert len(written) == expected_count
+
+    sample = tmp_path / "zh" / "context_recall__prompt.json"
+    assert sample.exists()
+    data = json.loads(sample.read_text(encoding="utf-8"))
+    assert data["instruction"] == "中文指令。"
+    assert data["metric"] == "context_recall"

tests/test_judge_prompt_localizer.py

@@ -0,0 +1,121 @@
+"""Tests for the judge-prompt localizer (no RAGAS dependency; uses fake prompts)."""
+
+import json
+
+from pydantic import BaseModel
+
+from rag_eval.metrics import judge_prompts as jp
+
+
+class _In(BaseModel):
+    question: str
+
+
+class _Out(BaseModel):
+    statements: list[str]
+
+
+class _FakePrompt:
+    """Minimal stand-in for a RAGAS BasePrompt with overridable attributes."""
+
+    def __init__(self):
+        self.input_model = _In
+        self.output_model = _Out
+        self.instruction = "English instruction."
+        self.examples = [(_In(question="q"), _Out(statements=["s"]))]
+        self.language = "english"
+
+
+class _FakeMetric:
+    def __init__(self):
+        self.prompt = _FakePrompt()
+
+
+def _cache_dict(prompt):
+    """Build a valid cache dict for the given fake prompt."""
+    return {
+        "metric": "context_recall",
+        "prompt_attr": "prompt",
+        "language": "chinese",
+        "ragas_version": "0.4.3",
+        "source_hash": jp.prompt_source_hash(prompt),
+        "instruction": "中文指令。",
+        "examples": [{"input": {"question": "问题"}, "output": {"statements": ["陈述"]}}],
+    }
+
+
+def setup_function(_):
+    jp.reset_cache()
+
+
+def test_apply_localized_prompt_overrides_instruction_and_examples():
+    """apply_localized_prompt swaps instruction/examples and rebuilds example models."""
+    prompt = _FakePrompt()
+    jp.apply_localized_prompt(prompt, _cache_dict(prompt))
+    assert prompt.instruction == "中文指令。"
+    assert prompt.examples[0][0].question == "问题"
+    assert prompt.examples[0][1].statements == ["陈述"]
+    assert prompt.language == "chinese"
+
+
+def test_localize_english_is_noop():
+    """language='en' leaves the registry untouched."""
+    metric = _FakeMetric()
+    report = jp.localize_pipeline_prompts({"context_recall": metric}, "en")
+    assert report.applied == []
+    assert metric.prompt.instruction == "English instruction."
+
+
+def test_localize_applies_from_cache_file(tmp_path, monkeypatch):
+    """localize reads <root>/zh/context_recall__prompt.json and applies it."""
+    metric = _FakeMetric()
+    root = tmp_path / "configs" / "judge_prompts"
+    (root / "zh").mkdir(parents=True)
+    (root / "zh" / "context_recall__prompt.json").write_text(
+        json.dumps(_cache_dict(metric.prompt), ensure_ascii=False), encoding="utf-8"
+    )
+    monkeypatch.setattr(jp, "CACHE_ROOT", root)
+    report = jp.localize_pipeline_prompts({"context_recall": metric}, "zh")
+    assert "context_recall.prompt" in report.applied
+    assert metric.prompt.instruction == "中文指令。"
+
+
+def test_localize_missing_cache_keeps_english(tmp_path, monkeypatch):
+    """A missing cache file degrades gracefully to the English prompt with a warning."""
+    metric = _FakeMetric()
+    monkeypatch.setattr(jp, "CACHE_ROOT", tmp_path / "empty")
+    report = jp.localize_pipeline_prompts({"context_recall": metric}, "zh")
+    assert metric.prompt.instruction == "English instruction."
+    assert "context_recall.prompt" in report.skipped
+    assert report.warnings
+
+
+def test_localize_stale_hash_warns_but_applies(tmp_path, monkeypatch):
+    """A source_hash mismatch still applies Chinese but records a stale warning."""
+    metric = _FakeMetric()
+    data = _cache_dict(metric.prompt)
+    data["source_hash"] = "deadbeef"
+    root = tmp_path / "configs" / "judge_prompts"
+    (root / "zh").mkdir(parents=True)
+    (root / "zh" / "context_recall__prompt.json").write_text(
+        json.dumps(data, ensure_ascii=False), encoding="utf-8"
+    )
+    monkeypatch.setattr(jp, "CACHE_ROOT", root)
+    report = jp.localize_pipeline_prompts({"context_recall": metric}, "zh")
+    assert metric.prompt.instruction == "中文指令。"
+    assert any("stale" in w for w in report.warnings)
+
+
+def test_cache_memoized(tmp_path, monkeypatch):
+    """A second localize call does not re-read the file (in-memory memo)."""
+    metric = _FakeMetric()
+    root = tmp_path / "configs" / "judge_prompts"
+    (root / "zh").mkdir(parents=True)
+    path = root / "zh" / "context_recall__prompt.json"
+    path.write_text(json.dumps(_cache_dict(metric.prompt), ensure_ascii=False), encoding="utf-8")
+    monkeypatch.setattr(jp, "CACHE_ROOT", root)
+    jp.localize_pipeline_prompts({"context_recall": _FakeMetric()}, "zh")
+    path.unlink()  # delete file; memo should still serve the parsed data
+    metric2 = _FakeMetric()
+    report = jp.localize_pipeline_prompts({"context_recall": metric2}, "zh")
+    assert "context_recall.prompt" in report.applied

tests/test_metric_presenter.py

@@ -40,7 +40,14 @@ const result = {{
   desc: p.describeMetric("faithfulness"),
   noiseDesc: p.describeMetric("noise_sensitivity"),
   noiseBin: p.binColor("noise_sensitivity", 0.0),
-  faithBin: p.binColor("faithfulness", 0.8)
+  faithBin: p.binColor("faithfulness", 0.8),
+  lowerBetterNoise: p.isLowerBetter("noise_sensitivity"),
+  lowerBetterFaith: p.isLowerBetter("faithfulness"),
+  upHigher: p.deltaInfo("faithfulness", 0.80, 0.60),
+  downHigher: p.deltaInfo("faithfulness", 0.60, 0.80),
+  noiseImproved: p.deltaInfo("noise_sensitivity", 0.10, 0.30),
+  noiseWorse: p.deltaInfo("noise_sensitivity", 0.30, 0.10),
+  noBaseline: p.deltaInfo("faithfulness", 0.80, null)
 }};
 console.log(JSON.stringify(result));
 """
@@ -55,6 +62,15 @@ console.log(JSON.stringify(result));
     assert '"noiseDesc":"' in output
     assert '"noiseBin":"#16a34a"' in output
     assert '"faithBin":"#16a34a"' in output
+    assert '"lowerBetterNoise":true' in output
+    assert '"lowerBetterFaith":false' in output
+    # higher-better: rising value is an improvement (green ▲); falling is a regression (red ▼)
+    assert '"upHigher":{"hasData":true,"delta":0.2,"improved":true,"arrow":"▲","magnitude":"0.20","cls":"delta-good"}' in output
+    assert '"downHigher":{"hasData":true,"delta":-0.2,"improved":false,"arrow":"▼","magnitude":"0.20","cls":"delta-bad"}' in output
+    # noise_sensitivity (lower-better): falling value is an improvement (green ▼)
+    assert '"noiseImproved":{"hasData":true,"delta":-0.2,"improved":true,"arrow":"▼","magnitude":"0.20","cls":"delta-good"}' in output
+    assert '"noiseWorse":{"hasData":true,"delta":0.2,"improved":false,"arrow":"▲","magnitude":"0.20","cls":"delta-bad"}' in output
+    assert '"noBaseline":{"hasData":false' in output
 
 
 def test_report_and_index_load_metric_presenter_helper() -> None:
@@ -66,3 +82,6 @@ def test_report_and_index_load_metric_presenter_helper() -> None:
     assert "js/metric_presenter.js" in index_html
     assert "MetricPresenter.describeMetric" in report_js
     assert "MetricPresenter.scoreClass" in app_js
+    # history comparison table uses the direction-aware delta helper
+    assert "MetricPresenter.deltaInfo" in report_js
+    assert "history-table" in report_js

tests/test_metric_registry.py

@@ -0,0 +1,27 @@
+"""Tests for the shared metric registry factory."""
+
+from unittest.mock import MagicMock
+
+from ragas.llms.base import InstructorBaseRagasLLM
+from ragas.embeddings.base import BaseRagasEmbedding
+
+from rag_eval.metrics.factory import build_metric_registry
+
+
+def _mock_llm():
+    """Return a mock that passes RAGAS InstructorLLM type checks."""
+    return MagicMock(spec=InstructorBaseRagasLLM)
+
+
+def _mock_emb():
+    """Return a mock that passes RAGAS embedding type checks."""
+    return MagicMock(spec=BaseRagasEmbedding)
+
+
+def test_build_metric_registry_has_all_seven_metrics():
+    """The registry exposes every supported metric keyed by its canonical name."""
+    registry = build_metric_registry(llm=_mock_llm(), embeddings=_mock_emb())
+    assert set(registry) == {
+        "faithfulness", "answer_relevancy", "context_recall", "context_precision",
+        "noise_sensitivity", "factual_correctness", "semantic_similarity",
+    }

tests/test_question_history.py

@@ -0,0 +1,131 @@
+"""Tests for the cross-run question-history index used by the report page."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from webapp.services.question_history import (
+    build_question_history_index,
+    lookup,
+    normalize_question,
+)
+
+
+def _write_run(
+    run_dir: Path,
+    *,
+    run_id: str,
+    scenario_name: str,
+    finished_at: str,
+    rows: list[tuple[str, float]],
+    metric: str = "faithfulness",
+) -> None:
+    """Create a minimal run directory (metadata.json + scores.csv)."""
+    run_dir.mkdir(parents=True, exist_ok=True)
+    import json
+
+    (run_dir / "metadata.json").write_text(
+        json.dumps(
+            {
+                "run_id": run_id,
+                "scenario_name": scenario_name,
+                "finished_at": finished_at,
+                "valid_samples": len(rows),
+                "invalid_samples": 0,
+            }
+        ),
+        encoding="utf-8",
+    )
+    lines = [f"sample_id,question,{metric}"]
+    for i, (question, score) in enumerate(rows):
+        lines.append(f"s{i},{question},{score}")
+    (run_dir / "scores.csv").write_text("\n".join(lines), encoding="utf-8")
+
+
+# A question unlikely to collide with anything under the real outputs/ tree.
+_Q = "UNIQTESTQ ball tube lifetime evaluation method 9f3a"
+
+
+def test_normalize_question_is_case_and_whitespace_insensitive() -> None:
+    assert normalize_question("  Hello   World ") == normalize_question("hello world")
+
+
+def test_index_matches_question_across_runs_newest_first(tmp_path: Path) -> None:
+    _write_run(
+        tmp_path / "runA",
+        run_id="runA",
+        scenario_name="scnA",
+        finished_at="2026-01-01T00:00:00",
+        rows=[(_Q, 0.40)],
+    )
+    _write_run(
+        tmp_path / "runB",
+        run_id="runB",
+        scenario_name="scnB",
+        finished_at="2026-02-01T00:00:00",
+        rows=[(_Q, 0.80)],
+    )
+
+    index = build_question_history_index(extra_roots=[tmp_path])
+    entries = lookup(index, _Q)
+
+    assert [e["run_id"] for e in entries] == ["runB", "runA"]  # newest first
+    assert entries[0]["metrics"]["faithfulness"] == 0.80
+    assert entries[1]["scenario_name"] == "scnA"
+
+
+def test_index_excludes_current_run(tmp_path: Path) -> None:
+    _write_run(
+        tmp_path / "cur",
+        run_id="cur",
+        scenario_name="scn",
+        finished_at="2026-03-01T00:00:00",
+        rows=[(_Q, 0.50)],
+    )
+    _write_run(
+        tmp_path / "prev",
+        run_id="prev",
+        scenario_name="scn",
+        finished_at="2026-01-01T00:00:00",
+        rows=[(_Q, 0.60)],
+    )
+
+    index = build_question_history_index(exclude_run_id="cur", extra_roots=[tmp_path])
+    entries = lookup(index, _Q)
+
+    assert [e["run_id"] for e in entries] == ["prev"]
+
+
+def test_index_keeps_latest_occurrence_within_a_run(tmp_path: Path) -> None:
+    _write_run(
+        tmp_path / "run1",
+        run_id="run1",
+        scenario_name="scn",
+        finished_at="2026-01-01T00:00:00",
+        rows=[(_Q, 0.30), (_Q, 0.70)],  # same question twice
+    )
+
+    index = build_question_history_index(extra_roots=[tmp_path])
+    entries = lookup(index, _Q)
+
+    assert len(entries) == 1
+    assert entries[0]["metrics"]["faithfulness"] == 0.70  # last occurrence wins
+
+
+def test_lookup_caps_at_limit(tmp_path: Path) -> None:
+    for i in range(7):
+        _write_run(
+            tmp_path / f"r{i}",
+            run_id=f"r{i}",
+            scenario_name="scn",
+            finished_at=f"2026-01-0{i + 1}T00:00:00",
+            rows=[(_Q, 0.1 * i)],
+        )
+
+    index = build_question_history_index(extra_roots=[tmp_path])
+    assert len(lookup(index, _Q, limit=3)) == 3
+
+
+def test_lookup_unknown_question_returns_empty(tmp_path: Path) -> None:
+    index = build_question_history_index(extra_roots=[tmp_path])
+    assert lookup(index, "NO SUCH QUESTION zzz 0000") == []

tests/test_report_builder_token_usage.py

@@ -0,0 +1,55 @@
+"""Tests for token_usage passthrough in the webapp report builder."""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from webapp.services.report_builder import build_report
+
+
+def _write_minimal_run(run_dir: Path, token_usage: dict | None) -> None:
+    run_dir.mkdir(parents=True, exist_ok=True)
+    (run_dir / "scores.csv").write_text(
+        "sample_id,faithfulness\ns1,0.9\n", encoding="utf-8"
+    )
+    (run_dir / "summary.md").write_text("summary", encoding="utf-8")
+    metadata = {"run_id": run_dir.name}
+    if token_usage is not None:
+        metadata["token_usage"] = token_usage
+    (run_dir / "metadata.json").write_text(json.dumps(metadata), encoding="utf-8")
+
+
+def test_build_report_passes_through_token_usage(tmp_path: Path) -> None:
+    run_dir = tmp_path / "run"
+    _write_minimal_run(
+        run_dir,
+        token_usage={"gpt-5": {"input_tokens": 100, "output_tokens": 50, "calls": 2}},
+    )
+
+    report = build_report(run_dir, ["faithfulness"])
+
+    assert report.token_usage == {
+        "gpt-5": {"input_tokens": 100, "output_tokens": 50, "calls": 2}
+    }
+
+
+def test_build_report_defaults_token_usage_to_empty_dict(tmp_path: Path) -> None:
+    run_dir = tmp_path / "run"
+    _write_minimal_run(run_dir, token_usage=None)
+
+    report = build_report(run_dir, ["faithfulness"])
+
+    assert report.token_usage == {}
+
+
+def test_build_report_early_return_branch_still_surfaces_token_usage(tmp_path: Path) -> None:
+    """metrics=[] forces the early-return branch; token_usage must still surface."""
+    run_dir = tmp_path / "run"
+    _write_minimal_run(
+        run_dir,
+        token_usage={"gpt-5": {"input_tokens": 5, "output_tokens": 5, "calls": 1}},
+    )
+
+    report = build_report(run_dir, [])
+
+    assert report.token_usage == {"gpt-5": {"input_tokens": 5, "output_tokens": 5, "calls": 1}}

tests/test_reporting_summary_token_usage.py

@@ -0,0 +1,63 @@
+"""Tests for the '## Token 用量' section rendered by build_summary_markdown."""
+from __future__ import annotations
+
+from pathlib import Path
+
+from rag_eval.reporting.summary import build_summary_markdown
+from rag_eval.shared.models import DatasetConfig, EvaluationResult, RuntimeConfig, Scenario
+
+
+def _scenario(tmp_path: Path) -> Scenario:
+    return Scenario(
+        scenario_name="summary-token-test",
+        mode="offline",
+        dataset=DatasetConfig(path=tmp_path / "dataset.csv"),
+        judge_model="gpt-5",
+        embedding_model="embedding-model",
+        metrics=["faithfulness"],
+        output_dir=tmp_path / "outputs",
+        runtime=RuntimeConfig(batch_size=1),
+    )
+
+
+def test_summary_includes_token_usage_table(tmp_path: Path) -> None:
+    scenario = _scenario(tmp_path)
+    result = EvaluationResult(
+        scenario=scenario,
+        run_id="run-1",
+        started_at="t0",
+        finished_at="t1",
+        valid_samples=[],
+        invalid_samples=[],
+        score_rows=[{"sample_id": "s1", "faithfulness": 0.9, "error": ""}],
+        token_usage={
+            "gpt-5": {"input_tokens": 12450, "output_tokens": 3200, "calls": 60},
+            "Qwen/Qwen3-Embedding-4B": {"input_tokens": 45000, "output_tokens": 0, "calls": 30},
+        },
+    )
+
+    markdown = build_summary_markdown(result)
+
+    assert "## Token 用量" in markdown
+    assert "gpt-5" in markdown
+    assert "12450" in markdown
+    assert "Qwen/Qwen3-Embedding-4B" in markdown
+    assert "45000" in markdown
+
+
+def test_summary_shows_fallback_text_when_token_usage_empty(tmp_path: Path) -> None:
+    scenario = _scenario(tmp_path)
+    result = EvaluationResult(
+        scenario=scenario,
+        run_id="run-2",
+        started_at="t0",
+        finished_at="t1",
+        valid_samples=[],
+        invalid_samples=[],
+        score_rows=[{"sample_id": "s1", "faithfulness": 0.9, "error": ""}],
+    )
+
+    markdown = build_summary_markdown(result)
+
+    assert "## Token 用量" in markdown
+    assert "未记录 token 用量" in markdown

tests/test_score_endpoint_language.py

@@ -0,0 +1,46 @@
+"""Tests that score endpoints forward the resolved judge_language to the scorer."""
+
+from unittest.mock import MagicMock, patch
+
+from fastapi.testclient import TestClient
+
+import webapp.api.score as score_mod
+from webapp.server import create_app
+
+
+def _client():
+    return TestClient(create_app())
+
+
+def test_score_route_forwards_judge_language(monkeypatch):
+    """A request with judge_language='zh' reaches inline_scorer.score."""
+    captured: dict = {}
+
+    def fake_score(**kwargs):
+        captured.update(kwargs)
+        return {"faithfulness": 0.9}
+
+    monkeypatch.setattr(score_mod.inline_scorer, "score", fake_score)
+    resp = _client().post("/api/score", json={
+        "question": "q", "answer": "a", "contexts": "c",
+        "ground_truth": "g", "metrics": ["faithfulness"], "judge_language": "zh",
+    })
+    assert resp.status_code == 200
+    assert captured.get("judge_language") == "zh"
+
+
+def test_score_route_defaults_language_to_en(monkeypatch):
+    """Omitting judge_language falls back to settings default ('en')."""
+    captured: dict = {}
+
+    def fake_score(**kwargs):
+        captured.update(kwargs)
+        return {"faithfulness": 0.9}
+
+    monkeypatch.setattr(score_mod.inline_scorer, "score", fake_score)
+    resp = _client().post("/api/score", json={
+        "question": "q", "answer": "a", "contexts": "c",
+        "ground_truth": "g", "metrics": ["faithfulness"],
+    })
+    assert resp.status_code == 200
+    assert captured.get("judge_language") == "en"

tests/test_token_tracker.py

@@ -0,0 +1,83 @@
+"""Tests for the per-run token usage accumulator and its context-scoped activation."""
+from __future__ import annotations
+
+from rag_eval.metrics.token_tracker import (
+    TokenUsageTracker,
+    get_current_tracker,
+    track_token_usage,
+)
+
+
+def test_record_accumulates_input_output_and_calls():
+    tracker = TokenUsageTracker()
+    tracker.record("gpt-5", 100, 50)
+    tracker.record("gpt-5", 20, 10)
+    assert tracker.as_dict() == {
+        "gpt-5": {"input_tokens": 120, "output_tokens": 60, "calls": 2}
+    }
+
+
+def test_record_groups_by_model_name():
+    tracker = TokenUsageTracker()
+    tracker.record("gpt-5", 100, 50)
+    tracker.record("Qwen/Qwen3-Embedding-4B", 30, 0)
+    result = tracker.as_dict()
+    assert set(result.keys()) == {"gpt-5", "Qwen/Qwen3-Embedding-4B"}
+    assert result["Qwen/Qwen3-Embedding-4B"] == {
+        "input_tokens": 30, "output_tokens": 0, "calls": 1
+    }
+
+
+def test_record_defaults_blank_model_to_unknown():
+    tracker = TokenUsageTracker()
+    tracker.record("", 10, 5)
+    assert "unknown" in tracker.as_dict()
+
+
+def test_merge_into_sums_with_existing_totals():
+    tracker = TokenUsageTracker()
+    tracker.record("gpt-5", 100, 50)
+    existing = {"gpt-5": {"input_tokens": 200, "output_tokens": 100, "calls": 3}}
+    merged = tracker.merge_into(existing)
+    assert merged == {"gpt-5": {"input_tokens": 300, "output_tokens": 150, "calls": 4}}
+
+
+def test_merge_into_keeps_models_only_in_existing():
+    tracker = TokenUsageTracker()
+    tracker.record("gpt-5", 10, 5)
+    existing = {"other-model": {"input_tokens": 1, "output_tokens": 1, "calls": 1}}
+    merged = tracker.merge_into(existing)
+    assert merged["other-model"] == {"input_tokens": 1, "output_tokens": 1, "calls": 1}
+    assert merged["gpt-5"] == {"input_tokens": 10, "output_tokens": 5, "calls": 1}
+
+
+def test_merge_into_does_not_mutate_existing_dict():
+    tracker = TokenUsageTracker()
+    tracker.record("gpt-5", 10, 5)
+    existing = {"gpt-5": {"input_tokens": 1, "output_tokens": 1, "calls": 1}}
+    tracker.merge_into(existing)
+    assert existing == {"gpt-5": {"input_tokens": 1, "output_tokens": 1, "calls": 1}}
+
+
+def test_get_current_tracker_returns_none_outside_context():
+    assert get_current_tracker() is None
+
+
+def test_track_token_usage_activates_and_resets_context():
+    assert get_current_tracker() is None
+    with track_token_usage() as tracker:
+        assert get_current_tracker() is tracker
+        tracker.record("gpt-5", 1, 1)
+    assert get_current_tracker() is None
+
+
+def test_track_token_usage_nested_contexts_are_isolated():
+    with track_token_usage() as outer:
+        outer.record("outer-model", 5, 5)
+        with track_token_usage() as inner:
+            inner.record("inner-model", 1, 1)
+            assert get_current_tracker() is inner
+        assert get_current_tracker() is outer
+    assert outer.as_dict() == {
+        "outer-model": {"input_tokens": 5, "output_tokens": 5, "calls": 1}
+    }

tests/test_token_usage_hook.py

@@ -0,0 +1,75 @@
+"""Tests for the token-usage HTTP response hook and attach_usage_hook wiring."""
+from __future__ import annotations
+
+import asyncio
+import json
+
+import httpx
+
+from rag_eval.metrics.factory import _usage_response_hook, attach_usage_hook
+from rag_eval.metrics.token_tracker import track_token_usage
+
+
+def _fake_response(payload: dict | None) -> httpx.Response:
+    """Build a real httpx.Response with a JSON (or broken) body for hook testing."""
+    content = b"not json" if payload is None else json.dumps(payload).encode("utf-8")
+    return httpx.Response(200, content=content, request=httpx.Request("POST", "http://test/x"))
+
+
+class TestUsageResponseHook:
+    def test_records_usage_when_tracker_active(self):
+        with track_token_usage() as tracker:
+            response = _fake_response({
+                "model": "gpt-5",
+                "usage": {"prompt_tokens": 120, "completion_tokens": 45},
+            })
+            asyncio.run(_usage_response_hook(response))
+        assert tracker.as_dict() == {
+            "gpt-5": {"input_tokens": 120, "output_tokens": 45, "calls": 1}
+        }
+
+    def test_noop_when_no_tracker_active(self):
+        response = _fake_response({"model": "gpt-5", "usage": {"prompt_tokens": 1, "completion_tokens": 1}})
+        # Must not raise even though no tracker is active.
+        asyncio.run(_usage_response_hook(response))
+
+    def test_noop_when_response_has_no_usage_field(self):
+        with track_token_usage() as tracker:
+            response = _fake_response({"model": "gpt-5"})
+            asyncio.run(_usage_response_hook(response))
+        assert tracker.as_dict() == {}
+
+    def test_noop_on_non_json_response(self):
+        with track_token_usage() as tracker:
+            response = _fake_response(None)
+            asyncio.run(_usage_response_hook(response))
+        assert tracker.as_dict() == {}
+
+    def test_embedding_response_without_completion_tokens_defaults_output_to_zero(self):
+        """Embeddings responses omit completion_tokens; output should default to 0."""
+        with track_token_usage() as tracker:
+            response = _fake_response({
+                "model": "Qwen/Qwen3-Embedding-4B",
+                "usage": {"prompt_tokens": 30, "total_tokens": 30},
+            })
+            asyncio.run(_usage_response_hook(response))
+        assert tracker.as_dict() == {
+            "Qwen/Qwen3-Embedding-4B": {"input_tokens": 30, "output_tokens": 0, "calls": 1}
+        }
+
+
+class TestAttachUsageHook:
+    def test_attaches_hook_to_client_event_hooks(self):
+        from openai import AsyncOpenAI
+
+        client = AsyncOpenAI(api_key="sk-test", base_url="http://localhost:1")
+        attach_usage_hook(client)
+        assert _usage_response_hook in client._client.event_hooks["response"]
+
+    def test_idempotent_when_called_twice_on_same_client(self):
+        from openai import AsyncOpenAI
+
+        client = AsyncOpenAI(api_key="sk-test", base_url="http://localhost:1")
+        attach_usage_hook(client)
+        attach_usage_hook(client)
+        assert client._client.event_hooks["response"].count(_usage_response_hook) == 1

tests/test_token_usage_persistence.py

@@ -0,0 +1,75 @@
+"""Tests that EvaluationResult.token_usage is persisted into metadata.json."""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from rag_eval.reporting.writers import write_run_artifacts
+from rag_eval.shared.models import DatasetConfig, EvaluationResult, RuntimeConfig, Scenario
+
+
+def _scenario(tmp_path: Path) -> Scenario:
+    return Scenario(
+        scenario_name="token-persist-test",
+        mode="offline",
+        dataset=DatasetConfig(path=tmp_path / "dataset.csv"),
+        judge_model="gpt-5",
+        embedding_model="embedding-model",
+        metrics=["faithfulness"],
+        output_dir=tmp_path / "outputs",
+        runtime=RuntimeConfig(batch_size=1),
+    )
+
+
+def test_evaluation_result_defaults_token_usage_to_empty_dict(tmp_path: Path) -> None:
+    result = EvaluationResult(
+        scenario=_scenario(tmp_path),
+        run_id="run-1",
+        started_at="t0",
+        finished_at="t1",
+        valid_samples=[],
+        invalid_samples=[],
+        score_rows=[],
+    )
+    assert result.token_usage == {}
+
+
+def test_write_run_artifacts_persists_token_usage(tmp_path: Path) -> None:
+    scenario = _scenario(tmp_path)
+    result = EvaluationResult(
+        scenario=scenario,
+        run_id="run-2",
+        started_at="t0",
+        finished_at="t1",
+        valid_samples=[],
+        invalid_samples=[],
+        score_rows=[],
+        token_usage={"gpt-5": {"input_tokens": 100, "output_tokens": 40, "calls": 2}},
+    )
+
+    write_run_artifacts(result)
+
+    metadata_path = scenario.output_dir / "run-2" / "metadata.json"
+    metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {
+        "gpt-5": {"input_tokens": 100, "output_tokens": 40, "calls": 2}
+    }
+
+
+def test_write_run_artifacts_writes_empty_token_usage_when_unset(tmp_path: Path) -> None:
+    scenario = _scenario(tmp_path)
+    result = EvaluationResult(
+        scenario=scenario,
+        run_id="run-3",
+        started_at="t0",
+        finished_at="t1",
+        valid_samples=[],
+        invalid_samples=[],
+        score_rows=[],
+    )
+
+    write_run_artifacts(result)
+
+    metadata_path = scenario.output_dir / "run-3" / "metadata.json"
+    metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {}

tests/test_webapp_report_builder.py

@@ -2,10 +2,15 @@
 
 from __future__ import annotations
 
+import json
 from pathlib import Path
 
+import pandas as pd
 import pytest
 
+from webapp.models import AdvisorComparison, AdvisorComparisonEntry
+from webapp.services import question_history
+from webapp.services import report_builder
 from webapp.services.report_builder import build_report
 from webapp.services.run_reader import _infer_metrics_from_scores, _read_weights_from_snapshot
 
@@ -115,3 +120,131 @@ def test_build_report_ranks_noise_sensitivity_with_lower_values_as_better(tmp_pa
         "s-warn",
         "s-good",
     ]
+
+
+def test_lowest_samples_attaches_history_from_index() -> None:
+    """Surfaced samples are annotated with the same question's prior-run scores."""
+    frame = pd.DataFrame(
+        [
+            {"sample_id": "s1", "question": "  How LONG is the tube? ", "faithfulness": 0.40},
+            {"sample_id": "s2", "question": "unrelated question", "faithfulness": 0.30},
+        ]
+    )
+    history_index = {
+        question_history.normalize_question("How long is the tube?"): [
+            {
+                "run_id": "prev",
+                "scenario_name": "scn",
+                "finished_at": "2026-01-01T00:00:00",
+                "metrics": {"faithfulness": 0.90},
+            }
+        ]
+    }
+
+    samples = report_builder._lowest_samples(frame, ["faithfulness"], history_index)
+    by_id = {s.sample_id: s for s in samples}
+
+    assert len(by_id["s1"].history) == 1
+    assert by_id["s1"].history[0].run_id == "prev"
+    assert by_id["s1"].history[0].metrics["faithfulness"] == 0.90
+    assert by_id["s2"].history == []  # no match → no history
+
+
+def test_build_report_attaches_question_history(tmp_path: Path, monkeypatch) -> None:
+    """build_report wires the question-history index into surfaced samples."""
+    run_dir = tmp_path / "run"
+    run_dir.mkdir(parents=True, exist_ok=True)
+    (run_dir / "scores.csv").write_text(
+        "\n".join(
+            [
+                "sample_id,question,faithfulness",
+                "s1,How long is the tube?,0.40",
+            ]
+        ),
+        encoding="utf-8",
+    )
+    (run_dir / "summary.md").write_text("summary", encoding="utf-8")
+    (run_dir / "optimization_advice.md").write_text("", encoding="utf-8")
+
+    captured: dict = {}
+
+    def _fake_index(exclude_run_id=None, extra_roots=None):
+        captured["exclude_run_id"] = exclude_run_id
+        return {
+            question_history.normalize_question("How long is the tube?"): [
+                {
+                    "run_id": "older",
+                    "scenario_name": "scn",
+                    "finished_at": "2026-01-01T00:00:00",
+                    "metrics": {"faithfulness": 0.95},
+                }
+            ]
+        }
+
+    monkeypatch.setattr(question_history, "build_question_history_index", _fake_index)
+
+    report = build_report(run_dir, ["faithfulness"])
+
+    assert captured["exclude_run_id"] == "run"  # current run excluded from history
+    assert report.lowest_samples[0].history[0].run_id == "older"
+    assert report.lowest_samples[0].history[0].metrics["faithfulness"] == 0.95
+
+
+def test_build_report_attaches_advisor_comparison(tmp_path: Path, monkeypatch) -> None:
+    """build_report wires advisor_comparison.build_advisor_comparison() into ReportData."""
+    run_dir = tmp_path / "run"
+    _write_run_artifacts(run_dir)
+    (run_dir / "metadata.json").write_text(
+        json.dumps({"run_id": "run-1", "scenario_name": "my-scenario"}), encoding="utf-8"
+    )
+
+    fake_comparison = AdvisorComparison(
+        previous_run_id="prev-1",
+        previous_finished_at="2026-01-01T00:00:00+00:00",
+        entries=[
+            AdvisorComparisonEntry(
+                metric="faithfulness",
+                status="resolved",
+                previous_score=0.5,
+                previous_severity="warning",
+                current_score=None,
+                current_severity=None,
+            )
+        ],
+    )
+
+    captured_args = {}
+
+    def _fake_build(run_dir_arg, scenario_name_arg, metrics_arg):
+        captured_args["scenario_name"] = scenario_name_arg
+        captured_args["metrics"] = metrics_arg
+        return fake_comparison
+
+    monkeypatch.setattr(
+        report_builder.advisor_comparison, "build_advisor_comparison", _fake_build
+    )
+
+    report = build_report(run_dir, ["faithfulness", "context_recall"])
+
+    assert report.advisor_comparison == fake_comparison
+    assert captured_args["scenario_name"] == "my-scenario"
+    assert captured_args["metrics"] == ["faithfulness", "context_recall"]
+
+
+def test_build_report_advisor_comparison_none_when_no_previous_run(
+    tmp_path: Path, monkeypatch
+) -> None:
+    """build_report leaves advisor_comparison as None when no predecessor exists."""
+    run_dir = tmp_path / "run"
+    _write_run_artifacts(run_dir)
+    (run_dir / "metadata.json").write_text(
+        json.dumps({"run_id": "run-1", "scenario_name": "my-scenario"}), encoding="utf-8"
+    )
+
+    monkeypatch.setattr(
+        report_builder.advisor_comparison, "build_advisor_comparison", lambda *a, **k: None
+    )
+
+    report = build_report(run_dir, ["faithfulness", "context_recall"])
+
+    assert report.advisor_comparison is None

tests/test_zh_cache_integration.py

@@ -0,0 +1,56 @@
+"""Integration test: committed zh cache files load cleanly onto real RAGAS metric instances."""
+
+from pathlib import Path
+
+from rag_eval.metrics.judge_prompts import CACHE_ROOT, METRIC_PROMPT_ATTRS
+
+
+def test_zh_cache_files_all_present():
+    """All expected zh cache files are committed and present on disk."""
+    zh_dir = CACHE_ROOT / "zh"
+    for metric, attrs in METRIC_PROMPT_ATTRS.items():
+        for attr in attrs:
+            path = zh_dir / f"{metric}__{attr}.json"
+            assert path.exists(), f"Missing committed cache: {path}"
+
+
+def test_zh_cache_loads_onto_real_metrics():
+    """Localize all registry metrics to zh — zero skips, zero errors."""
+    from unittest.mock import MagicMock
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+    from rag_eval.metrics.factory import build_metric_registry
+    from rag_eval.metrics.judge_prompts import localize_pipeline_prompts, reset_cache
+
+    reset_cache()
+    llm = MagicMock(spec=InstructorBaseRagasLLM)
+    emb = MagicMock(spec=BaseRagasEmbedding)
+    registry = build_metric_registry(llm, emb)
+    report = localize_pipeline_prompts(registry, "zh")
+
+    expected_count = sum(len(attrs) for attrs in METRIC_PROMPT_ATTRS.values())
+    assert len(report.applied) == expected_count, (
+        f"Expected {expected_count} applied, got {len(report.applied)}; "
+        f"skipped={report.skipped}, warnings={report.warnings}"
+    )
+    assert not report.skipped, f"Unexpected skips: {report.skipped}"
+
+
+def test_zh_cache_instructions_are_chinese():
+    """After localization, faithfulness instruction starts with Chinese characters."""
+    from unittest.mock import MagicMock
+    from ragas.llms.base import InstructorBaseRagasLLM
+    from ragas.embeddings.base import BaseRagasEmbedding
+    from rag_eval.metrics.factory import build_metric_registry
+    from rag_eval.metrics.judge_prompts import localize_pipeline_prompts, reset_cache
+
+    reset_cache()
+    llm = MagicMock(spec=InstructorBaseRagasLLM)
+    emb = MagicMock(spec=BaseRagasEmbedding)
+    registry = build_metric_registry(llm, emb)
+    localize_pipeline_prompts(registry, "zh")
+
+    instr = registry["faithfulness"].statement_generator_prompt.instruction
+    # Chinese instruction should contain at least one CJK character.
+    has_chinese = any("\u4e00" <= ch <= "\u9fff" for ch in instr)
+    assert has_chinese, f"Instruction does not contain Chinese: {instr[:60]}"

tests/webapp/test_score_job_manager_token_usage.py

@@ -0,0 +1,73 @@
+"""Tests that /api/score/async persists token usage captured during scoring."""
+from __future__ import annotations
+
+import json
+import time
+
+from webapp.models import ScoreRequest
+from webapp.services.score_job_manager import ScoreJobManager
+
+
+def _wait_for_status(mgr: ScoreJobManager, job_id: str, timeout: float = 2.0):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        status = mgr.get(job_id)
+        if status is not None and status.status in ("completed", "failed"):
+            return status
+        time.sleep(0.02)
+    raise TimeoutError(f"job {job_id} did not complete in time")
+
+
+def test_run_writes_token_usage_to_metadata(tmp_path, monkeypatch):
+    """_run() wraps inline_scorer.score in track_token_usage and persists totals."""
+    from rag_eval.metrics.token_tracker import get_current_tracker
+
+    mgr = ScoreJobManager(
+        output_dir=tmp_path / "score-async",
+        index_dir=tmp_path / "score-jobs",
+        max_workers=1,
+    )
+
+    def _fake_score(**kwargs):
+        tracker = get_current_tracker()
+        if tracker is not None:
+            tracker.record("gpt-5", 120, 45)
+        return {m: 0.9 for m in kwargs["metrics"]}
+
+    monkeypatch.setattr(
+        "webapp.services.inline_scorer.inline_scorer.score", _fake_score
+    )
+
+    request = ScoreRequest(question="q?", answer="a.", metrics=["answer_relevancy"])
+    status = mgr.submit(request)
+    final_status = _wait_for_status(mgr, status.job_id)
+
+    assert final_status.status == "completed"
+    run_dir = tmp_path / "score-async" / final_status.run_id
+    metadata = json.loads((run_dir / "metadata.json").read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {
+        "gpt-5": {"input_tokens": 120, "output_tokens": 45, "calls": 1}
+    }
+
+
+def test_run_writes_empty_token_usage_when_nothing_recorded(tmp_path, monkeypatch):
+    mgr = ScoreJobManager(
+        output_dir=tmp_path / "score-async",
+        index_dir=tmp_path / "score-jobs",
+        max_workers=1,
+    )
+
+    def _fake_score(**kwargs):
+        return {m: 0.9 for m in kwargs["metrics"]}
+
+    monkeypatch.setattr(
+        "webapp.services.inline_scorer.inline_scorer.score", _fake_score
+    )
+
+    request = ScoreRequest(question="q?", answer="a.", metrics=["answer_relevancy"])
+    status = mgr.submit(request)
+    final_status = _wait_for_status(mgr, status.job_id)
+
+    run_dir = tmp_path / "score-async" / final_status.run_id
+    metadata = json.loads((run_dir / "metadata.json").read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {}

tests/webapp/test_session_score_manager_token_usage.py

@@ -0,0 +1,84 @@
+"""Tests that session-grouped async scoring accumulates token usage across calls."""
+from __future__ import annotations
+
+import json
+import time
+
+from webapp.models import ScoreRequest
+from webapp.services.session_score_manager import SessionScoreJobManager
+
+
+def _wait_for_call_count(mgr: SessionScoreJobManager, session_id: str, expected: int, timeout: float = 2.0):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        session = mgr.get_session(session_id)
+        if session is not None and session.call_count >= expected:
+            all_done = all(j.status in ("completed", "failed") for j in session.jobs)
+            if all_done:
+                return session
+        time.sleep(0.02)
+    raise TimeoutError(f"session {session_id} did not reach {expected} completed calls in time")
+
+
+def test_session_accumulates_token_usage_across_calls(tmp_path, monkeypatch):
+    from rag_eval.metrics.token_tracker import get_current_tracker
+
+    mgr = SessionScoreJobManager(
+        output_dir=tmp_path / "score-session",
+        index_dir=tmp_path / "score-session-jobs",
+        max_workers=1,
+    )
+
+    call_usages = iter([(100, 40), (30, 10)])
+
+    def _fake_score(**kwargs):
+        tracker = get_current_tracker()
+        input_tok, output_tok = next(call_usages)
+        if tracker is not None:
+            tracker.record("gpt-5", input_tok, output_tok)
+        return {m: 0.9 for m in kwargs["metrics"]}
+
+    monkeypatch.setattr(
+        "webapp.services.inline_scorer.inline_scorer.score", _fake_score
+    )
+
+    request = ScoreRequest(question="q?", answer="a.", metrics=["answer_relevancy"])
+
+    _, run_id = mgr.submit("session-token-test", request)
+    _wait_for_call_count(mgr, "session-token-test", 1)
+    mgr.submit("session-token-test", request)
+    _wait_for_call_count(mgr, "session-token-test", 2)
+
+    run_dir = tmp_path / "score-session" / run_id
+    metadata = json.loads((run_dir / "metadata.json").read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {
+        "gpt-5": {"input_tokens": 130, "output_tokens": 50, "calls": 2}
+    }
+
+
+def test_session_first_call_writes_token_usage_from_scratch(tmp_path, monkeypatch):
+    from rag_eval.metrics.token_tracker import get_current_tracker
+
+    mgr = SessionScoreJobManager(
+        output_dir=tmp_path / "score-session",
+        index_dir=tmp_path / "score-session-jobs",
+        max_workers=1,
+    )
+
+    def _fake_score(**kwargs):
+        tracker = get_current_tracker()
+        if tracker is not None:
+            tracker.record("gpt-5", 50, 20)
+        return {m: 0.9 for m in kwargs["metrics"]}
+
+    monkeypatch.setattr(
+        "webapp.services.inline_scorer.inline_scorer.score", _fake_score
+    )
+
+    request = ScoreRequest(question="q?", answer="a.", metrics=["answer_relevancy"])
+    _, run_id = mgr.submit("session-first-call-test", request)
+    _wait_for_call_count(mgr, "session-first-call-test", 1)
+
+    run_dir = tmp_path / "score-session" / run_id
+    metadata = json.loads((run_dir / "metadata.json").read_text(encoding="utf-8"))
+    assert metadata["token_usage"] == {"gpt-5": {"input_tokens": 50, "output_tokens": 20, "calls": 1}}

webapp/api/score.py

@@ -91,6 +91,12 @@ def score_sample(
     - `factual_correctness` — 回答与参考答案的事实准确性（需 ground_truth）
     - `semantic_similarity` — 回答与参考答案的语义相似度（需 ground_truth）
 
+    **中文评判 `judge_language`**：
+    - 默认 `"en"`（使用英文评判 prompt）。
+    - 传 `"zh"` 开启中文评判 prompt，提升对中文语料的判定一致性（架构设计 §10.5）。
+    - 可通过 `.env` 设置 `RAGAS_JUDGE_LANGUAGE=zh` 作为全局默认，请求字段优先级更高。
+    - 注意：`noise_sensitivity` 使用内置函数式 prompt，不受此参数影响，始终保持英文。
+
     **推荐模型配置**：
     - `judge_model`: `gpt-5`
     - `embedding_model`: `text-embedding-3-small`
@@ -116,6 +122,7 @@ def score_sample(
 
     judge_model = request.judge_model or settings.ragas_judge_model
     embedding_model = request.embedding_model or settings.ragas_embedding_model
+    judge_language = request.judge_language or settings.ragas_judge_language
     effective = request.effective_metrics()
     requested = set(request.metrics)
     skipped = sorted(requested - set(effective))
@@ -139,6 +146,7 @@ def score_sample(
             judge_model=judge_model,
             embedding_model=embedding_model,
             settings=settings,
+            judge_language=judge_language,
         )
     except Exception as exc:  # noqa: BLE001
         latency_ms = int((time.monotonic() - t0) * 1000)

webapp/api/score_jobs.py

@@ -45,10 +45,13 @@ def submit_async_score(request: ScoreRequest) -> AsyncScoreJobResponse:
     **适合 Dify 工作流**：HTTP 节点无需等待评分完成（无超时风险），
     工作流立即继续，评分结果在 RAGAS 平台「运行列表」中查看。
 
-    评分完成后自动生成：
+    接受与 `POST /api/score` 完全相同的请求体（含 `judge_language`），
+    但以异步方式在后台运行，完成后自动生成：
     - 各指标得分（`scores.csv`）
     - 摘要报告（`summary.md`）
     - LLM 优化建议（`optimization_advice.md`）
+
+    **`judge_language`**：传 `"zh"` 开启中文评判 prompt；默认 `null`（使用全局配置，出厂为 `"en"`）。
     """
     logger.info(
         "[score_async] submit  metrics=%s  has_ctx=%s  has_gt=%s",

webapp/api/session_score_jobs.py

@@ -54,7 +54,8 @@ logger = logging.getLogger("webapp.api.session_score_jobs")
         "- `contexts`：检索片段拼接字符串，按 `context_separator` 拆分。\n"
         "- `ground_truth`：标准答案，可选；缺失时会自动跳过依赖它的指标。\n"
         "- `metrics`：本次需要计算的指标列表。\n"
-        "- `judge_model` / `embedding_model`：可选；为空时回退到系统默认配置。\n\n"
+        "- `judge_model` / `embedding_model`：可选；为空时回退到系统默认配置。\n"
+        "- `judge_language`：评判 prompt 语言，`\"zh\"` 开启中文评判，默认 `null`（全局配置，出厂为 `\"en\"`）。\n\n"
         "**处理行为**\n"
         "1. 服务端立即返回 `202 Accepted`，并生成本次调用的 `job_id`。\n"
         "2. 系统根据 `session_id` 计算固定 `run_id`，格式为 `session-<sanitized-session_id>`。\n"
@@ -75,7 +76,8 @@ logger = logging.getLogger("webapp.api.session_score_jobs")
         "  \"context_separator\": \" |||| \",\n"
         "  \"metrics\": [\"answer_relevancy\", \"faithfulness\"],\n"
         "  \"judge_model\": \"gpt-5.5\",\n"
-        "  \"embedding_model\": \"text-embedding-3-small\"\n"
+        "  \"embedding_model\": \"text-embedding-3-small\",\n"
+        "  \"judge_language\": \"zh\"\n"
         "}\n"
         "```"
     ),
@@ -113,6 +115,8 @@ def submit_session_async_score(request: SessionScoreRequest) -> SessionScoreJobR
 
     **适合 Dify 工作流**：在循环节点中批量调用，所有轮次共用同一 `session_id`，
     最终在 RAGAS 平台「运行列表」中查看完整的批量评估报告。
+
+    **`judge_language`**：传 `"zh"` 开启中文评判 prompt；默认 `null`（使用全局配置，出厂为 `"en"`）。
     """
     logger.info(
         "[session_async] submit  session_id=%s  metrics=%s  has_ctx=%s  has_gt=%s",

webapp/models.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 from datetime import datetime, timezone
-from typing import Any
+from typing import Any, Literal
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator
 
@@ -48,6 +48,15 @@ class DistributionBin(BaseModel):
     count: int
 
 
+class SampleHistoryEntry(BaseModel):
+    """One past evaluation of the same question, for the history comparison table."""
+
+    run_id: str
+    scenario_name: str = ""
+    finished_at: str = ""
+    metrics: dict[str, float | None] = Field(default_factory=dict)
+
+
 class SampleScore(BaseModel):
     """Per-sample row used for the lowest-score review table."""
 
@@ -62,6 +71,32 @@ class SampleScore(BaseModel):
     metrics: dict[str, float | None] = Field(default_factory=dict)
     mean_score: float | None = None
     error: str = ""
+    history: list[SampleHistoryEntry] = Field(
+        default_factory=list,
+        description="同一问题在以往运行中的评分（按时间倒序），用于历史对比。",
+    )
+
+
+class AdvisorComparisonEntry(BaseModel):
+    """One metric's diagnosis delta between the current run and its predecessor."""
+
+    metric: str
+    status: Literal["resolved", "regressed", "still_triggered", "new_metric"]
+    previous_score: float | None = None
+    previous_severity: str | None = None
+    current_score: float | None = None
+    current_severity: str | None = None
+
+
+class AdvisorComparison(BaseModel):
+    """Advisor-diagnosis comparison against the same scenario's previous run."""
+
+    previous_run_id: str
+    previous_finished_at: str
+    previous_judge_model: str = ""
+    current_judge_model: str = ""
+    judge_model_changed: bool = False
+    entries: list[AdvisorComparisonEntry] = Field(default_factory=list)
 
 
 class ReportData(BaseModel):
@@ -86,6 +121,14 @@ class ReportData(BaseModel):
         default_factory=dict,
         description="该次运行使用的文档权重配置（来自 scenario.snapshot.yaml）。",
     )
+    token_usage: dict[str, dict[str, int]] = Field(
+        default_factory=dict,
+        description="按模型累计的 token 用量：{model: {input_tokens, output_tokens, calls}}。",
+    )
+    advisor_comparison: AdvisorComparison | None = Field(
+        default=None,
+        description="相比同场景上一次运行的顾问诊断差异；无可比对象时为 None。",
+    )
 
 
 class RunDetail(BaseModel):
@@ -430,6 +473,7 @@ class ScoreRequest(BaseModel):
                 ],
                 "judge_model": "gpt-5",
                 "embedding_model": "text-embedding-3-small",
+                "judge_language": "zh",
             }
         }
     )
@@ -460,6 +504,10 @@ class ScoreRequest(BaseModel):
         default=None,
         description="Embedding 模型名称；为 null 时使用 .env 中的 RAGAS_EMBEDDING_MODEL。",
     )
+    judge_language: str | None = Field(
+        default=None,
+        description="评判 prompt 语言；'zh' 启用中文评判，为 null 时使用 RAGAS_JUDGE_LANGUAGE（默认 en）。",
+    )
 
     @field_validator("metrics")
     @classmethod
@@ -556,6 +604,7 @@ class SessionScoreRequest(ScoreRequest):
                         "metrics": ["answer_relevancy", "faithfulness"],
                         "judge_model": "gpt-5.5",
                         "embedding_model": "text-embedding-3-small",
+                        "judge_language": "zh",
                     },
                 }
             ]

webapp/services/advisor_comparison.py

@@ -0,0 +1,149 @@
+"""Advisor-diagnosis comparison against the same scenario's previous run.
+
+The report detail page shows, for the run currently being viewed, whether
+metrics flagged by the optimization advisor in the immediately preceding run
+of the same scenario have since improved, regressed, or remain unresolved.
+
+No new artifact files are written: diagnoses are recomputed on demand from
+each run's scores.csv via the existing rag_eval.advisor.diagnose(), so the
+comparison always reflects the current threshold rules in rules.py.
+"""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+from typing import Any
+
+import pandas as pd
+
+from rag_eval.advisor import diagnose
+from webapp.models import AdvisorComparison, AdvisorComparisonEntry, RunSummary
+from webapp.services import run_reader
+
+logger = logging.getLogger("webapp.services.advisor_comparison")
+
+# Worse-first ordering so regressions and unresolved issues surface above
+# resolved ones in the rendered comparison list.
+_STATUS_ORDER = {"regressed": 0, "still_triggered": 1, "new_metric": 2, "resolved": 3}
+
+
+def find_previous_run(
+    scenario_name: str,
+    current_run_id: str,
+    current_finished_at: str,
+    extra_roots: list[Path] | None = None,
+) -> RunSummary | None:
+    """Return the most recent prior run with the same scenario_name, or None.
+
+    "Prior" means finished_at strictly earlier than current_finished_at, so a
+    run that finished later (e.g. a concurrent run) is never mistaken for a
+    historical baseline.
+    """
+    candidates = [
+        summary
+        for summary in run_reader.list_run_summaries(extra_roots)
+        if summary.scenario_name == scenario_name
+        and summary.run_id != current_run_id
+        and (summary.finished_at or "") < (current_finished_at or "")
+    ]
+    if not candidates:
+        return None
+    candidates.sort(key=lambda summary: summary.finished_at or "", reverse=True)
+    return candidates[0]
+
+
+def _rows_as_records(frame: pd.DataFrame) -> list[dict[str, Any]]:
+    """Convert a scores dataframe into plain dict records, NaN -> None."""
+    if frame.empty:
+        return []
+    return frame.where(pd.notnull(frame), None).to_dict("records")
+
+
+def build_advisor_comparison(
+    run_dir: Path,
+    scenario_name: str,
+    metrics: list[str],
+    extra_roots: list[Path] | None = None,
+) -> AdvisorComparison | None:
+    """Build the current run's advisor-diagnosis delta vs. its same-scenario predecessor.
+
+    Returns None when there is no predecessor, the predecessor's data cannot be
+    read, or there is nothing worth surfacing (no metric changed status). Never
+    raises — any failure degrades to None so the report detail page is
+    unaffected (mirrors run_advisor()'s defensive error handling).
+    """
+    try:
+        metadata = run_reader._read_json(run_dir / "metadata.json")
+        current_run_id = str(metadata.get("run_id") or run_dir.name)
+        current_finished_at = str(metadata.get("finished_at") or "")
+        current_judge_model = str(metadata.get("judge_model", ""))
+
+        previous = find_previous_run(
+            scenario_name, current_run_id, current_finished_at, extra_roots
+        )
+        if previous is None:
+            return None
+
+        previous_dir = Path(previous.output_path)
+        previous_frame = run_reader.read_scores_frame(previous_dir)
+        if previous_frame.empty:
+            return None
+        previous_rows = _rows_as_records(previous_frame)
+        previous_metrics = previous.metrics
+
+        current_frame = run_reader.read_scores_frame(run_dir)
+        current_rows = _rows_as_records(current_frame)
+
+        previous_diagnoses = {d.metric: d for d in diagnose(previous_rows, previous_metrics)}
+        current_diagnoses = {d.metric: d for d in diagnose(current_rows, metrics)}
+
+        entries: list[AdvisorComparisonEntry] = []
+        for metric in sorted(set(previous_diagnoses) | set(current_diagnoses)):
+            prev_d = previous_diagnoses.get(metric)
+            curr_d = current_diagnoses.get(metric)
+
+            if prev_d is not None and curr_d is None:
+                if metric not in metrics:
+                    # Dropped from the current scope entirely — cannot fairly
+                    # claim "resolved" without a current-side measurement.
+                    continue
+                status = "resolved"
+            elif prev_d is None and curr_d is not None:
+                status = "regressed" if metric in previous_metrics else "new_metric"
+            else:
+                status = "still_triggered"
+
+            entries.append(AdvisorComparisonEntry(
+                metric=metric,
+                status=status,
+                previous_score=prev_d.mean_score if prev_d else None,
+                previous_severity=prev_d.severity if prev_d else None,
+                current_score=curr_d.mean_score if curr_d else None,
+                current_severity=curr_d.severity if curr_d else None,
+            ))
+
+        if not entries:
+            return None
+
+        entries.sort(key=lambda e: (_STATUS_ORDER.get(e.status, 99), e.metric))
+
+        previous_judge_model = previous.judge_model
+        return AdvisorComparison(
+            previous_run_id=previous.run_id,
+            previous_finished_at=previous.finished_at,
+            previous_judge_model=previous_judge_model,
+            current_judge_model=current_judge_model,
+            judge_model_changed=(
+                bool(previous_judge_model)
+                and bool(current_judge_model)
+                and previous_judge_model != current_judge_model
+            ),
+            entries=entries,
+        )
+    except Exception as exc:  # noqa: BLE001
+        logger.warning(
+            "[advisor_comparison] failed to build comparison for run_dir=%s: %s",
+            run_dir, exc,
+        )
+        return None

webapp/services/inline_scorer.py

@@ -13,36 +13,23 @@ import threading
 from typing import Any
 
 from rag_eval.compat import ensure_ragas_import_compat
-from rag_eval.metrics.factory import build_models
+from rag_eval.metrics.factory import build_metric_registry, build_models
+from rag_eval.metrics.judge_prompts import localize_pipeline_prompts
 from rag_eval.metrics.pipeline import MetricPipeline
 from rag_eval.settings import EvaluationSettings
 from rag_eval.shared.models import NormalizedSample
 
 ensure_ragas_import_compat()
 
-from ragas.metrics.collections import (  # noqa: E402
-    AnswerRelevancy,
-    ContextPrecision,
-    ContextRecall,
-    FactualCorrectness,
-    Faithfulness,
-    NoiseSensitivity,
-    SemanticSimilarity,
-)
 
-
+def _build_metric_instances(
-def _build_metric_instances(metrics: list[str], llm: Any, embeddings: Any) -> dict[str, Any]:
+    metrics: list[str], llm: Any, embeddings: Any, judge_language: str = "en"
-    """Instantiate only the RAGAS metric objects requested."""
+) -> dict[str, Any]:
-    registry: dict[str, Any] = {
+    """Instantiate only the RAGAS metric objects requested, localized if needed."""
-        "faithfulness": Faithfulness(llm=llm),
+    registry = build_metric_registry(llm, embeddings)
-        "answer_relevancy": AnswerRelevancy(llm=llm, embeddings=embeddings),
+    selected = {name: registry[name] for name in metrics if name in registry}
-        "context_recall": ContextRecall(llm=llm),
+    localize_pipeline_prompts(selected, judge_language)
-        "context_precision": ContextPrecision(llm=llm),
+    return selected
-        "noise_sensitivity": NoiseSensitivity(llm=llm),
-        "factual_correctness": FactualCorrectness(llm=llm),
-        "semantic_similarity": SemanticSimilarity(embeddings=embeddings),
-    }
-    return {name: registry[name] for name in metrics if name in registry}
 
 
 class InlineScorer:
@@ -87,10 +74,11 @@ class InlineScorer:
         judge_model: str,
         embedding_model: str,
         settings: EvaluationSettings,
+        judge_language: str = "en",
     ) -> dict[str, float | None]:
         """Score one sample synchronously and return {metric_name: score | None}."""
         llm, embeddings = self._get_models(judge_model, embedding_model, settings)
-        metric_instances = _build_metric_instances(metrics, llm, embeddings)
+        metric_instances = _build_metric_instances(metrics, llm, embeddings, judge_language)
 
         pipeline = MetricPipeline(
             metrics=metric_instances,

webapp/services/question_history.py

@@ -0,0 +1,102 @@
+"""Build a cross-run index of per-question RAGAS scores for historical comparison.
+
+The report detail page surfaces, for each low-scoring sample, how the same
+question scored in previous evaluations. Matching is by normalized question text
+(case-insensitive, whitespace-collapsed) across all discovered run directories,
+so a question evaluated in any earlier run shows up as history.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import pandas as pd
+
+from webapp.services import run_reader
+from webapp.services.run_reader import NON_METRIC_COLUMNS, _read_json
+
+
+def normalize_question(question: Any) -> str:
+    """Return a stable match key for a question (case/whitespace-insensitive)."""
+    return " ".join(str(question or "").split()).lower()
+
+
+def _row_metrics(row: dict[str, Any]) -> dict[str, float | None]:
+    """Extract numeric metric scores from a single scores.csv row."""
+    metrics: dict[str, float | None] = {}
+    for key, value in row.items():
+        if key in NON_METRIC_COLUMNS:
+            continue
+        try:
+            num = float(value)
+        except (TypeError, ValueError):
+            continue
+        if pd.isna(num):
+            continue
+        metrics[str(key)] = round(num, 4)
+    return metrics
+
+
+def build_question_history_index(
+    exclude_run_id: str | None = None,
+    extra_roots: list[Path] | None = None,
+) -> dict[str, list[dict[str, Any]]]:
+    """Scan all run dirs and group per-question score entries for history lookup.
+
+    Args:
+        exclude_run_id: Run whose rows are skipped, so "history" means *other*
+            evaluations (typically the run currently being viewed).
+        extra_roots: Additional output roots to scan (beyond the defaults).
+
+    Returns:
+        Map of normalized_question -> list of entries, each shaped as
+        ``{run_id, scenario_name, finished_at, metrics: {metric: value}}`` and
+        sorted by finished_at descending (most recent first). Within a single
+        run, only the last occurrence of a question is kept.
+    """
+    # question_key -> run_id -> entry (last write wins within the same run)
+    grouped: dict[str, dict[str, dict[str, Any]]] = {}
+
+    for run_dir in run_reader.discover_run_dirs(extra_roots):
+        metadata = _read_json(run_dir / "metadata.json")
+        run_id = str(metadata.get("run_id") or run_dir.name)
+        if exclude_run_id and run_id == exclude_run_id:
+            continue
+        scenario_name = str(metadata.get("scenario_name", ""))
+        finished_at = str(metadata.get("finished_at") or metadata.get("started_at") or "")
+
+        frame = run_reader.read_scores_frame(run_dir)
+        if frame.empty or "question" not in frame.columns:
+            continue
+
+        for record in frame.where(pd.notnull(frame), None).to_dict("records"):
+            key = normalize_question(record.get("question"))
+            if not key:
+                continue
+            metrics = _row_metrics(record)
+            if not metrics:
+                continue
+            grouped.setdefault(key, {})[run_id] = {
+                "run_id": run_id,
+                "scenario_name": scenario_name,
+                "finished_at": finished_at,
+                "metrics": metrics,
+            }
+
+    index: dict[str, list[dict[str, Any]]] = {}
+    for key, per_run in grouped.items():
+        entries = list(per_run.values())
+        entries.sort(key=lambda entry: entry["finished_at"], reverse=True)
+        index[key] = entries
+    return index
+
+
+def lookup(
+    index: dict[str, list[dict[str, Any]]],
+    question: Any,
+    limit: int = 5,
+) -> list[dict[str, Any]]:
+    """Return up to ``limit`` historical entries for a question (newest first)."""
+    entries = index.get(normalize_question(question), [])
+    return entries[: max(0, limit)]

webapp/services/report_builder.py

@@ -23,9 +23,10 @@ from webapp.models import (
     DistributionBin,
     GroupStat,
     ReportData,
+    SampleHistoryEntry,
     SampleScore,
 )
-from webapp.services import run_reader
+from webapp.services import advisor_comparison, question_history, run_reader
 
 
 # Number of equal-width buckets used for metric score histograms.
@@ -37,6 +38,9 @@ GROUPING_FIELDS = ("difficulty", "question_type", "language")
 # How many lowest-scoring samples to surface for manual review.
 LOWEST_SAMPLE_COUNT = 10
 
+# How many past evaluations of the same question to show in the history table.
+HISTORY_LIMIT = 5
+
 # Metrics whose lower raw value means stronger performance.
 LOWER_IS_BETTER_METRICS = {"noise_sensitivity"}
 
@@ -124,8 +128,16 @@ def _cell_text(row: pd.Series, column: str) -> str:
     return str(row[column]).strip()
 
 
-def _lowest_samples(frame: pd.DataFrame, metrics: list[str]) -> list[SampleScore]:
+def _lowest_samples(
-    """Select and shape the lowest-scoring samples for the review table."""
+    frame: pd.DataFrame,
+    metrics: list[str],
+    history_index: dict[str, list[dict]] | None = None,
+) -> list[SampleScore]:
+    """Select and shape the lowest-scoring samples for the review table.
+
+    When a history_index is supplied, each surfaced sample is annotated with the
+    same question's scores from previous runs (newest first) for comparison.
+    """
     if frame.empty:
         return []
 
@@ -154,7 +166,19 @@ def _lowest_samples(frame: pd.DataFrame, metrics: list[str]) -> list[SampleScore
         enriched.append((sort_key, sample))
 
     enriched.sort(key=lambda item: item[0])
-    return [sample for _, sample in enriched[:LOWEST_SAMPLE_COUNT]]
+    selected = [sample for _, sample in enriched[:LOWEST_SAMPLE_COUNT]]
+
+    # Attach per-question history only for the surfaced samples (keeps lookups cheap).
+    if history_index is not None:
+        for sample in selected:
+            if not sample.question:
+                continue
+            entries = question_history.lookup(
+                history_index, sample.question, limit=HISTORY_LIMIT
+            )
+            sample.history = [SampleHistoryEntry(**entry) for entry in entries]
+
+    return selected
 
 
 def build_report(run_dir: Path, metrics: list[str]) -> ReportData:
@@ -163,6 +187,9 @@ def build_report(run_dir: Path, metrics: list[str]) -> ReportData:
     summary_markdown = run_reader.read_summary_markdown(run_dir)
     advice_markdown = run_reader.read_advice_markdown(run_dir)
     metric_weights, doc_weights = _read_weights_from_snapshot(run_dir)
+    # Read once up front so both the empty-frame and full branches can surface it.
+    metadata = run_reader._read_json(run_dir / "metadata.json")
+    token_usage = metadata.get("token_usage") or {}
 
     if frame.empty or not metrics:
         return ReportData(
@@ -172,6 +199,8 @@ def build_report(run_dir: Path, metrics: list[str]) -> ReportData:
             advice_markdown=advice_markdown,
             metric_weights=metric_weights,
             doc_weights=doc_weights,
+            token_usage=token_usage,
+            advisor_comparison=None,
         )
 
     score_rows_list = frame.to_dict(orient="records")
@@ -192,15 +221,31 @@ def build_report(run_dir: Path, metrics: list[str]) -> ReportData:
         if metric in frame.columns
     }
 
+    # Cross-run history: scores of the same question in *other* runs (Approach A —
+    # on-demand global scan, excluding the run currently being viewed).
+    current_run_id = str(metadata.get("run_id") or run_dir.name)
+    history_index = question_history.build_question_history_index(
+        exclude_run_id=current_run_id
+    )
+
+    scenario_name = str(metadata.get("scenario_name") or "")
+    comparison = (
+        advisor_comparison.build_advisor_comparison(run_dir, scenario_name, metrics)
+        if scenario_name
+        else None
+    )
+
     return ReportData(
         metrics=metrics,
         metric_means=rounded_means,
         distributions=distributions,
         groupings=_groupings(frame, metrics),
-        lowest_samples=_lowest_samples(frame, metrics),
+        lowest_samples=_lowest_samples(frame, metrics, history_index),
         summary_markdown=summary_markdown,
         advice_markdown=advice_markdown,
         weighted_score_mean=_round_or_none(overall_ws),
         metric_weights=metric_weights,
         doc_weights=doc_weights,
+        token_usage=token_usage,
+        advisor_comparison=comparison,
     )

webapp/services/score_job_manager.py

@@ -73,6 +73,7 @@ class ScoreJobManager:
                 "metrics": list(request.metrics),
                 "judge_model": request.judge_model or "",
                 "embedding_model": request.embedding_model or "",
+                "judge_language": request.judge_language or "",
                 "has_contexts": bool(request.contexts),
                 "has_ground_truth": bool(request.ground_truth),
             },
@@ -107,7 +108,7 @@ class ScoreJobManager:
 
         # Lazy imports to keep web server bootable if ragas is not installed.
         from rag_eval.advisor import run_advisor
-        from rag_eval.metrics.factory import build_models
+        from rag_eval.metrics.token_tracker import track_token_usage
         from rag_eval.metrics.weights import compute_weighted_score
         from rag_eval.reporting.writers import write_run_artifacts
         from rag_eval.settings import EvaluationSettings
@@ -121,6 +122,7 @@ class ScoreJobManager:
         settings = EvaluationSettings()
         judge_model = request.judge_model or settings.ragas_judge_model
         embedding_model = request.embedding_model or settings.ragas_embedding_model
+        judge_language = request.judge_language or settings.ragas_judge_language
         effective = request.effective_metrics()
         requested = set(request.metrics)
         skipped = sorted(requested - set(effective))
@@ -129,6 +131,7 @@ class ScoreJobManager:
         started_at = utc_now_iso()
 
         try:
+            with track_token_usage() as usage_tracker:
                 if effective:
                     raw_scores = inline_scorer.score(
                         question=request.question,
@@ -139,6 +142,7 @@ class ScoreJobManager:
                         judge_model=judge_model,
                         embedding_model=embedding_model,
                         settings=settings,
+                        judge_language=judge_language,
                     )
                 else:
                     raw_scores = {}
@@ -199,6 +203,7 @@ class ScoreJobManager:
                 valid_samples=[sample],
                 invalid_samples=[],
                 score_rows=[score_row],
+                token_usage=usage_tracker.as_dict(),
             )
 
             write_run_artifacts(result)
@@ -206,8 +211,7 @@ class ScoreJobManager:
 
             # Run optimization advisor (builds optimization_advice.md)
             try:
-                llm, _ = build_models(judge_model, embedding_model, settings)
+                run_advisor(result, scenario, settings=settings)
-                run_advisor(result, scenario, llm)
                 logger.info("[score_job] advisor done  job_id=%s", job_id)
             except Exception as adv_exc:  # noqa: BLE001
                 logger.warning("[score_job] advisor failed  job_id=%s  err=%s", job_id, adv_exc)

webapp/services/session_score_manager.py

@@ -113,6 +113,7 @@ class SessionScoreJobManager:
                 "metrics": list(request.metrics),
                 "judge_model": request.judge_model or "",
                 "embedding_model": request.embedding_model or "",
+                "judge_language": request.judge_language or "",
                 "has_contexts": bool(request.contexts),
                 "has_ground_truth": bool(request.ground_truth),
                 "session_id": session_id,
@@ -192,7 +193,7 @@ class SessionScoreJobManager:
 
         # Lazy imports — keep web server bootable if ragas is not installed.
         from rag_eval.advisor import run_advisor
-        from rag_eval.metrics.factory import build_models
+        from rag_eval.metrics.token_tracker import track_token_usage
         from rag_eval.metrics.weights import compute_weighted_score
         from rag_eval.reporting.writers import write_run_artifacts
         from rag_eval.settings import EvaluationSettings
@@ -206,6 +207,7 @@ class SessionScoreJobManager:
         settings = EvaluationSettings()
         judge_model = request.judge_model or settings.ragas_judge_model
         embedding_model = request.embedding_model or settings.ragas_embedding_model
+        judge_language = request.judge_language or settings.ragas_judge_language
         effective = request.effective_metrics()
         requested = set(request.metrics)
         skipped = sorted(requested - set(effective))
@@ -214,6 +216,7 @@ class SessionScoreJobManager:
 
         try:
             # --- Scoring (can run concurrently for the same session) ----------
+            with track_token_usage() as usage_tracker:
                 if effective:
                     raw_scores = inline_scorer.score(
                         question=request.question,
@@ -224,6 +227,7 @@ class SessionScoreJobManager:
                         judge_model=judge_model,
                         embedding_model=embedding_model,
                         settings=settings,
+                        judge_language=judge_language,
                     )
                 else:
                     raw_scores = {}
@@ -248,6 +252,14 @@ class SessionScoreJobManager:
                 run_dir = self._output_dir / run_id
                 run_dir.mkdir(parents=True, exist_ok=True)
 
+                # Merge this call's token usage into the session's running total, so
+                # repeated calls accumulate instead of overwriting (mirrors the
+                # scores.csv append-only accumulation below).
+                existing_metadata = self._read_metadata(run_dir)
+                merged_token_usage = usage_tracker.merge_into(
+                    existing_metadata.get("token_usage", {})
+                )
+
                 # Read all existing rows, then append the new one
                 existing_rows = self._read_score_rows(run_dir)
                 call_number = len(existing_rows) + 1
@@ -310,6 +322,7 @@ class SessionScoreJobManager:
                     valid_samples=valid_samples,
                     invalid_samples=[],
                     score_rows=all_rows,
+                    token_usage=merged_token_usage,
                 )
 
                 write_run_artifacts(result)
@@ -320,8 +333,7 @@ class SessionScoreJobManager:
 
                 # Regenerate optimization advice over all accumulated rows
                 try:
-                    llm, _ = build_models(judge_model, embedding_model, settings)
+                    run_advisor(result, scenario, settings=settings)
-                    run_advisor(result, scenario, llm)
                     logger.info("[session_job] advisor done  job_id=%s  session=%s", job_id, session_id)
                 except Exception as adv_exc:  # noqa: BLE001
                     logger.warning(
@@ -375,6 +387,16 @@ class SessionScoreJobManager:
         except (OSError, ValueError):
             return []
 
+    def _read_metadata(self, run_dir: Path) -> dict[str, Any]:
+        """Read this session's existing metadata.json, returning {} if absent/unreadable."""
+        metadata_path = run_dir / "metadata.json"
+        if not metadata_path.is_file():
+            return {}
+        try:
+            return json.loads(metadata_path.read_text(encoding="utf-8"))
+        except (OSError, ValueError):
+            return {}
+
     def _read_metric_means(self, run_dir: Path) -> dict[str, float | None]:
         """Compute per-metric means from the session's scores.csv."""
         scores_path = run_dir / "scores.csv"

webapp/static/css/app.css

@@ -253,6 +253,41 @@ table.group-table td { border-bottom: 1px solid #f1f5f9; font-variant-numeric: t
 }
 .detail-gt { color: var(--good); }
 
+/* 历史评分小表格：本次行高亮 + 涨跌着色（绿=改善 红=退步） */
+table.history-table { width: 100%; border-collapse: collapse; font-size: 12px; margin-top: 4px; }
+table.history-table th, table.history-table td {
+  padding: 5px 8px; text-align: left; border-bottom: 1px solid #f1f5f9;
+}
+table.history-table th { color: var(--slate); font-weight: 600; border-bottom: 1px solid var(--line); }
+table.history-table td { font-variant-numeric: tabular-nums; }
+.history-table tr.hist-current { background: #f0f9ff; }
+.history-table tr.hist-current .hist-label { font-weight: 700; color: #0369a1; }
+.hist-when { white-space: nowrap; }
+.hist-label { display: inline-block; }
+.hist-sub { display: block; font-size: 11px; color: var(--slate-light); }
+.hist-delta { font-size: 11px; font-weight: 700; font-variant-numeric: tabular-nums; }
+.hist-delta.delta-good { color: #16a34a; }
+.hist-delta.delta-bad { color: #dc2626; }
+.hist-delta.delta-flat { color: var(--slate-light); }
+
+.advisor-comparison-panel { border-left: 3px solid #0ea5e9; }
+.advisor-comparison-header {
+  display: flex; align-items: baseline; justify-content: space-between; gap: 10px;
+  margin-bottom: 10px;
+}
+.advisor-comparison-row {
+  display: flex; align-items: center; gap: 10px;
+  padding: 6px 0; border-bottom: 1px solid var(--line);
+  font-size: 13px;
+}
+.advisor-comparison-row:last-of-type { border-bottom: none; }
+.advisor-comparison-status { font-weight: 600; white-space: nowrap; }
+.advisor-comparison-metric { flex: 1; color: var(--slate); }
+.advisor-comparison-scores { font-variant-numeric: tabular-nums; white-space: nowrap; }
+.advisor-comparison-status.delta-good { color: #16a34a; }
+.advisor-comparison-status.delta-bad { color: #dc2626; }
+.advisor-comparison-status.delta-flat { color: var(--slate); }
+
 .empty { text-align: center; padding: 60px 20px; color: var(--slate); }
 .empty p { margin-bottom: 8px; }
 
@@ -514,6 +549,14 @@ table.group-table td { border-bottom: 1px solid #f1f5f9; font-variant-numeric: t
   table.group-table td { padding: 4pt 6pt; border-bottom: 1px solid #e2e8f0; }
   table.group-table th { font-weight: 700; color: #64748b; }
 
+  /* ── 历史评分表 ── */
+  table.history-table { width: 100%; font-size: 9pt; border-collapse: collapse; }
+  table.history-table th,
+  table.history-table td { padding: 3pt 6pt; border-bottom: 1px solid #e2e8f0; }
+  .history-table tr.hist-current { background: #f0f9ff !important; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
+  .hist-delta.delta-good { color: #16a34a !important; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
+  .hist-delta.delta-bad  { color: #dc2626 !important; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
+
   /* ── 颜色保留（部分浏览器打印默认去色） ── */
   .good  { color: #16a34a !important; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
   .warn  { color: #eab308 !important; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
@@ -546,3 +589,67 @@ table.group-table td { border-bottom: 1px solid #f1f5f9; font-variant-numeric: t
 .advice-md ul { padding-left: 20px; margin: 6px 0; }
 .advice-md li { margin: 3px 0; font-size: 13px; }
 .advice-md strong { color: var(--ink); font-weight: 600; }
+
+/* ---------- 指标看板 Dashboard ---------- */
+.dashboard-charts {
+  display: flex;
+  flex-direction: column;
+  gap: 20px;
+}
+.dashboard-chart-panel {
+  min-width: 0;
+  /* 上下布局时给图表面板稍微更宽裕的高度 */
+}
+.dashboard-chart-panel canvas {
+  max-height: 340px;
+  height: 320px !important;
+}
+
+/* 运行选择器列表 */
+.db-run-list {
+  display: flex;
+  flex-direction: column;
+  gap: 5px;
+  max-height: 240px;
+  overflow-y: auto;
+  margin-top: 10px;
+  padding-right: 4px;
+}
+.db-run-row {
+  display: flex;
+  align-items: center;
+  gap: 10px;
+  padding: 8px 12px;
+  border: 1px solid var(--line);
+  border-radius: 8px;
+  cursor: pointer;
+  transition: background 0.12s, border-color 0.12s;
+  background: var(--surface);
+}
+.db-run-row:hover { background: #f0fbfb; border-color: var(--petrol); }
+.db-run-row:has(input:checked) {
+  background: #e8f7f7;
+  border-color: #7ecece;
+}
+.db-run-row input[type="checkbox"] { flex-shrink: 0; accent-color: var(--petrol); width: 15px; height: 15px; }
+.db-run-label { display: flex; flex-direction: column; gap: 2px; flex: 1; min-width: 0; }
+.db-run-name { font-size: 13px; font-weight: 600; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
+.db-run-chips { display: flex; flex-wrap: wrap; gap: 6px; flex-shrink: 0; }
+.db-chip-name { color: var(--slate); }
+.btn-sm { padding: 5px 12px; font-size: 12px; }
+
+/* 看板图表面板头：标题左 + 下拉右 对齐优化 */
+.db-panel-head-bar {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  flex-wrap: wrap;
+  gap: 8px;
+  margin-bottom: 14px;
+}
+.db-chart-hint {
+  font-size: 11px;
+  color: var(--slate-light);
+  margin-top: 6px;
+  text-align: center;
+}

webapp/static/index.html

@@ -31,6 +31,9 @@
         <button class="nav-item" data-view="scorejobs">
           <span class="nav-ico">📋</span><span>评分记录</span>
         </button>
+        <button class="nav-item" data-view="dashboard">
+          <span class="nav-ico">📊</span><span>指标看板</span>
+        </button>
         <button class="nav-item" data-view="apidocs">
           <span class="nav-ico">⎔</span><span>API 文档</span>
         </button>
@@ -171,6 +174,17 @@
           <div class="section-label">④ 最低分样本（点击展开逐条复核）</div>
           <div class="lowest-table" id="lowest-table"></div>
 
+          <!-- 相比上次运行（同 scenario_name 的顾问诊断对比，自动匹配，找不到则不显示） -->
+          <div id="advisor-comparison-section" hidden>
+            <div class="panel advisor-comparison-panel">
+              <div class="advisor-comparison-header">
+                <span class="section-label tight">相比上次运行</span>
+                <span class="muted tiny" id="advisor-comparison-meta"></span>
+              </div>
+              <div id="advisor-comparison-body"></div>
+            </div>
+          </div>
+
           <!-- ⑤ 优化建议（optimization_advisor: true 时显示） -->
           <div id="advice-section" hidden>
             <div class="section-label">⑤ 优化建议 OPTIMIZATION ADVICE</div>
@@ -182,6 +196,10 @@
               <div class="advice-body" id="advice-body"></div>
             </div>
           </div>
+
+          <!-- ⑥ Token 用量（按模型分组，不换算金额） -->
+          <div class="section-label">⑥ Token 用量</div>
+          <div class="panel" id="token-usage-wrap"></div>
         </div>
       </section>
 
@@ -263,6 +281,11 @@
           allowfullscreen>
         </iframe>
       </section>
+
+      <!-- 指标看板视图 -->
+      <section class="view" id="view-dashboard" hidden>
+        <div id="dashboard-wrap"></div>
+      </section>
     </main>
   </div>
 
@@ -272,6 +295,7 @@
   <script src="/static/js/profiles.js"></script>
   <script src="/static/js/runner.js"></script>
   <script src="/static/js/score_jobs.js"></script>
+  <script src="/static/js/dashboard.js"></script>
   <script src="/static/js/app.js"></script>
 </body>
 </html>

webapp/static/js/app.js

@@ -5,8 +5,8 @@
 const App = {
   currentRunId: null,
   activeView: null,
-  views: ["runs", "new", "report", "profiles", "scorejobs", "apidocs"],
+  views: ["runs", "new", "report", "profiles", "scorejobs", "dashboard", "apidocs"],
-  titles: { runs: "运行列表", new: "新建评估", report: "报告详情", profiles: "LLM 配置", scorejobs: "评分记录", apidocs: "API 文档" },
+  titles: { runs: "运行列表", new: "新建评估", report: "报告详情", profiles: "LLM 配置", scorejobs: "评分记录", dashboard: "指标看板", apidocs: "API 文档" },
 
   // 初始化：绑定导航、从 URL/sessionStorage 恢复上次位置、启动健康检查。
   init() {
@@ -73,6 +73,7 @@ const App = {
     if (view === "report")     Report.render(App.currentRunId);
     if (view === "profiles")   Profiles.load();
     if (view === "scorejobs")  ScoreJobs.load();
+    if (view === "dashboard")  Dashboard.load();
   },
 
   // ----------------------------------------------------------------

webapp/static/js/dashboard.js

@@ -0,0 +1,418 @@
+// dashboard.js — 指标看板：运行选择器 + 折线图（指标趋势） + 柱状图（vs 达标阈值）。
+// 纯前端，数据来自 GET /api/runs，复用 MetricPresenter 的方向语义与阈值。
+
+(function attachDashboard(globalObj) {
+const Dashboard = {
+  _runs: [],          // 全量 runs（已按 finished_at 倒序来自 API）
+  _selected: new Set(), // 当前勾选的 run_id 集合
+  _focusId: null,     // 柱状图聚焦的 run_id
+  _trendChart: null,
+  _barChart: null,
+
+  // ── 入口 ─────────────────────────────────────────────────────────────────
+
+  async load() {
+    const wrap = document.getElementById("dashboard-wrap");
+    if (!wrap) return;
+    wrap.innerHTML = '<p class="muted">加载中…</p>';
+
+    try {
+      const data = await API.runs();
+      Dashboard._runs = (data.runs || []).slice().sort(
+        (a, b) => (a.finished_at || "").localeCompare(b.finished_at || "")
+      );
+      Dashboard._selected = new Set(Dashboard._runs.map((r) => r.run_id));
+      Dashboard._focusId = Dashboard._runs.length
+        ? Dashboard._runs[Dashboard._runs.length - 1].run_id
+        : null;
+      Dashboard._render(wrap);
+    } catch (err) {
+      wrap.innerHTML = `<p class="muted">加载失败：${App.escape(err.message)}</p>`;
+    }
+  },
+
+  // ── 渲染 ─────────────────────────────────────────────────────────────────
+
+  _render(wrap) {
+    if (!Dashboard._runs.length) {
+      wrap.innerHTML = `
+        <div class="empty">
+          <p>暂无评测运行数据。</p>
+          <p class="muted">触发一次评测或通过 Dify 工具调用后，数据将在此显示。</p>
+        </div>`;
+      return;
+    }
+
+    wrap.innerHTML = "";
+
+    // 运行选择器面板
+    wrap.appendChild(Dashboard._buildSelector());
+
+    // 图表区
+    const chartRow = document.createElement("div");
+    chartRow.className = "dashboard-charts";
+    chartRow.innerHTML = `
+      <div class="panel dashboard-chart-panel">
+        <div class="db-panel-head-bar">
+          <div>
+            <div class="section-label tight">📈 指标趋势折线图</div>
+            <div class="muted" style="font-size:12px;margin-top:2px">按时间顺序展示所选运行的各指标均值变化</div>
+          </div>
+        </div>
+        <canvas id="db-trend-chart"></canvas>
+        <p class="db-chart-hint" id="db-trend-hint"></p>
+      </div>
+      <div class="panel dashboard-chart-panel">
+        <div class="db-panel-head-bar">
+          <div>
+            <div class="section-label tight">📊 指标达标对比柱状图</div>
+            <div class="muted" style="font-size:12px;margin-top:2px">实际均值 vs 达标阈值（深绿柱）</div>
+          </div>
+          <select class="select" id="db-focus-select" style="min-width:220px"></select>
+        </div>
+        <canvas id="db-bar-chart"></canvas>
+        <p class="db-chart-hint">达标阈值：higher-better 指标 0.85 · noise_sensitivity 0.15</p>
+      </div>
+    `;
+    wrap.appendChild(chartRow);
+
+    Dashboard._populateFocusSelect();
+    Dashboard._drawTrend();
+    Dashboard._drawBar();
+  },
+
+  // 运行选择器
+  _buildSelector() {
+    const panel = document.createElement("div");
+    panel.className = "panel";
+    panel.innerHTML = `
+      <div class="panel-head">
+        <div>
+          <span class="section-label tight">运行选择（折线图数据源）</span>
+          <span class="muted" style="font-size:12px; margin-left:10px">勾选≥2个运行可看趋势</span>
+        </div>
+        <div style="display:flex;gap:8px">
+          <button class="btn btn-sm" id="db-sel-all">全选</button>
+          <button class="btn btn-sm" id="db-sel-none">清空</button>
+          <input class="form-input" id="db-filter-input" placeholder="按场景名过滤…" style="width:180px;padding:6px 10px" />
+        </div>
+      </div>
+      <div class="db-run-list" id="db-run-list"></div>
+    `;
+    setTimeout(() => {
+      Dashboard._renderRunList();
+      document.getElementById("db-sel-all").onclick = () => {
+        Dashboard._runs.forEach((r) => Dashboard._selected.add(r.run_id));
+        Dashboard._renderRunList();
+        Dashboard._drawTrend();
+      };
+      document.getElementById("db-sel-none").onclick = () => {
+        Dashboard._selected.clear();
+        Dashboard._renderRunList();
+        Dashboard._drawTrend();
+      };
+      document.getElementById("db-filter-input").oninput = (e) => {
+        Dashboard._renderRunList(e.target.value.toLowerCase());
+      };
+    });
+    return panel;
+  },
+
+  _renderRunList(filter) {
+    const list = document.getElementById("db-run-list");
+    if (!list) return;
+    list.innerHTML = "";
+    const visible = filter
+      ? Dashboard._runs.filter((r) =>
+          (r.scenario_name || r.run_id).toLowerCase().includes(filter)
+        )
+      : Dashboard._runs;
+    [...visible].reverse().forEach((run) => {
+      const row = document.createElement("label");
+      row.className = "db-run-row";
+      const chips = (run.metrics || [])
+        .slice(0, 4)
+        .map((m) => {
+          const v = run.metric_means ? run.metric_means[m] : null;
+          const cls = App.scoreClass(m, v);
+          const text = v === null || v === undefined ? "n/a" : Number(v).toFixed(2);
+          return `<span class="metric-chip"><span class="db-chip-name">${App.escape(App.shortMetric(m))}</span> <b class="${cls}">${text}</b></span>`;
+        })
+        .join("");
+      row.innerHTML = `
+        <input type="checkbox" class="db-run-cb" data-id="${App.escape(run.run_id)}"
+          ${Dashboard._selected.has(run.run_id) ? "checked" : ""} />
+        <span class="db-run-label">
+          <span class="db-run-name">${App.escape(run.scenario_name || run.run_id)}</span>
+          <span class="muted" style="font-size:11px">${App.escape(App.shortTime(run.finished_at))} · ${App.escape(run.judge_model || "")}</span>
+        </span>
+        <span class="db-run-chips">${chips}</span>
+      `;
+      row.querySelector(".db-run-cb").addEventListener("change", (e) => {
+        if (e.target.checked) Dashboard._selected.add(run.run_id);
+        else Dashboard._selected.delete(run.run_id);
+        Dashboard._drawTrend();
+      });
+      list.appendChild(row);
+    });
+  },
+
+  // 填充柱状图聚焦下拉
+  _populateFocusSelect() {
+    const sel = document.getElementById("db-focus-select");
+    if (!sel) return;
+    sel.innerHTML = "";
+    [...Dashboard._runs].reverse().forEach((run) => {
+      const opt = document.createElement("option");
+      opt.value = run.run_id;
+      opt.textContent = `${run.scenario_name || run.run_id}  ${App.shortTime(run.finished_at)}`;
+      if (run.run_id === Dashboard._focusId) opt.selected = true;
+      sel.appendChild(opt);
+    });
+    sel.onchange = () => {
+      Dashboard._focusId = sel.value;
+      Dashboard._drawBar();
+    };
+  },
+
+  // ── 折线图 ────────────────────────────────────────────────────────────────
+
+  _drawTrend() {
+    const canvas = document.getElementById("db-trend-chart");
+    const hint = document.getElementById("db-trend-hint");
+    if (!canvas) return;
+
+    const selected = Dashboard._runs.filter((r) => Dashboard._selected.has(r.run_id));
+    if (selected.length === 0) {
+      if (Dashboard._trendChart) { Dashboard._trendChart.destroy(); Dashboard._trendChart = null; }
+      if (hint) hint.textContent = "请在上方勾选至少 1 个运行。";
+      return;
+    }
+    if (hint) {
+      hint.textContent = selected.length === 1
+        ? "只有 1 个运行，折线退化为单点——勾选更多运行可看趋势。"
+        : `共 ${selected.length} 个运行 · 横轴按完成时间升序`;
+    }
+
+    const { labels, datasets } = Dashboard._buildTrendDatasets(selected);
+    // 精选对比度高、色盲友好的颜色组合
+    const colors = [
+      "#009999", // petrol brand
+      "#3b82f6", // blue
+      "#f97316", // orange
+      "#8b5cf6", // violet
+      "#ec4899", // pink
+      "#06b6d4", // cyan
+      "#f59e0b", // amber
+    ];
+
+    if (Dashboard._trendChart) Dashboard._trendChart.destroy();
+    Dashboard._trendChart = new Chart(canvas, {
+      type: "line",
+      data: {
+        labels,
+        datasets: [
+          // 达标参考线（0.85，虚线，不显示在图例前列）
+          {
+            label: "达标参考线 0.85",
+            data: Array(labels.length).fill(0.85),
+            borderColor: "#16a34a",
+            borderDash: [6, 4],
+            borderWidth: 1.5,
+            pointRadius: 0,
+            fill: false,
+            order: 99,
+          },
+          ...datasets.map((ds, i) => ({
+            label: ds.label,
+            data: ds.data,
+            borderColor: colors[i % colors.length],
+            backgroundColor: colors[i % colors.length] + "18",
+            borderWidth: 2.5,
+            pointRadius: 5,
+            pointHoverRadius: 7,
+            pointBackgroundColor: colors[i % colors.length],
+            pointBorderColor: "#fff",
+            pointBorderWidth: 2,
+            tension: 0.25,
+            spanGaps: false,
+          })),
+        ],
+      },
+      options: {
+        responsive: true,
+        maintainAspectRatio: false,
+        interaction: { mode: "index", intersect: false },
+        plugins: {
+          legend: {
+            position: "bottom",
+            labels: { font: { size: 12 }, boxWidth: 14, padding: 16, usePointStyle: true, pointStyleWidth: 12 },
+          },
+          tooltip: {
+            backgroundColor: "#1a2942",
+            titleColor: "#e2e8f0",
+            bodyColor: "#cbd5e1",
+            borderColor: "#334155",
+            borderWidth: 1,
+            padding: 10,
+            callbacks: {
+              label: (ctx) => {
+                if (ctx.raw === null) return `  ${ctx.dataset.label}: —`;
+                return `  ${ctx.dataset.label}: ${Number(ctx.raw).toFixed(3)}`;
+              },
+            },
+          },
+        },
+        scales: {
+          y: {
+            min: 0, max: 1,
+            ticks: { stepSize: 0.1, font: { size: 11 }, color: "#94a3b8" },
+            grid: { color: "#f1f5f9" },
+            border: { display: false },
+          },
+          x: {
+            ticks: { font: { size: 11 }, color: "#64748b", maxRotation: 30 },
+            grid: { display: false },
+            border: { display: false },
+          },
+        },
+      },
+    });
+  },
+
+  // 柱状图 ──────────────────────────────────────────────────────────────────
+
+  _drawBar() {
+    const canvas = document.getElementById("db-bar-chart");
+    if (!canvas) return;
+
+    const run = Dashboard._runs.find((r) => r.run_id === Dashboard._focusId);
+    if (!run) return;
+
+    const { labels, actual, thresholds, colors, targetMet } =
+      Dashboard._buildComparisonData(run);
+
+    if (Dashboard._barChart) Dashboard._barChart.destroy();
+    Dashboard._barChart = new Chart(canvas, {
+      type: "bar",
+      data: {
+        labels,
+        datasets: [
+          {
+            label: "实际分数",
+            data: actual,
+            backgroundColor: colors,
+            borderRadius: 5,
+            barPercentage: 0.6,
+            categoryPercentage: 0.75,
+          },
+          {
+            label: "达标阈值",
+            data: thresholds,
+            // 深绿实心柱（不透明，无边框线）
+            backgroundColor: "#15803d",
+            borderRadius: 5,
+            barPercentage: 0.6,
+            categoryPercentage: 0.75,
+          },
+        ],
+      },
+      options: {
+        responsive: true,
+        maintainAspectRatio: false,
+        interaction: { mode: "index", intersect: false },
+        plugins: {
+          legend: {
+            position: "bottom",
+            labels: { font: { size: 12 }, boxWidth: 14, padding: 16, usePointStyle: false },
+          },
+          tooltip: {
+            backgroundColor: "#1a2942",
+            titleColor: "#e2e8f0",
+            bodyColor: "#cbd5e1",
+            borderColor: "#334155",
+            borderWidth: 1,
+            padding: 10,
+            callbacks: {
+              afterBody: (ctx) => {
+                if (!ctx.length) return [];
+                const idx = ctx[0].dataIndex;
+                return [targetMet[idx] ? "✓ 达标" : "✗ 未达标"];
+              },
+            },
+          },
+        },
+        scales: {
+          y: {
+            min: 0, max: 1,
+            ticks: { stepSize: 0.1, font: { size: 11 }, color: "#94a3b8" },
+            grid: { color: "#f1f5f9" },
+            border: { display: false },
+          },
+          x: {
+            ticks: { font: { size: 11 }, color: "#64748b" },
+            grid: { display: false },
+            border: { display: false },
+          },
+        },
+      },
+    });
+  },
+
+  // ── 纯数据函数（便于测试）────────────────────────────────────────────────
+
+  /**
+   * 从选中的 runs（按时间升序）构建折线图数据集。
+   * 每条线 = 一个指标；X 轴 = 各运行的简短标签；无值处补 null（断线）。
+   * @param {Array} runs - 已按 finished_at 升序排序的 run 对象数组
+   * @returns {{ labels: string[], datasets: Array<{label:string, data:Array<number|null>}> }}
+   */
+  _buildTrendDatasets(runs) {
+    // 合并所有 runs 出现过的指标（保持首次出现顺序）
+    const metricSet = [];
+    runs.forEach((r) => {
+      (r.metrics || []).forEach((m) => {
+        if (!metricSet.includes(m)) metricSet.push(m);
+      });
+    });
+
+    const labels = runs.map(
+      (r) => `${r.scenario_name || r.run_id}\n${App.shortTime(r.finished_at)}`
+    );
+
+    const datasets = metricSet.map((m) => ({
+      label: m + (MetricPresenter.isLowerBetter(m) ? " (越低越好)" : ""),
+      data: runs.map((r) => {
+        const v = r.metric_means ? r.metric_means[m] : null;
+        return v !== null && v !== undefined ? Number(v) : null;
+      }),
+    }));
+
+    return { labels, datasets };
+  },
+
+  /**
+   * 从单个 run 构建柱状图对比数据。
+   * @param {Object} run
+   * @returns {{ labels, actual, thresholds, colors, targetMet }}
+   */
+  _buildComparisonData(run) {
+    const metrics = run.metrics || [];
+    const labels = metrics.map((m) => App.shortMetric(m));
+    const actual = metrics.map((m) => {
+      const v = run.metric_means ? run.metric_means[m] : null;
+      return v !== null && v !== undefined ? Number(v) : null;
+    });
+    const thresholds = metrics.map((m) => MetricPresenter.passThreshold(m));
+    const targetMet = metrics.map((m, i) =>
+      MetricPresenter.meetsTarget(m, actual[i])
+    );
+    const colorMap = { good: "#4ade80", warn: "#fbbf24", bad: "#f87171", na: "#cbd5e1" };
+    const colors = metrics.map((m, i) => colorMap[App.scoreClass(m, actual[i])] || "#cbd5e1");
+
+    return { labels, actual, thresholds, colors, targetMet };
+  },
+};
+
+  globalObj.Dashboard = Dashboard;
+})(typeof window !== "undefined" ? window : this);

webapp/static/js/metric_presenter.js

@@ -69,9 +69,42 @@
     return "#dc2626";
   }
 
+  // 计算某指标本次相对上一次的涨跌信息，方向语义随指标而定
+  // （noise_sensitivity 越低越好：下降=改善）。
+  function deltaInfo(metricName, current, previous) {
+    const isNum = (v) => v !== null && v !== undefined && !Number.isNaN(Number(v));
+    if (!isNum(current) || !isNum(previous)) {
+      return { hasData: false, delta: null, improved: null, arrow: "", magnitude: "", cls: "delta-flat" };
+    }
+    const delta = Number(current) - Number(previous);
+    const rounded = Math.round(delta * 10000) / 10000;
+    const arrow = rounded > 0 ? "▲" : rounded < 0 ? "▼" : "→";
+    const magnitude = Math.abs(rounded).toFixed(2);
+    const improved = isLowerBetter(metricName) ? rounded < 0 : rounded > 0;
+    const cls = rounded === 0 ? "delta-flat" : improved ? "delta-good" : "delta-bad";
+    return { hasData: true, delta: rounded, improved, arrow, magnitude, cls };
+  }
+
+  // 返回指标的"达标阈值"（柱状图对比用，方向感知）。
+  // higher-better 指标：0.85；lower-better (noise_sensitivity)：0.15。
+  function passThreshold(metricName) {
+    return isLowerBetter(metricName) ? 0.15 : 0.85;
+  }
+
+  // 判断某指标的值是否达标。
+  function meetsTarget(metricName, value) {
+    if (value === null || value === undefined || Number.isNaN(Number(value))) return false;
+    const v = Number(value);
+    return isLowerBetter(metricName) ? v <= passThreshold(metricName) : v >= passThreshold(metricName);
+  }
+
   globalObj.MetricPresenter = {
     scoreClass,
     describeMetric,
     binColor,
+    isLowerBetter,
+    deltaInfo,
+    passThreshold,
+    meetsTarget,
   };
 })(window);

webapp/static/js/report.js

@@ -31,7 +31,9 @@ const Report = {
       Report.renderDistribution(detail.report);
       Report.renderGroupings(detail.report);
       Report.renderLowest(detail.report);
+      Report.renderAdvisorComparison(detail.report);
       Report.renderAdvice(detail.summary, detail.report);
+      Report.renderTokenUsage(detail.report);
       content.style.opacity = "1";
 
       // 同步下拉选中项
@@ -254,6 +256,81 @@ const Report = {
     tableEl.innerHTML = `<table class="group-table">${head}${body}</table>`;
   },
 
+  // 渲染"Token 用量"面板：按模型分组的 input/output/调用次数表格。
+  renderTokenUsage(report) {
+    const wrap = document.getElementById("token-usage-wrap");
+    if (!wrap) return;
+    const usage = report.token_usage || {};
+    const models = Object.keys(usage).sort();
+    if (models.length === 0) {
+      wrap.innerHTML = '<p class="muted tiny">暂无 token 用量数据。</p>';
+      return;
+    }
+    let head = "<tr><th>模型</th><th>input_tokens</th><th>output_tokens</th><th>调用次数</th></tr>";
+    let body = "";
+    models.forEach((model) => {
+      const u = usage[model] || {};
+      body += `<tr><td>${App.escape(model)}</td><td>${u.input_tokens ?? 0}</td>` +
+        `<td>${u.output_tokens ?? 0}</td><td>${u.calls ?? 0}</td></tr>`;
+    });
+    wrap.innerHTML = `<table class="group-table">${head}${body}</table>`;
+  },
+
+  // 相比上次运行的顾问诊断对比（同 scenario_name 的最近一次前序运行，自动匹配）。
+  renderAdvisorComparison(report) {
+    const section = document.getElementById("advisor-comparison-section");
+    const meta = document.getElementById("advisor-comparison-meta");
+    const body = document.getElementById("advisor-comparison-body");
+    if (!section || !meta || !body) return;
+
+    const comparison = report.advisor_comparison;
+    if (!comparison || !comparison.entries || comparison.entries.length === 0) {
+      section.hidden = true;
+      return;
+    }
+
+    section.hidden = false;
+    meta.textContent = `对比：${comparison.previous_run_id}（${App.shortTime(comparison.previous_finished_at)}）`;
+
+    const STATUS_LABEL = {
+      resolved: "✅ 已改善",
+      regressed: "⚠️ 新触发",
+      still_triggered: "❌ 仍未解决",
+      new_metric: "🆕 新增指标",
+    };
+    const STATUS_CLASS = {
+      resolved: "delta-good",
+      regressed: "delta-bad",
+      still_triggered: "delta-bad",
+      new_metric: "delta-flat",
+    };
+
+    const fmt = (v) => (v === null || v === undefined ? "—" : Number(v).toFixed(2));
+
+    let rows = "";
+    comparison.entries.forEach((entry) => {
+      const label = STATUS_LABEL[entry.status] || entry.status;
+      const cls = STATUS_CLASS[entry.status] || "delta-flat";
+      const d = MetricPresenter.deltaInfo(entry.metric, entry.current_score, entry.previous_score);
+      const deltaHtml = d.hasData && d.delta !== 0
+        ? ` <span class="hist-delta ${d.cls}">${d.arrow}${d.magnitude}</span>`
+        : "";
+      rows += `
+        <div class="advisor-comparison-row">
+          <span class="advisor-comparison-status ${cls}">${label}</span>
+          <span class="advisor-comparison-metric">${App.escape(App.shortMetric(entry.metric))}</span>
+          <span class="advisor-comparison-scores">${fmt(entry.previous_score)} → ${fmt(entry.current_score)}${deltaHtml}</span>
+        </div>`;
+    });
+
+    let caveat = "";
+    if (comparison.judge_model_changed) {
+      caveat = `<p class="muted tiny">⚠️ judge_model 不同（${App.escape(comparison.previous_judge_model)} → ${App.escape(comparison.current_judge_model)}），对比仅供参考。</p>`;
+    }
+
+    body.innerHTML = rows + caveat;
+  },
+
   // ④ 最低分样本逐条复核表（点击展开）。
   renderLowest(report) {    const wrap = document.getElementById("lowest-table");
     const samples = report.lowest_samples || [];
@@ -283,7 +360,7 @@ const Report = {
       const detail = document.createElement("div");
       detail.className = "lowest-detail";
       detail.hidden = true;
-      detail.innerHTML = Report._detailHtml(sample);
+      detail.innerHTML = Report._detailHtml(sample, metrics);
 
       row.addEventListener("click", () => {
         detail.hidden = !detail.hidden;
@@ -293,8 +370,8 @@ const Report = {
     });
   },
 
-  // 单条样本的展开详情：question / contexts / answer / ground_truth。
+  // 单条样本的展开详情：question / contexts / answer / ground_truth / 历史评分。
-  _detailHtml(sample) {
+  _detailHtml(sample, metrics) {
     const contexts = (sample.contexts || [])
       .map((c, i) => `<div class="ctx-item">[${i + 1}] ${App.escape(c)}</div>`)
       .join("");
@@ -320,6 +397,63 @@ const Report = {
           <div class="detail-gt">${App.escape(sample.ground_truth || "—")}</div>
         </div>
         ${errorBlock}
+        ${Report._historyHtml(sample, metrics || [])}
+      </div>
+    `;
+  },
+
+  // 同一问题的历史评分小表格：本次 + 历次（按时间倒序），逐行标注较更早一次的涨跌。
+  _historyHtml(sample, metrics) {
+    const history = sample.history || [];
+    if (!history.length) return "";
+
+    // 只展示当前样本与历史中实际出现过的指标列，避免空列。
+    const cols = metrics.filter(
+      (m) =>
+        (sample.metrics && sample.metrics[m] !== undefined && sample.metrics[m] !== null) ||
+        history.some((h) => h.metrics && h.metrics[m] !== undefined && h.metrics[m] !== null),
+    );
+    if (!cols.length) return "";
+
+    // 组合行：[本次, 历次...]，相邻两行做涨跌对比（行 r 对比更早的行 r+1）。
+    const rows = [
+      { label: "本次", sub: "", metrics: sample.metrics || {}, current: true },
+      ...history.map((h) => ({
+        label: App.escape(h.scenario_name || h.run_id || "历史"),
+        sub: App.escape(App.shortTime(h.finished_at)),
+        metrics: h.metrics || {},
+        current: false,
+      })),
+    ];
+
+    let head = "<tr><th>评测</th>";
+    cols.forEach((m) => (head += `<th>${App.escape(App.shortMetric(m))}</th>`));
+    head += "</tr>";
+
+    let body = "";
+    rows.forEach((row, r) => {
+      const older = rows[r + 1];
+      body += `<tr class="${row.current ? "hist-current" : ""}">`;
+      body += `<td class="hist-when"><span class="hist-label">${row.label}</span>${row.sub ? `<span class="hist-sub">${row.sub}</span>` : ""}</td>`;
+      cols.forEach((m) => {
+        const v = row.metrics ? row.metrics[m] : null;
+        const cls = App.scoreClass(m, v);
+        const text = v === null || v === undefined ? "—" : Number(v).toFixed(2);
+        let deltaHtml = "";
+        const baseline = older && older.metrics ? older.metrics[m] : undefined;
+        const d = MetricPresenter.deltaInfo(m, v, baseline);
+        if (d.hasData && d.delta !== 0) {
+          deltaHtml = ` <span class="hist-delta ${d.cls}">${d.arrow}${d.magnitude}</span>`;
+        }
+        body += `<td><span class="score-badge ${cls}">${text}</span>${deltaHtml}</td>`;
+      });
+      body += "</tr>";
+    });
+
+    return `
+      <div class="detail-field">
+        <div class="detail-label">历史评分 history（同一问题，最近 ${history.length} 次，含本次对比）</div>
+        <table class="history-table">${head}${body}</table>
       </div>
     `;
   },