AIRegulation-DocAnalysis/docs/superpowers/specs/2026-07-02-status-llm-model-usage-design.md

# System Status — Connected AI Models & Token Usage Design

**Date:** 2026-07-02
**Scope:** Extend the existing System Status module with a new "AI Models" panel showing which LLM/Embedding/Reranker models are configured, their connection status, and cumulative token consumption.
**Relationship to existing roadmap:** This is a lightweight, self-contained first slice of the "P0-A observability" priority already identified in `AI_Agent_优化分析报告_2026-06-18.md` (full Langfuse tracing + Ragas evaluation remains a separate, larger future effort — see Out of Scope).

---

## Goals

1. Show all "connected" AI models in one place: main answer-generation LLM, the dedicated HyDE query-expansion LLM, the embedding model, and the reranker (even when disabled).
2. Show connection status per model, derived passively from real traffic (no extra cost), plus an optional manual "test connection" action for an on-demand active check.
3. Show cumulative token consumption per model since process start (in-memory; resets on restart — no new database table).
4. Guarantee accuracy by instrumenting the single shared LLM client factory, so intermediate Agentic RAG steps, HyDE, regulation-perception analysis, compliance review, and document summarization are all captured — not just the final chat answer.

## Non-Goals (see "Out of Scope" at the end)

- Persistent/historical token usage (DB-backed, survives restart) — deferred.
- Cost/spend estimation in currency — deferred (no reliable public pricing for the internal gateway).
- Per-session or per-user token breakdown — deferred.
- Accurate token counting for **streaming** chat responses — deferred (see Known Limitations).
- Full distributed tracing / LLM-as-judge faithfulness scoring (Langfuse + Ragas, `P0-A` in the existing roadmap) — this feature is a lightweight precursor, not a replacement.

---

## Architecture Overview

### Layering (must not be violated — per `docs/architecture/backend-project-architecture.md`)

```
api/routes/status.py        →  thin handlers, reads tracker + settings, no business logic
shared/model_usage_tracker.py →  cross-cutting support (same tier as shared/bootstrap.py)
services/llm/llm_factory.py  →  wraps clients with TrackedLLMClient at creation time
infrastructure/embedding/…   →  direct instrumentation (single implementation)
infrastructure/vectorstore/cross_encoder_reranker.py → direct instrumentation (single implementation)
```

No new business orchestration is added to `services/*` or `workflows/*`. The tracker is passive, cross-cutting infrastructure support, consistent with how `shared/bootstrap.py` and `shared/errors.py` are described in the backend README as "composition root 与横切支撑".

### Data Model

`ModelUsageTracker` keys its internal state by **`f"{provider}:{model}"`**, not by business role. This is more robust than keying by role: if a future Agentic sub-step uses a different provider/model, it is still captured under its own key rather than being silently dropped because no role mapping exists for it. "Role" (`main_llm` / `hyde_llm` / `embedding` / `reranker`) is purely a **presentation-layer label**, resolved at read time in the `/status/models` handler by looking up the current `settings` (`llm_provider`/`llm_model`, `hyde_llm_provider`/`hyde_llm_model` with its existing "empty means reuse main" fallback, `embedding_model`, `reranker_model`).

```python
# backend/app/shared/model_usage_tracker.py
@dataclass
class ModelUsageEntry:
    """Represent accumulated usage/connection state for one provider+model pair."""
    provider: str
    model: str
    total_tokens: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    call_count_ok: int = 0
    call_count_error: int = 0
    last_called_at: datetime | None = None
    last_latency_ms: int | None = None
    last_error: str | None = None

    @property
    def status(self) -> str:
        """Derive display status from call history: never_called | ok | error.

        Note: this only reflects the tracker's own history. The route handler
        (not this class) overrides the value to "disabled" for the reranker role
        when settings.reranker_enabled is False — config always wins over any
        stale historical data, e.g. if the reranker was enabled in the past and
        later turned off in .env.
        """
        if self.last_called_at is None:
            return "never_called"
        return "error" if self.last_error else "ok"


class ModelUsageTracker:
    """Thread-safe in-memory registry of per-model call/usage stats.

    Never raises: a bug here must not break a real user-facing LLM call.
    """

    def __init__(self) -> None:
        self._entries: dict[str, ModelUsageEntry] = {}
        self._lock = threading.Lock()

    def record(
        self,
        *,
        provider: str,
        model: str,
        success: bool,
        usage: dict | None = None,
        latency_ms: int | None = None,
        error: str | None = None,
    ) -> None:
        """Record the outcome of one call to provider/model. Safe to call from any thread."""
        ...

    def snapshot(self) -> dict[str, ModelUsageEntry]:
        """Return a shallow copy of all tracked entries, safe to iterate without the lock."""
        ...


@lru_cache
def get_model_usage_tracker() -> ModelUsageTracker:
    """Return the process-wide singleton tracker (mirrors get_settings()/get_llm_factory() pattern)."""
    return ModelUsageTracker()
```

All `record()` bodies are wrapped in `try/except Exception: logger.warning(...)` internally — tracking failures are logged and swallowed, never propagated.

### LLM Instrumentation — `TrackedLLMClient` Wrapper

Every LLM call in the codebase goes through `get_llm_client()` in `backend/app/services/llm/llm_factory.py` (confirmed call sites: `agentic_service.py`, `hyde_expander.py`, `perception/services.py`, `perception/llm_pipeline.py`, `api/routes/compliance.py` ×2, `infrastructure/llm/openai_compatible_answer_generator.py` ×2, `services/llm/document_summarizer.py`). `LLMFactory.create()` wraps the concrete client (`DeepSeekClient`/`QwenClient`/`QwenVLClient`) in `TrackedLLMClient` before caching it, so every current and future call site is covered automatically with **one** change point.

```python
# backend/app/services/llm/tracked_client.py
class TrackedLLMClient:
    """Transparent decorator that records usage/latency into ModelUsageTracker.

    Deliberately does NOT subclass BaseLLMClient: that ABC declares abstract
    methods (_init_client, get_available_models) which would have to be stubbed
    out, defeating the point of __getattr__ delegation and instantiation would
    fail with "Can't instantiate abstract class" before __getattr__ ever runs.
    Plain composition + __getattr__ forwarding is sufficient since callers only
    ever use duck-typed access (.chat(), .stream_chat(), .get_available_models(), .close()).
    """

    def __init__(self, inner: BaseLLMClient, tracker: ModelUsageTracker) -> None:
        self._inner = inner
        self._tracker = tracker

    def chat(self, messages, max_tokens=None, temperature=None, tools=None, **kwargs) -> LLMResponse:
        """Delegate to the wrapped client's chat(), then record usage/latency/outcome."""
        start = time.time()
        response = self._inner.chat(messages, max_tokens, temperature, tools, **kwargs)
        self._tracker.record(
            provider=self._inner.config.provider.value,
            model=response.model or self._inner.config.model,
            success=response.is_success,
            usage=response.usage,
            latency_ms=int((time.time() - start) * 1000),
            error=response.error,
        )
        return response

    def stream_chat(self, messages, *args, **kwargs):
        """Delegate to stream_chat(); records call success/latency only (no token usage — see Known Limitations)."""
        ...

    def __getattr__(self, name):
        """Forward any other attribute/method access to the wrapped client."""
        return getattr(self._inner, name)
```

### Embedding & Reranker Instrumentation

Both have a single concrete implementation today, so they are instrumented directly (no wrapper needed):

- `OpenAICompatibleEmbeddingProvider._request()` — additionally reads `data.get("usage", {})` from the OpenAI-compatible embeddings response and calls `get_model_usage_tracker().record(provider="embedding", model=self.model, ...)`.
- `OpenAICompatibleReranker._call_reranker()` / `rerank()` — records call success/failure + latency only. TEI/Cohere-style rerank responses do not include token usage, so `total_tokens` for the reranker role will always show as unavailable (`—`), which is factually correct, not a bug to fix later.

---

## API

Both endpoints are added to the existing `backend/app/api/routes/status.py` (no new router file), returning plain dicts — matching the existing convention in this file and in `perception.py` (no Pydantic response models for these "reporting" endpoints).

### `GET /status/models`

Passive read: no outbound network calls, just tracker snapshot + settings resolution.

```json
{
  "models": [
    {
      "role": "main_llm",
      "role_label": "主问答 LLM",
      "provider": "deepseek",
      "model": "deepseek-v4-flash",
      "enabled": true,
      "status": "ok",
      "total_tokens": 12345,
      "call_count_ok": 42,
      "call_count_error": 1,
      "last_called_at": "2026-07-02T10:00:00+08:00",
      "last_latency_ms": 350,
      "last_error": null,
      "shares_usage_with": null
    }
  ]
}
```

Always returns exactly 4 entries in a fixed order: `main_llm`, `hyde_llm`, `embedding`, `reranker` — even if a model has never been called (`status: "never_called"`, all counters zero) or is disabled (`reranker.enabled: false` when `settings.reranker_enabled` is `False`). When `hyde_llm_provider`/`hyde_llm_model` are empty (config falls back to the main LLM), `hyde_llm.shares_usage_with` is set to `"main_llm"` and both rows naturally show identical numbers because they resolve to the same tracker key.

`status` precedence (resolved by the route handler, not by `ModelUsageEntry` itself): if the role is disabled by config (`reranker` only, when `reranker_enabled=False`) the handler always reports `"disabled"`, regardless of any historical call data the tracker may still hold from when it was previously enabled. Otherwise it passes through the tracker's own `ok` / `error` / `never_called`.

### `POST /status/models/ping`

Active check, run only for `enabled` models, in parallel (`asyncio.gather` over `run_in_threadpool`, since the underlying clients are synchronous `httpx`):

- `main_llm` / `hyde_llm`: `chat([{"role": "user", "content": "ping"}], max_tokens=1)`
- `embedding`: `embed_query("ping")`
- `reranker`: `rerank("ping", [one placeholder chunk], top_k=1)` — only when `reranker_enabled=True`

Each ping is wrapped independently so one timeout doesn't block the others. Ping calls go through the same instrumented code paths, so they naturally (and honestly) add a small amount to the token counters — this is not hidden or special-cased. Response shape is identical to `GET /status/models`, reflecting the fresh post-ping state.

---

## Frontend

### New Card: "AI Models" in `frontend/src/pages/Status/StatusPage.tsx`

Placed in `panel-left`, directly after the existing "System Health" card (conceptually related — both are live connectivity views).

- Card header: title + a "Test Connection" button (`POST /status/models/ping`, disabled + spinner while in flight).
- Body: 4 rows reusing the existing `StatusIcon` + `service-row` styling, extended with a right-aligned token count column (monospace, `toLocaleString()`, matching `ConfigRow`'s number formatting) and a small last-called relative-time hint.
- `never_called` and `disabled` map to the existing muted/info badge styles already used elsewhere on this page (no new visual language needed).
- **Cleanup**: the existing "Runtime" card (`panel-right`) currently shows a single Reranker enabled/model line — this is removed from that card since the new "AI Models" card now shows it with richer detail (status + tokens), avoiding duplicate information on the page.

### Data & Types

- `frontend/src/api/status.ts`: add `getModelUsage()` (`GET /status/models`) and `pingModelConnections()` (`POST /status/models/ping`).
- `frontend/src/api/index.ts`: add `ModelUsageEntry` / `ModelUsageResponse` types alongside the existing `SystemStats`/`SystemConfig`/`SystemHealth`.
- `StatusPage.tsx`: extend the existing `Promise.allSettled([...])` fetch-on-mount/refresh with a 4th parallel call for model usage, following the same "partial failure doesn't crash the page" pattern already used for stats/health/config.
- i18n: add new keys under the existing `t.status.*` namespace in both `frontend/src/locales/en.ts` and `zh.ts` (card title, role labels, status labels, button label, "shares usage with main LLM" note).
- Desktop-first, no responsive/mobile work, per `AGENTS.md`.

### Files Changed

| File | Action |
|---|---|
| `backend/app/shared/model_usage_tracker.py` | New — `ModelUsageEntry`, `ModelUsageTracker`, `get_model_usage_tracker()` |
| `backend/app/services/llm/tracked_client.py` | New — `TrackedLLMClient` wrapper |
| `backend/app/services/llm/llm_factory.py` | Wrap client with `TrackedLLMClient` in `LLMFactory.create()` before caching |
| `backend/app/infrastructure/embedding/openai_compatible_embedding_provider.py` | Capture `usage` from embeddings response, record to tracker |
| `backend/app/infrastructure/vectorstore/cross_encoder_reranker.py` | Record call success/failure + latency to tracker |
| `backend/app/api/routes/status.py` | Add `GET /status/models`, `POST /status/models/ping` |
| `frontend/src/api/status.ts` | Add `getModelUsage()`, `pingModelConnections()` |
| `frontend/src/api/index.ts` | Add `ModelUsageEntry`/`ModelUsageResponse` types |
| `frontend/src/pages/Status/StatusPage.tsx` | Add "AI Models" card; remove duplicate reranker line from "Runtime" card |
| `frontend/src/locales/en.ts`, `zh.ts` | Add new `status.*` keys |

---

## Error Handling

- Tracker `record()` never raises — internal `try/except Exception: logger.warning(...)`, so a bug in observability code cannot break a real RAG answer, HyDE expansion, or compliance review call.
- `GET /status/models` mirrors the existing per-service try/except pattern already used in `/status/health` — a failure resolving one role's config falls back to a safe "unknown" entry rather than a 500 for the whole endpoint.
- `POST /status/models/ping`: each per-model ping is wrapped individually (`asyncio.gather(..., return_exceptions=True)` or equivalent per-task try/except); one model's timeout/error does not prevent the other three from completing and being reported.
- Frontend: ping failures surface as inline text on that row (existing `service-row` already supports a muted "detail" slot); page-level fetch failures already degrade gracefully via the existing `Promise.allSettled` fallback pattern.

## Testing

Backend (existing `pytest` setup, `backend/tests/`):
- `backend/tests/shared/test_model_usage_tracker.py` (new) — accumulation across multiple `record()` calls, status transitions (`never_called` → `ok` → `error`), basic concurrent-write safety.
- Test for `TrackedLLMClient` — verifies it delegates `chat()` faithfully (return value unchanged) while recording usage, and that a wrapped-client exception still propagates correctly.
- `backend/tests/api/test_status_models_routes.py` (new) — `GET /status/models` returns exactly 4 roles with correct defaults when nothing has been called yet (including `reranker.enabled == settings.reranker_enabled`); `POST /status/models/ping` with mocked clients (no real network calls in tests), verifying partial-failure handling.

Frontend: no test framework exists in this repo today (`frontend/package.json` has no test script, no vitest/jest config) — per project convention, this feature does not introduce one. Verification is `npm --prefix frontend run lint` + `npm --prefix frontend run build`, plus manual visual check of the new card.

## Known Limitations

- **Streaming token gap**: `stream_chat()` implementations in `DeepSeekClient`/`QwenClient` currently only yield content deltas and do not parse a trailing `usage` chunk (would require requesting `stream_options: {include_usage: true}` from the gateway and handling the final SSE chunk). This means token counts from streamed chat (the main RAG chat UI's default interaction mode) are **not** captured in this iteration — only call count/latency/success are recorded for streaming calls. Non-streaming calls (HyDE, agentic intent/plan/grounding steps, compliance review, document summarization, perception analysis) are fully captured. This gap is called out explicitly rather than silently under-counting without explanation, and is a natural follow-up.
- In-memory only: counters reset on every backend restart/redeploy; acceptable per explicit product decision in this design (no new DB table).

## Out of Scope (deferred to future iterations)

- Persistent historical token usage (Postgres-backed, time-windowed charts).
- Cost/spend estimation in currency.
- Per-session/per-user attribution.
- Parsing streaming `usage` chunks for exact streaming token counts.
- Full Langfuse distributed tracing + Ragas/LLM-as-Judge faithfulness scoring (existing roadmap `P0-A` remains the larger follow-on effort; this feature's tracker data model is intentionally simple and would need to coexist with, not replace, a future Langfuse integration).