# Regulatory Signals Intelligence Enhancement — Design Spec

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the 20-item hardcoded MockEventStore with real regulatory data from Chinese and international sources, add LLM-driven structured extraction, impact assessment, and semantic change diff — all accessible through a manual-trigger crawl in the frontend.

**Architecture:** Crawler Service (httpx + BeautifulSoup) → PostgreSQL EventStore → LLM Pipeline (extract → assess → diff) → existing PerceptionService interface. New code follows `api → application → domain ports → infrastructure` layering; no new files in `services/*` or `workflows/*`; `shared/bootstrap.py` is the composition root.

**Tech Stack:** httpx, BeautifulSoup4, sentence-transformers (for diff), existing LLM factory (deepseek/qwen), existing KnowledgeRetrievalService (RAG), PostgreSQL (already available), existing SSE infrastructure.

---

## 1. Data Sources

| Source | URL | Method | Coverage |
|--------|-----|--------|----------|
| CATARC 汽车标准 | `https://www.catarc.org.cn/bzzxd/qcbz/index.html` | httpx + BeautifulSoup (static pages) | 国家/行业汽车标准列表 |
| 国标委强制性标准 | `https://openstd.samr.gov.cn/bzgk/std/std_list_type?p.p1=1&p.p2=车&p.p90=circulation_date&p.p91=desc` | httpx + JSON API parse | 强制性国家标准，按"车"过滤 |
| 国标委推荐性标准 | `https://openstd.samr.gov.cn/bzgk/std/std_list_type?p.p1=2&p.p2=车&p.p90=circulation_date&p.p91=desc` | httpx + JSON API parse | 推荐性国家标准，按"车"过滤 |
| EUR-Lex | RSS + CELLAR REST API | pyeurlex / httpx | EU AI Act, automotive directives |
| UN R155/R156 | CELLAR REST API (CELEX lookup) | httpx | UN-ECE cybersecurity/OTA regulations |

Crawl is **manual-trigger only** — no cron/Celery Beat. Admin clicks "刷新数据源" in the frontend UI.

---

## 2. Database Schema

### New table: `regulation_events`

```sql
CREATE TABLE IF NOT EXISTS regulation_events (
    id              TEXT PRIMARY KEY,          -- sha256(source + standard_code)[:12]
    source          TEXT NOT NULL,             -- 'CATARC' | '国标委' | 'EUR-Lex' | 'UN-ECE'
    source_label    TEXT,                      -- Human-readable source label
    standard_code   TEXT NOT NULL,             -- e.g. "GB 18384-2025", "EU/2024/1689"
    title           TEXT NOT NULL,
    summary         TEXT,                      -- Crawled abstract or first paragraph
    full_text_url   TEXT,                      -- Original page URL
    status          TEXT,                      -- 'enacted' | 'draft' | 'consultation'
    impact_level    TEXT,                      -- 'high' | 'medium' | 'low' (LLM-assigned)
    published_at    DATE,
    effective_at    DATE,
    category        TEXT,
    tags            TEXT[],
    -- LLM structured extraction
    obligations     JSONB,       -- [{text, deontic, subject, object, condition}]
    deadlines       JSONB,       -- [{date, description}]
    scope           TEXT,        -- Applicability scope summary
    penalties       TEXT,        -- Penalty / consequence summary
    -- Change tracking
    content_hash    TEXT,        -- SHA256 of crawled full text
    previous_hash   TEXT,        -- Hash from prior crawl (NULL on first crawl)
    change_summary  TEXT,        -- LLM-generated description of changes
    changed_sections JSONB,      -- [{old_text, new_text, change_type}] where cosine<0.85
    -- Impact assessment
    affected_docs   JSONB,       -- [{doc_id, doc_name, score, key_clauses, recommendation}]
    -- Metadata
    crawled_at      TIMESTAMPTZ DEFAULT now(),
    processed_at    TIMESTAMPTZ,
    raw_storage_key TEXT         -- MinIO path for raw HTML/PDF (optional)
);

CREATE INDEX IF NOT EXISTS regulation_events_source_date
    ON regulation_events (source, published_at DESC);
CREATE INDEX IF NOT EXISTS regulation_events_impact_date
    ON regulation_events (impact_level, published_at DESC);
CREATE INDEX IF NOT EXISTS regulation_events_tags
    ON regulation_events USING gin(tags);
```

---

## 3. Backend Architecture

### 3.1 File Map

**New files (infrastructure layer):**
- `backend/app/infrastructure/perception/crawlers/catarc_crawler.py` — CATARC scraper
- `backend/app/infrastructure/perception/crawlers/guobiao_crawler.py` — 国标委 JSON API crawler
- `backend/app/infrastructure/perception/crawlers/eurlex_crawler.py` — EUR-Lex RSS + CELLAR
- `backend/app/infrastructure/perception/crawlers/base.py` — Abstract base class
- `backend/app/infrastructure/perception/postgres_event_store.py` — PostgresEventStore (replaces MockEventStore)
- `backend/app/infrastructure/perception/llm_pipeline.py` — Extract / assess / diff pipeline

**New files (application layer):**
- `backend/app/application/perception/crawl_service.py` — Orchestrates crawlers + LLM pipeline, exposes `run_crawl(sources)` + progress generator

**Modified files:**
- `backend/app/api/routes/perception.py` — Add `POST /crawl`, `GET /crawl/status` (SSE), `POST /events/{id}/process`, `GET /events/{id}/diff`
- `backend/app/shared/bootstrap.py` — Wire `PostgresEventStore` + `CrawlService` + `LlmPipeline` when `DOCUMENT_REPOSITORY_BACKEND=postgres`; fallback to `MockEventStore` when `json`
- `backend/app/config/settings.py` — Add `perception_crawl_timeout_seconds`, `perception_max_events_per_source`

**Unchanged files:**
- `backend/app/application/perception/services.py` — `PerceptionService` interface unchanged; only `_store` swap
- `backend/app/infrastructure/perception/mock_event_store.py` — Kept for `json` backend mode

### 3.2 Domain Port (Abstract Interface)

```python
# backend/app/infrastructure/perception/base_event_store.py
from abc import ABC, abstractmethod

class BaseEventStore(ABC):
    @abstractmethod
    def all(self) -> list[dict]: ...
    @abstractmethod
    def get(self, event_id: str) -> dict | None: ...
    @abstractmethod
    def filter(self, source=None, impact_level=None, limit=50) -> list[dict]: ...
    @abstractmethod
    def stats(self) -> dict: ...
    @abstractmethod
    def upsert(self, event: dict) -> None: ...      # new — needed for crawl writes
    @abstractmethod
    def get_by_standard_code(self, code: str) -> dict | None: ...  # for change detection
```

`MockEventStore` and `PostgresEventStore` both implement this interface.

### 3.3 Crawler Base Contract

```python
# backend/app/infrastructure/perception/crawlers/base.py
from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class RawEvent:
    source: str
    source_label: str
    standard_code: str
    title: str
    summary: str
    full_text_url: str
    status: str           # 'enacted' | 'draft' | 'consultation'
    published_at: str     # YYYY-MM-DD string
    effective_at: str | None
    category: str
    tags: list[str]
    raw_text: str         # full crawled text for hashing + LLM

class BaseCrawler(ABC):
    @abstractmethod
    def fetch(self, limit: int = 50) -> list[RawEvent]: ...
```

### 3.4 LLM Pipeline

```python
# backend/app/infrastructure/perception/llm_pipeline.py

class LlmPipeline:
    """Runs three sequential LLM steps on a regulation event."""

    def extract_structure(self, event: dict) -> dict:
        """Step 1: Extract obligations, deadlines, scope, penalties, impact_level.
        
        Returns dict with keys: obligations, deadlines, scope, penalties, impact_level.
        Uses JSON-mode or structured prompt; model retries once on parse failure.
        """

    def assess_impact(self, event: dict, retrieval_service) -> list[dict]:
        """Step 2: RAG-based impact on existing knowledge base documents.
        
        Query = standard_code + title + first obligation texts.
        Returns list of {doc_id, doc_name, score, key_clauses, recommendation}.
        """

    def compute_diff(self, old_text: str, new_text: str) -> dict:
        """Step 3: Semantic diff between old and new regulation text.
        
        Splits both texts by paragraph. Calls existing EmbeddingService (text-embedding-v3
        via EMBEDDING_BASE_URL) to embed each paragraph, then computes cosine similarity.
        Changed paragraphs (cosine < 0.85) sent to LLM for change_type classification:
          'tightened' | 'relaxed' | 'added' | 'removed'
        Returns {changed_sections: [...], change_summary: str}.
        Only called when content_hash differs from previous_hash.
        """
```

### 3.5 CrawlService

```python
# backend/app/application/perception/crawl_service.py

class CrawlService:
    def __init__(self, crawlers, event_store, llm_pipeline, retrieval_service): ...

    def run_crawl(self, sources: list[str] | None = None) -> Generator[dict, None, None]:
        """Manual-trigger crawl. Yields progress SSE dicts:
          {event: 'progress', data: {source, fetched, new, updated, stage}}
          {event: 'done', data: {total_new, total_updated, duration_ms}}
          {event: 'error', data: {source, message}}
        
        For each crawler:
          1. fetch() RawEvents
          2. hash check vs stored event → skip if unchanged
          3. upsert raw event to DB
          4. run LLM pipeline (extract → assess → diff)
          5. upsert enriched event to DB
          6. yield progress
        """
```

---

## 4. API Endpoints

### Existing (unchanged interface, new store backend)
- `GET /api/v1/perception/stats`
- `GET /api/v1/perception/events`
- `GET /api/v1/perception/events/{id}`
- `POST /api/v1/perception/events/{id}/analyze` (streaming)

### New endpoints

```
POST /api/v1/perception/crawl
  Body: { sources?: ["CATARC", "国标委", "EUR-Lex", "UN-ECE"] }
  Response: text/event-stream (SSE)
  Auth: requires current_user (admin/legal role)
  Streams progress events until done or error.

POST /api/v1/perception/events/{id}/process
  Trigger LLM pipeline for a single already-crawled event.
  Response: { status: "ok", processed_at: "..." }
  Auth: requires current_user

GET /api/v1/perception/events/{id}/diff
  Returns: { changed_sections: [...], change_summary: str, previous_hash: str }
  Returns 404 if no diff available (first crawl or no change detected).
```

---

## 5. Frontend Changes

### 5.1 New: Crawl Control Bar (top of PerceptionPage)

Above the stats-bar, add a `<CrawlBar>` component:
- "刷新数据源" button — triggers `POST /crawl` (all sources)
- Inline progress display: shows SSE progress events as a mini status line
  - e.g. "CATARC: 抓取中… | 国标委: 12 条新增 | EUR-Lex: 等待中"
- On completion: shows "更新完成 — 新增 N 条，更新 M 条"
- Disabled while crawl is in progress (prevents double-trigger)

### 5.2 Signal Card Enhancement

Existing cards get two new indicators:
- **NEW badge** — shown when `crawled_at` is within last 24h (green dot)
- **CHANGED badge** — shown when `previous_hash != content_hash` and `change_summary` exists

### 5.3 Right Panel — Structured Tab

Right detail panel adds a tab bar: **概览 | 义务条款 | 影响评估 | 变更对比**

**义务条款 tab:**
- Table: 义务描述 | 主体 | 对象 | 截止日期
- Tags for deontic type: 强制 / 禁止 / 允许
- Shows `obligations[]` + `deadlines[]` from DB

**影响评估 tab:**
- Replaces hardcoded MOCK_DOCS with real `affected_docs[]` from DB
- Each row: document name, similarity score (%), key clause excerpt, LLM recommendation
- "Run fresh assessment" button → triggers `POST /events/{id}/process`

**变更对比 tab:**
- Only visible when `change_summary` is non-null
- Top: `change_summary` text (LLM prose)
- Below: diff table with old/new paragraph pairs, change_type badge per row
- Hidden (tab disabled) on first-crawl events with no prior version

### 5.4 Existing behavior preserved
- `analyze` streaming (AI analysis) unchanged
- Search/filter (source, impact) unchanged — now hits real DB data
- Stats bar — now reflects real counts from PostgreSQL

---

## 6. Settings Additions

```python
# backend/app/config/settings.py additions
perception_crawl_timeout_seconds: int = Field(default=120, ...)
perception_max_events_per_source: int = Field(default=100, ...)
perception_diff_similarity_threshold: float = Field(default=0.85, ...)
```

```env
# .env additions
PERCEPTION_CRAWL_TIMEOUT_SECONDS=120
PERCEPTION_MAX_EVENTS_PER_SOURCE=100
PERCEPTION_DIFF_SIMILARITY_THRESHOLD=0.85
```

---

## 7. Dependencies

```
# requirements.txt additions
httpx>=0.27.0              # already likely present; confirm
beautifulsoup4>=4.12.0     # HTML parsing for CATARC
lxml>=5.0.0                # BeautifulSoup parser backend
# sentence-transformers NOT added — diff uses existing text-embedding-v3 API (EMBEDDING_BASE_URL)
```

No new infrastructure required (PostgreSQL + MinIO + Milvus already available).

---

## 8. Backward Compatibility

- `DOCUMENT_REPOSITORY_BACKEND=json` → `bootstrap.py` uses `MockEventStore` (unchanged behavior)
- `DOCUMENT_REPOSITORY_BACKEND=postgres` → uses `PostgresEventStore`
- Migration: run `CREATE TABLE` SQL on first startup (idempotent `CREATE TABLE IF NOT EXISTS`)
- Existing 20 mock events are not seeded to PostgreSQL; PostgreSQL starts empty until first crawl

---

## 9. Out of Scope (this phase)

- Automatic/scheduled crawling (Celery Beat) — manual trigger only
- Playwright-based JS-rendered pages — all target sites work with httpx
- Knowledge Graph (Neo4j / LightRAG) — future phase
- Email/Slack webhook notifications — future phase
- User-facing diff history (versioning beyond one prior snapshot) — future phase