AIRegulation/AIRegulation-DocAnalysis
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
91651682a3a41be38db65d16cab4f6617b4f9d3d
AIRegulation-DocAnalysis/docs/architecture/aliyun-ingest-implementation.md

72 lines
2.1 KiB
Markdown
Raw Normal View History

feat: Migrate document parsing to Aliyun and update embedding configurations - Updated LocalDocumentParser to include raw_layouts and artifact_prefix from settings. - Added new documents with failure reasons and metadata to documents.json for better error tracking. - Created a new documentation file detailing the Aliyun ingest implementation process. - Updated RFC to reflect changes in the parsing backend and embedding dimensions. - Modified tests to accommodate the new embedding dimension of 1024 and updated parser and chunk builder assertions. - Verified migration configurations to ensure correct settings for embedding model and backend.
2026-05-18 22:30:28 +08:00
# 阿里云解析主链路实现说明
本文档描述当前仓库已经落地的文档 ingest 主链路实现，作为迁移设计到代码实现之间的收口说明。
## 1. 当前默认链路
- 上传入口保持为 `/api/v1/documents/upload`
- 默认 `PARSER_BACKEND=aliyun`
- 默认 `CHUNK_BACKEND=aliyun`
Refactor document handling and update Milvus collection settings - Removed multiple failed document entries from `documents.json`. - Added a new document entry with updated metadata and changed the index name to `regulations_dense_1024_v2`. - Updated architecture documentation to reflect changes in the Milvus collection name. - Adjusted requirements by removing the sqlalchemy dependency. - Modified test cases to align with new document structure and naming conventions. - Introduced a new test file for Milvus vector index runtime recovery and error handling. - Updated assertions in various test files to ensure compatibility with the new schema.
2026-05-26 20:21:31 +08:00
- 默认 Milvus collection 为 `regulations_dense_1024_v2`
feat: Migrate document parsing to Aliyun and update embedding configurations - Updated LocalDocumentParser to include raw_layouts and artifact_prefix from settings. - Added new documents with failure reasons and metadata to documents.json for better error tracking. - Created a new documentation file detailing the Aliyun ingest implementation process. - Updated RFC to reflect changes in the parsing backend and embedding dimensions. - Modified tests to accommodate the new embedding dimension of 1024 and updated parser and chunk builder assertions. - Verified migration configurations to ensure correct settings for embedding model and backend.
2026-05-18 22:30:28 +08:00
- 解析产物落到 MinIO `artifacts/{doc_id}/`
完整主链路如下：
1. 原始文件上传到 MinIO
2. `AliyunDocmindGateway` 提交阿里云异步解析任务
3. 轮询任务状态直到成功或超时
4. 分页拉取 `layouts`
5. 转换为 `structure_nodes / semantic_blocks / vector_chunks`
6. 三层结构 JSON 回写 MinIO
7. 使用 `vector_chunks[*].embedding_text` 调 embedding API
Refactor document handling and update Milvus collection settings - Removed multiple failed document entries from `documents.json`. - Added a new document entry with updated metadata and changed the index name to `regulations_dense_1024_v2`. - Updated architecture documentation to reflect changes in the Milvus collection name. - Adjusted requirements by removing the sqlalchemy dependency. - Modified test cases to align with new document structure and naming conventions. - Introduced a new test file for Milvus vector index runtime recovery and error handling. - Updated assertions in various test files to ensure compatibility with the new schema.
2026-05-26 20:21:31 +08:00
8. 写入 `regulations_dense_1024_v2`
feat: Migrate document parsing to Aliyun and update embedding configurations - Updated LocalDocumentParser to include raw_layouts and artifact_prefix from settings. - Added new documents with failure reasons and metadata to documents.json for better error tracking. - Created a new documentation file detailing the Aliyun ingest implementation process. - Updated RFC to reflect changes in the parsing backend and embedding dimensions. - Modified tests to accommodate the new embedding dimension of 1024 and updated parser and chunk builder assertions. - Verified migration configurations to ensure correct settings for embedding model and backend.
2026-05-18 22:30:28 +08:00
9. 文档状态更新为 `indexed`
运行时转换逻辑位于 `backend/app/infrastructure/parser/aliyun_layout_normalizer.py`。
旧的 `backend/app/aliyun_parser/` 示例目录已移除，不参与生产运行时。
## 2. 解析产物持久化
每个文档会额外写入以下对象：
- `artifacts/{doc_id}/layouts.json`
- `artifacts/{doc_id}/structure_nodes.json`
- `artifacts/{doc_id}/semantic_blocks.json`
- `artifacts/{doc_id}/vector_chunks.json`
`documents.json` 仅保留对象 key、统计信息和处理阶段，不保存完整大 JSON。
## 3. 失败策略
- 当前 `PARSER_FAILURE_MODE=fail`
- 阿里云解析失败不自动回退到本地 parser
- 失败时保留原始文件与已写入的 artifacts，便于排障
## 4. 运行参数
关键环境变量如下：
- `ALIBABA_ACCESS_KEY_ID`
- `ALIBABA_ACCESS_KEY_SECRET`
- `ALIBABA_ENDPOINT`
- `ALIYUN_PARSE_POLL_INTERVAL_SECONDS`
- `ALIYUN_PARSE_TIMEOUT_SECONDS`
- `ALIYUN_PARSE_LAYOUT_STEP_SIZE`
- `ALIYUN_LLM_ENHANCEMENT`
- `ALIYUN_ENHANCEMENT_MODE`
- `DOCUMENT_PARSE_ARTIFACT_PREFIX`
- `PARSER_BACKEND`
- `CHUNK_BACKEND`
## 5. 运行态确认
可通过 `/api/v1/status/config` 确认以下字段：
- `parser_backend`
- `chunk_backend`
- `milvus_collection`
- `artifact_prefix`
- `parser_failure_mode`
这几个值用于确认服务是否实际运行在迁移后的默认链路上。
Reference in New Issue Copy Permalink