Guangfei.Zhao
8c0fcd84e8
- Added object naming conventions (PO/DAO/BO/DTO/VO) in 02-layering.md to clarify terminology and usage within the team. - Updated 06-api-design.md to include MapStruct for DTO and entity conversion, providing examples and configuration details. - Expanded 07-config-governance.md with local development instructions and strategies for running without K8s, including two recommended approaches. - Included K8s probe configuration details in 08-observability.md for liveness and readiness checks. - Clarified CI/CD processes in 09-build-deploy.md, detailing environment distinctions and deployment strategies for local, Dev, UAT, and Prod. - Introduced ArchUnit for architectural testing in 10-testing.md, ensuring adherence to defined layering rules and coverage verification with Jacoco.
197 lines
9.2 KiB
Markdown
197 lines
9.2 KiB
Markdown
# 08. 可观测性
|
||
|
||
## 决策
|
||
|
||
统一 Trace ID + 结构化（JSON）日志 + Micrometer 指标，对应架构图 `Cross-Cutting` 里的 `Observability` 要求；关键行为单独走审计日志通道，对应 `Audit / Security`。
|
||
|
||
## 结构约定
|
||
|
||
```
|
||
platform-observability/
|
||
TraceIdFilter # 入口生成/透传 traceId，写入 MDC
|
||
logback-spring.xml # 结构化日志格式配置
|
||
MetricsConfig # Micrometer 基础配置，暴露 /actuator/prometheus
|
||
AuditLogAspect # AOP 切面，标注 @Audited 的方法自动记录审计日志
|
||
```
|
||
|
||
## `TraceIdFilter` 示例
|
||
|
||
```kotlin
|
||
// platform-observability/.../TraceIdFilter.kt
|
||
class TraceIdFilter : OncePerRequestFilter() {
|
||
override fun doFilterInternal(request: HttpServletRequest, response: HttpServletResponse, chain: FilterChain) {
|
||
val traceId = request.getHeader("X-Trace-Id") ?: UUID.randomUUID().toString()
|
||
MDC.put("traceId", traceId)
|
||
response.setHeader("X-Trace-Id", traceId)
|
||
try {
|
||
chain.doFilter(request, response)
|
||
} finally {
|
||
MDC.clear() // 必须清理，否则线程池复用线程会带出上一个请求的 traceId
|
||
}
|
||
}
|
||
}
|
||
|
||
object TraceIdHolder {
|
||
fun current(): String = MDC.get("traceId") ?: "unknown"
|
||
}
|
||
```
|
||
|
||
调用 F6/Mini 域时，把当前 `traceId` 透传到下游请求头，方便跨系统关联日志：
|
||
|
||
```kotlin
|
||
webClient.get()
|
||
.uri("/f6/procurement/list")
|
||
.header("X-Trace-Id", TraceIdHolder.current())
|
||
.retrieve()
|
||
// ...
|
||
```
|
||
|
||
## 结构化日志配置示例
|
||
|
||
```xml
|
||
<!-- logback-spring.xml -->
|
||
<configuration>
|
||
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
|
||
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
|
||
<includeMdcKeyName>traceId</includeMdcKeyName>
|
||
<customFields>{"app":"conti-backend"}</customFields>
|
||
</encoder>
|
||
</appender>
|
||
<root level="INFO">
|
||
<appender-ref ref="JSON" />
|
||
</root>
|
||
</configuration>
|
||
```
|
||
|
||
```groovy
|
||
// build.gradle
|
||
implementation 'net.logstash.logback:logstash-logback-encoder:7.4'
|
||
```
|
||
|
||
输出的每条日志会带上 `traceId` 字段，直接对接现有 ELK 方案（见 `Architecture-Diagram/ODP ELK Logging Solution Project - Overview.pdf`）时可以直接按 `traceId` 过滤出一次请求的完整链路日志。
|
||
|
||
## Micrometer / Actuator 配置
|
||
|
||
```yaml
|
||
# application.yml
|
||
management:
|
||
endpoints:
|
||
web:
|
||
exposure:
|
||
include: health, prometheus, info
|
||
endpoint:
|
||
health:
|
||
probes:
|
||
enabled: true # 暴露 /actuator/health/liveness、/readiness，供 K8s 探针使用
|
||
```
|
||
|
||
```groovy
|
||
implementation 'org.springframework.boot:spring-boot-starter-actuator'
|
||
implementation 'io.micrometer:micrometer-registry-prometheus'
|
||
```
|
||
|
||
## K8s 探针配置（liveness / readiness）
|
||
|
||
`management.endpoint.health.probes.enabled=true` 只是让 Spring Boot 暴露出 `/actuator/health/liveness`、`/actuator/health/readiness` 两个分组端点，真正让 K8s 用起来还需要在 Deployment 里配置探针指向这两个端点：
|
||
|
||
```yaml
|
||
# k8s/deployment-uat.yaml（节选，补充探针配置）
|
||
spec:
|
||
containers:
|
||
- name: conti-backend
|
||
livenessProbe:
|
||
httpGet:
|
||
path: /actuator/health/liveness
|
||
port: 8080
|
||
initialDelaySeconds: 30 # 给 JVM 启动、Flyway migration 留够时间，太短会导致刚启动就被误杀重启
|
||
periodSeconds: 10
|
||
readinessProbe:
|
||
httpGet:
|
||
path: /actuator/health/readiness
|
||
port: 8080
|
||
initialDelaySeconds: 10
|
||
periodSeconds: 5
|
||
```
|
||
|
||
两者失败后的处理完全不同，容易搞混：
|
||
|
||
- **`livenessProbe` 失败** → K8s 认为这个 Pod 已经"死掉"（比如死锁、内存泄漏导致完全无响应），直接**重启**这个 Pod。
|
||
- **`readinessProbe` 失败** → K8s 只是把这个 Pod 从 Service 的 Endpoints 里**摘除**（不再转发流量给它），不重启；等探针恢复健康后自动重新加回来——典型场景是数据库连接池暂时耗尽、正在处理慢请求，这种情况不需要重启，只需要暂时别把新流量导过去。
|
||
|
||
`readiness` group 默认会包含数据库连接（`DataSourceHealthIndicator`）等下游依赖检查，`liveness` group 默认只检查应用自身状态（不含外部依赖）——这个区分本身也是为了避免"F6 挂了导致 liveness 失败、Pod 被不断重启"这种误杀，外部依赖异常应该走 [05-integration-layer.md](./05-integration-layer.md) 的熔断降级，而不是拖累 K8s 探针。
|
||
|
||
## Resilience4j 指标接入 Micrometer
|
||
|
||
[05-integration-layer.md](./05-integration-layer.md) 里给 F6/Mini 调用配置的超时、重试、熔断器，本身的运行状态（比如熔断器当前是 `CLOSED`/`OPEN`/`HALF_OPEN`，重试了多少次）也应该能在监控里看到，不然只能等到线上报错才知道降级生效了：
|
||
|
||
```groovy
|
||
// build.gradle
|
||
implementation 'io.github.resilience4j:resilience4j-micrometer:2.2.0'
|
||
```
|
||
|
||
加上这个依赖后，`CircuitBreakerRegistry`/`RetryRegistry`/`TimeLimiterRegistry` 会自动把状态注册成 Micrometer meter，不需要手写埋点代码，跟着现有的 `/actuator/prometheus` 一起暴露出去，常用的几个：
|
||
|
||
- `resilience4j_circuitbreaker_state{name="f6-api", state="open"}`：熔断器当前状态（0/1），可以直接在 Grafana 上画出"F6 熔断器什么时候跳闸"的时间线。
|
||
- `resilience4j_circuitbreaker_calls{name="f6-api", kind="failed"}`：调用失败次数，配合 `kind="successful"` 算出实时失败率。
|
||
- `resilience4j_retry_calls{name="f6-api", kind="successful_with_retry"}`：重试后成功的次数，能看出"降级到底靠不靠重试兜住的"。
|
||
|
||
这几个指标配合 [Prometheus 告警规则](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)，可以在熔断器进入 `OPEN` 状态时直接告警，而不是等用户反馈"下单功能卡住了"才发现。
|
||
|
||
## 审计日志示例
|
||
|
||
```kotlin
|
||
// platform-observability/.../Audited.kt
|
||
@Target(AnnotationTarget.FUNCTION)
|
||
@Retention(AnnotationRetention.RUNTIME)
|
||
annotation class Audited(val action: String)
|
||
|
||
// platform-observability/.../AuditLogAspect.kt
|
||
@Aspect
|
||
@Component
|
||
class AuditLogAspect(private val storeContextHolder: StoreContextHolder) {
|
||
private val auditLog = LoggerFactory.getLogger("AUDIT")
|
||
|
||
@Around("@annotation(audited)")
|
||
fun logAudit(joinPoint: ProceedingJoinPoint, audited: Audited): Any? {
|
||
val result = runCatching { joinPoint.proceed() }
|
||
auditLog.info(
|
||
"action={} userId={} storeId={} traceId={} success={}",
|
||
audited.action, storeContextHolder.userId, storeContextHolder.storeId,
|
||
TraceIdHolder.current(), result.isSuccess,
|
||
)
|
||
return result.getOrThrow()
|
||
}
|
||
}
|
||
|
||
// 使用方式
|
||
@Audited(action = "WEBVIEW_TICKET_ISSUE")
|
||
fun issueTicket(userId: Long, storeId: Long): WebviewTicket { ... }
|
||
```
|
||
|
||
审计日志走独立 logger（`AUDIT`），在 `logback-spring.xml` 里单独配置一个 appender 写到专门的审计日志文件/索引，不和普通业务日志混在一起，方便设置更长的保留期和更严格的访问权限。
|
||
|
||
## 关键规则
|
||
|
||
- `traceId` 从入口 filter 生成，贯穿到 `f6-integration` / `mini-clients` 调用外部系统，失败时把 `traceId` 一起返回给前端（已经在 [06-api-design.md](./06-api-design.md) 的 `ApiResult` 里），方便排障（对应架构图 Flow 2 的"失败可支持排障"要求）。
|
||
- 审计相关的关键行为（登录、换票、供应商调用失败）走单独的审计日志通道，不和普通业务日志混在一起。
|
||
- 日志/指标最终对接现有 ELK 方案，具体接入方式（Filebeat 采集 stdout，还是直接推 Logstash）待确认。
|
||
|
||
## 附录：为什么要在 MDC 里放 traceId，而不是每条日志手动传参
|
||
|
||
不用 `MDC` 的话，每个方法打日志都要显式传 `traceId` 参数：`log.info("traceId={} 门店切换成功", traceId)`，深层调用链里每一层都要多加一个参数，代码侵入性很强，还容易漏传。`MDC`（Mapped Diagnostic Context）是日志框架提供的"线程内隐式上下文"，在 filter 里设置一次，同一线程内后续所有日志调用（不管调用链多深）都会自动带上这个字段，日志格式配置里声明 `includeMdcKeyName` 即可，业务代码完全不需要感知 `traceId` 的传递。
|
||
|
||
代价和 [04-security-auth.md](./04-security-auth.md) 里提到的 `ThreadLocal` 类似：`MDC` 底层也是 `ThreadLocal` 实现的，异步线程池、协程切换线程的场景需要手动透传（`MDC.getCopyOfContextMap()` 传给子线程），我们当前同步 Servlet 栈下不需要特殊处理，但如果某个模块引入异步处理要注意这一点。
|
||
|
||
## 待补充
|
||
|
||
- 具体接入现有 ELK / APM 的方式和字段规范。
|
||
- 审计日志的存储和保留策略。
|
||
|
||
## 参考链接
|
||
|
||
- [SLF4J MDC 官方文档](https://www.slf4j.org/manual.html#mdc)
|
||
- [Micrometer 官方文档](https://docs.micrometer.io/micrometer/reference/)
|
||
- [Spring Boot Actuator 官方文档](https://docs.spring.io/spring-boot/reference/actuator/index.html)
|
||
- [Spring Boot Kubernetes Probes 官方文档](https://docs.spring.io/spring-boot/reference/actuator/kubernetes-probes.html)
|
||
- [Resilience4j Micrometer 官方文档](https://resilience4j.readme.io/docs/micrometer)
|