feat: 初始化 bls-onoffline-backend 项目基础结构
添加 Kafka 消费者、数据库写入、Redis 集成等核心模块,实现设备上下线事件处理 - 创建项目基础目录结构与配置文件 - 实现 Kafka 消费逻辑与手动提交偏移量 - 添加 PostgreSQL 数据库连接与分区表管理 - 集成 Redis 用于错误队列和项目心跳 - 包含数据处理逻辑,区分重启与非重启数据 - 提供数据库初始化脚本与分区创建工具 - 添加单元测试与代码校验脚本
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
# Change: Fix Kafka Partitioning and Schema Issues
|
||||
|
||||
## Why
|
||||
Production deployment revealed issues with data ingestion:
|
||||
1. Kafka Topic name changed to include partition suffix.
|
||||
2. Legacy data contains second-level timestamps (1970s) causing partition lookup failures in PostgreSQL (which expects ms).
|
||||
3. Variable-length fields (reboot reason, status) exceeded VARCHAR(10) limits, causing crashes.
|
||||
|
||||
## What Changes
|
||||
- **Modified Requirement**: Update Kafka Topic to `blwlog4Nodejs-rcu-onoffline-topic-0`.
|
||||
- **New Requirement**: Implement heuristic timestamp conversion (Sec -> MS) for values < 100B.
|
||||
- **New Requirement**: Truncate specific fields to VARCHAR(255) to prevent DB rejection.
|
||||
- **Modified Requirement**: Update DB Schema to VARCHAR(255) for robustness.
|
||||
|
||||
## Impact
|
||||
- Affected specs: `onoffline`
|
||||
- Affected code: `src/processor/index.js`, `scripts/init_db.sql`
|
||||
@@ -0,0 +1,25 @@
|
||||
## MODIFIED Requirements
|
||||
### Requirement: 消费并落库
|
||||
系统 SHALL 从 blwlog4Nodejs-rcu-onoffline-topic-0 消费消息,并写入 log_platform.onoffline.onoffline_record。
|
||||
|
||||
#### Scenario: 非重启数据写入
|
||||
- **GIVEN** RebootReason 为空或不存在
|
||||
- **WHEN** 消息被处理
|
||||
- **THEN** current_status 等于 CurrentStatus (截断至 255 字符)
|
||||
|
||||
## ADDED Requirements
|
||||
### Requirement: 字段长度限制与截断
|
||||
系统 SHALL 将部分变长字段截断至数据库允许的最大长度 (VARCHAR(255)),防止写入失败。
|
||||
|
||||
#### Scenario: 超长字段处理
|
||||
- **GIVEN** LauncherVersion, CurrentStatus 或 RebootReason 超过 255 字符
|
||||
- **WHEN** 消息被处理
|
||||
- **THEN** 字段被截断为前 255 个字符并入库
|
||||
|
||||
### Requirement: 时间戳单位自动识别
|
||||
系统 SHALL 自动识别 UnixTime 字段是秒还是毫秒,并统一转换为毫秒。
|
||||
|
||||
#### Scenario: 秒级时间戳转换
|
||||
- **GIVEN** UnixTime < 100000000000 (约 1973 年前)
|
||||
- **WHEN** 解析时间戳
|
||||
- **THEN** 自动乘以 1000 转换为毫秒
|
||||
@@ -0,0 +1,6 @@
|
||||
## 1. Implementation
|
||||
- [x] Update Kafka Topic in .env and config
|
||||
- [x] Implement timestamp unit detection and conversion in processor
|
||||
- [x] Implement field truncation logic in processor
|
||||
- [x] Update database schema definition (init_db.sql) to VARCHAR(255)
|
||||
- [x] Verify data ingestion with production stream
|
||||
Reference in New Issue
Block a user