feat: 更新 Kafka 配置和数据库管理逻辑
- 在 .env.example 中添加 Kafka 配置项：KAFKA_FETCH_MAX_BYTES, KAFKA_FETCH_MIN_BYTES, KAFKA_FETCH_MAX_WAIT_MS。 - 删除 room_status_sync 提案及相关文档。 - 删除 fix_uint64_overflow 提案及相关文档。 - 更新数据库管理器以支持使用 COPY 语句进行高效数据写入，替换批量 INSERT 逻辑。 - 实现心跳数据的整数溢出处理，确保无效数据被持久化到 heartbeat_events_errors 表。 - 更新处理器规范，确保心跳数据成功写入历史表后触发 room_status 同步。 - 添加新文档，描述新的分区方法案例。 - 归档旧的提案和规范文档以保持项目整洁。
This commit is contained in:
@@ -0,0 +1,20 @@
|
||||
# Proposal: Synchronize Heartbeat Data to Room Status Table
|
||||
|
||||
## Background
|
||||
The `room_status.room_status_moment` table is a shared table for real-time device status. The heartbeat service needs to synchronize relevant fields from Kafka messages to this table.
|
||||
|
||||
## Changes
|
||||
1. **Database Schema**:
|
||||
* Add `bright_g` (INT2) and `agreement_ver` (TEXT) columns to `room_status.room_status_moment`.
|
||||
* Add a UNIQUE INDEX on `(hotel_id, room_id, device_id)` to support efficient UPSERT operations.
|
||||
|
||||
2. **Application Logic**:
|
||||
* Implement `upsertRoomStatus` in `DatabaseManager`.
|
||||
* Call this method in `HeartbeatProcessor` after successful insertion into the history table.
|
||||
* Map `version` to `agreement_ver` and `bright_g` to `bright_g`.
|
||||
|
||||
## Tasks
|
||||
- [x] Update `docs/room_status_moment.sql` with new columns and index.
|
||||
- [x] Update `docs/plan-room-status-sync.md` with new fields and finalized plan.
|
||||
- [x] Implement `upsertRoomStatus` in `DatabaseManager`.
|
||||
- [x] Integrate into `HeartbeatProcessor`.
|
||||
@@ -0,0 +1,9 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 数据库表结构管理
|
||||
系统 MUST 支持 room_status 实时状态表与心跳历史表的协同写入能力。
|
||||
|
||||
#### Scenario: room_status 结构与约束支持
|
||||
- **WHEN** 心跳服务执行 room_status upsert 同步
|
||||
- **THEN** 目标表应具备支撑 UPSERT 的唯一约束（hotel_id, room_id, device_id）
|
||||
- **AND** 需要的同步字段应存在并可写入
|
||||
@@ -0,0 +1,9 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: 心跳数据转换
|
||||
系统 MUST 在历史表写入成功后触发 room_status 同步。
|
||||
|
||||
#### Scenario: 历史写入成功后同步状态表
|
||||
- **WHEN** 一批心跳数据成功写入历史分区表
|
||||
- **THEN** 系统应调用 room_status 的 upsert 同步逻辑
|
||||
- **AND** 同步失败不应阻塞主历史写入流程
|
||||
@@ -0,0 +1,24 @@
|
||||
# Change: Handle integer overflows, persist unprocessable data, and use COPY for extreme write performance
|
||||
|
||||
## Why
|
||||
1. **Overflows**: Hardware devices report bitmasks and counters where unsigned values exceed PostgreSQL's signed boundaries (e.g. `uint64`), causing out of range Database errors.
|
||||
2. **Missing History**: Previously, fully unprocessable rows directly crashed out or vanished into Redis logs, causing data loss without DB traceability.
|
||||
3. **Database Write Pressure**: The multi-row `INSERT INTO ... VALUES (...)` logic built queries with up to 30,000 parameter bindings for a batch, creating massive CPU load for parsing scaling synchronously via a single database connection.
|
||||
|
||||
## What Changes
|
||||
- Use `pg-copy-streams` to replace batch `INSERT` with `COPY heartbeat.heartbeat_events FROM STDIN WITH (FORMAT text)`. The objects are automatically formatted into a memory buffer and flushed down a pipeline stream, completely saving parameter parsing overhead by ~90%.
|
||||
- Refine array text mapping natively avoiding SQL JSONB mapping latency and safely escaping special string quotes.
|
||||
- Safely map completely oversized integers to signed `int64`, `int32`, `int16` 2's complement equivalents natively in Javascript.
|
||||
- Implement an upstream catch that automatically redirects isolated PostgreSQL validation exceptions alongside JSON parsing defects purely to the new `heartbeat_events_errors` table batch-sink for complete persistence.
|
||||
|
||||
## Impact
|
||||
- Affected specs: `processor`
|
||||
- Affected code:
|
||||
- `src/processor/heartbeatProcessor.js`
|
||||
- `src/db/databaseManager.js`
|
||||
- `package.json`
|
||||
|
||||
## Recorded Follow-ups (2026-03-03)
|
||||
- Partition maintenance strategy is adjusted to avoid redundant per-partition index creation during daily partition pre-creation; parent-table indexes remain the primary index definition source.
|
||||
- Legacy compatibility paths for old heartbeat schema access are removed from implementation scope, and the project baseline is aligned to the new partitioned schema.
|
||||
- Kafka consume/write throughput tuning knobs are documented and exposed via runtime envs (`KAFKA_FETCH_MAX_BYTES`, `KAFKA_FETCH_MIN_BYTES`, `KAFKA_FETCH_MAX_WAIT_MS`, `KAFKA_MAX_IN_FLIGHT_MESSAGES`, `PROCESSOR_BATCH_SIZE`, `PROCESSOR_BATCH_TIMEOUT`).
|
||||
@@ -0,0 +1,17 @@
|
||||
## MODIFIED Requirements
|
||||
### Requirement: 心跳数据转换
|
||||
系统 MUST 能够将解包后的心跳数据转换为数据库存储格式。
|
||||
|
||||
#### Scenario: 转换为 v2 明细表字段
|
||||
- **WHEN** 心跳数据验证通过时
|
||||
- **THEN** 系统应输出与 v2 明细表字段一致的数据结构
|
||||
- **AND** 添加必要的元数据
|
||||
|
||||
#### Scenario: 缺失必填字段
|
||||
- **WHEN** 心跳数据缺失必填字段时
|
||||
- **THEN** 系统应判定为无效数据并丢弃
|
||||
|
||||
#### Scenario: 兼容超限的 uint64 字段转换为有符号 bigint
|
||||
- **WHEN** `service_mask` 等字段由于无符号位满转为十进制后超出 PostgreSQL 的 signed bigint 上限
|
||||
- **THEN** 系统 MUST 将该长数字利用补码映射转换其等价二进制结构对应的有符号负数
|
||||
- **AND** 保证原始 Bit 掩码结构完整并确保能顺利存入数据库
|
||||
@@ -0,0 +1,6 @@
|
||||
## 1. Implementation
|
||||
- [x] 1.1 Support all numeric overflow types (`int16`, `int32`, `int64`) with JavaScript two's complement mapped conversions.
|
||||
- [x] 1.2 Replace batch `INSERT` logic with `COPY... FROM STDIN` streaming using `pg-copy-streams` to achieve massive raw write throughput.
|
||||
- [x] 1.3 Add dynamic string formatter mapping arrays properly for native PostgreSQL Tab-Separated ingestion formatting.
|
||||
- [x] 1.4 Wire fallback inserts explicitly tracking fully failed outputs into the isolated `heartbeat_events_errors` capture bucket.
|
||||
- [x] 1.5 Update the `openspec` specs representing these architectural and constraint transitions.
|
||||
Reference in New Issue
Block a user