feat: 处理整数溢出并持久化无法处理的数据
- 更新 heartbeatProcessor.js 以处理所有数字溢出类型(int16、int32、int64)并使用二进制补码。 - 防止仅与数据相关的 PostgreSQL 失败抛出个别回退错误。 - 在 databaseManager.js 中添加 insertHeartbeatEventsErrors 方法以存储被拒绝的记录。 - 更新 heartbeatProcessor.js 中的 _emitRejectedRecord 方法,直接将所有无法处理的心跳数据写入 heartbeat_events_errors 数据库。 - 更新 openspec 规范以支持新的溢出和验证回退状态。 - 添加测试文件以验证大整数处理。
This commit is contained in:
15
openspec/changes/fix-uint64-overflow/proposal.md
Normal file
15
openspec/changes/fix-uint64-overflow/proposal.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Change: Handle integer overflows and persist unprocessable data
|
||||
|
||||
## Why
|
||||
Hardware devices occasionally report `service_mask` or other bitmasks and counters (like `power_state`) where their unsigned values exceed PostgreSQL's signed boundaries (e.g. `uint64`, `uint32`, `uint16`). This triggers out of range database insertion errors which causes batch failure and falls back to individual row insertions. Previously, rows that failed due to data range constraints directly crashed out or were only logged to Redis, meaning fully invalid data or boundary constraint violations were fundamentally lost from DB history.
|
||||
|
||||
## What Changes
|
||||
- Safely map completely oversized integers to signed `int64`, `int32`, `int16` 2's complement equivalents natively in Javascript (e.g. `(v << 16) >> 16` for `int2`).
|
||||
- Refine the loop mechanism in `databaseManager.js` to avoid throwing errors exclusively built from data-level constraint mismatches when doing individual row fallback.
|
||||
- Extend `_emitRejectedRecord` to persist any unprocessable, validation-failing, or insert-failing raw records directly into a dedicated error database table: `heartbeat_events_errors`.
|
||||
|
||||
## Impact
|
||||
- Affected specs: `processor`
|
||||
- Affected code:
|
||||
- `src/processor/heartbeatProcessor.js`
|
||||
- `src/db/databaseManager.js`
|
||||
17
openspec/changes/fix-uint64-overflow/specs/processor/spec.md
Normal file
17
openspec/changes/fix-uint64-overflow/specs/processor/spec.md
Normal file
@@ -0,0 +1,17 @@
|
||||
## MODIFIED Requirements
|
||||
### Requirement: 心跳数据转换
|
||||
系统 MUST 能够将解包后的心跳数据转换为数据库存储格式。
|
||||
|
||||
#### Scenario: 转换为 v2 明细表字段
|
||||
- **WHEN** 心跳数据验证通过时
|
||||
- **THEN** 系统应输出与 v2 明细表字段一致的数据结构
|
||||
- **AND** 添加必要的元数据
|
||||
|
||||
#### Scenario: 缺失必填字段
|
||||
- **WHEN** 心跳数据缺失必填字段时
|
||||
- **THEN** 系统应判定为无效数据并丢弃
|
||||
|
||||
#### Scenario: 兼容超限的 uint64 字段转换为有符号 bigint
|
||||
- **WHEN** `service_mask` 等字段由于无符号位满转为十进制后超出 PostgreSQL 的 signed bigint 上限
|
||||
- **THEN** 系统 MUST 将该长数字利用补码映射转换其等价二进制结构对应的有符号负数
|
||||
- **AND** 保证原始 Bit 掩码结构完整并确保能顺利存入数据库
|
||||
6
openspec/changes/fix-uint64-overflow/tasks.md
Normal file
6
openspec/changes/fix-uint64-overflow/tasks.md
Normal file
@@ -0,0 +1,6 @@
|
||||
## 1. Implementation
|
||||
- [x] 1.1 Update `heartbeatProcessor.js` to handle all numeric overflow types (`int16`, `int32`, `int64`) with two's complement.
|
||||
- [x] 1.2 Prevent purely data-related Postgres failures from throwing away individual fallbacks within `databaseManager.js`.
|
||||
- [x] 1.3 Add `insertHeartbeatEventsErrors` to `databaseManager.js` to sink rejected records.
|
||||
- [x] 1.4 Wire `_emitRejectedRecord` in `heartbeatProcessor.js` to directly write all completely unprocessable heartbeats into the `heartbeat_events_errors` DB.
|
||||
- [x] 1.5 Update the `openspec` specs with these newly supported overflow & validation fallback states.
|
||||
Reference in New Issue
Block a user