Web_BLS_ProjectConsole/openspec/specs/logging/design.md

Logging Capability Design

Context

This design document describes the technical implementation of the logging capability for the BLS Project Console, which allows the system to read log records from Redis queues and display them in the console interface.

Goals / Non-Goals

Goals

• Implement real-time log reading from Redis queues
• Provide a user-friendly log display interface
• Support log filtering by level and time range
• Ensure high performance and low latency
• Implement proper error handling and reconnection mechanisms

Non-Goals

• Log storage or persistence beyond memory
• Log analysis or visualization (charts, graphs)
• Advanced log search capabilities

Decisions

Decision: Redis Queue Implementation

• What: Use Redis List as the queue data structure
• Why: Redis Lists provide efficient push/pop operations with O(1) time complexity, making them ideal for message queues
• Alternatives considered:
  • Redis Streams: More advanced but overkill for our use case
  • Redis Pub/Sub: No persistence, so logs would be lost if the server is down

Decision: Real-time Updates

• What: Use Server-Sent Events (SSE) for real-time log updates
• Why: SSE is simpler than WebSockets for one-way communication, has better browser support, and is easier to implement
• Alternatives considered:
  • WebSockets: More complex for one-way communication
  • Polling: Higher latency and more resource-intensive

Decision: Log Storage

• What: Store logs in memory with a configurable maximum size
• Why: In-memory storage provides fast access times and avoids the complexity of database management
• Alternatives considered:
  • Database storage: Adds complexity and latency
  • File system: Not suitable for real-time access

Architecture

Frontend Architecture

LogView Component
├── LogList Component
├── LogFilter Component
└── LogService

Backend Architecture

Log Routes
├── Log Service
│   ├── Redis Client
│   └── Log Manager
└── SSE Controller

Implementation Details

Redis Connection

• Use the redis npm package to connect to Redis
• Implement automatic reconnection with exponential backoff
• Handle connection errors gracefully

Log Reading

1. Server establishes connection to Redis
2. Server listens for new log records using BLPOP command (blocking pop)
3. When a log record is received, it's added to the in-memory log store
4. The log is then sent to all connected SSE clients

Log Storage

• Use an array to store log records in memory
• Implement a circular buffer to limit memory usage
• Default maximum log count: 10,000
• Configurable via environment variable

Log Display

• Use a scrollable list to display logs
• Implement virtual scrolling for large log sets to improve performance
• Color-code logs by level (INFO: gray, WARN: yellow, ERROR: red, DEBUG: blue)

Log Filtering

• Implement client-side filtering for performance
• Allow filtering by log level (INFO, WARN, ERROR, DEBUG)
• Allow filtering by time range using a date picker

Risks / Trade-offs

Risk: Redis Connection Failure

• Risk: If Redis connection is lost, logs won't be received
• Mitigation: Implement automatic reconnection with exponential backoff, and notify users when connection is lost

Risk: High Log Volume

• Risk: Large number of logs could cause performance issues
• Mitigation: Implement a circular buffer to limit memory usage, and use virtual scrolling in the frontend

Risk: Browser Performance

• Risk: Displaying thousands of logs could slow down the browser
• Mitigation: Use virtual scrolling and limit the number of logs displayed at once

Migration Plan

No migration is required as this is a new feature.

Open Questions

• What is the expected maximum log volume per minute?
• Should we add support for log persistence to disk?
• Should we implement log search functionality?