# Reliable Kafka Consumption & DB Offline Handling

- **Status**: Completed
- **Author**: AI Assistant
- **Created**: 2026-02-04

## Context

Currently, the Kafka consumer is configured with `autoCommit: true`. This means offsets are committed periodically regardless of whether the data was successfully processed and stored in the database. If the database insertion fails (e.g., due to a constraint violation or connection loss), the message is considered "consumed" by Kafka, leading to data loss.

Additionally, if the PostgreSQL database goes offline, the consumer continues to try processing messages, likely filling logs with errors and potentially losing data if retries aren't handled correctly. The user requires a mechanism to pause consumption during DB outages and resume only when the DB is back online.

## Proposal

We propose to enhance the reliability of the ingestion pipeline by:

1.  **Disabling Auto-Commit**:
    -   Set `autoCommit: false` in the Kafka `ConsumerGroup` configuration.
    -   Implement manual offset committing only after the database insertion is confirmed successful.

2.  **Implementing DB Offline Handling (Circuit Breaker)**:
    -   Detect database connection errors during insertion.
    -   If a connection error occurs:
        1.  Pause the Kafka consumer immediately.
        2.  Log a warning and enter a "Recovery Mode".
        3.  Wait for 1 minute.
        4.  Periodically check database connectivity (every 1 minute).
        5.  Once the database is reachable, resume the Kafka consumer.

## Technical Details

### Configuration
-   No new environment variables are strictly required, but `KAFKA_AUTO_COMMIT` could be forced to `false` or removed if we enforce this behavior.
-   Retry interval (60 seconds) can be a constant or a config.

### Implementation Steps
1.  Modify `src/kafka/consumer.js`:
    -   Change `autoCommit` to `false`.
    -   Update the message processing flow to await the `onMessage` handler.
    -   Call `consumer.commit()` explicitly after successful processing.
    -   Add logic to handle errors from `onMessage`. If it's a DB connection error, trigger the pause/retry loop.
2.  Update `src/db/databaseManager.js` (Optional but helpful):
    -   Ensure it exposes a method to check connectivity (e.g., `testConnection()`) for the recovery loop.

## Impact

-   **Reliability**: drastically improved. Zero data loss guarantee for DB outages.
-   **Performance**: Slight overhead due to manual commits (can be batched if needed, but per-message or small batch is safer for now).
-   **Operations**: System will self-recover from DB maintenance or crashes.
