fix: use stable Kafka warmup group ID and offset-aware catch-up (fixes #19)

[jishanahmed-shaikh]

Problem (Closes #19)

The warmupKafkaCache() function in
backend/src/utils/warmupKafka.ts used a time-based consumer group ID on every startup:

groupId: `warmup-consumer-${Date.now()}`

Because Kafka had no record of what this group had already processed, it always started from offset 0 and replayed every message ever produced.

Impact

• Startup time grew linearly with total message history
• ~100K messages → several minutes
• ~1M messages → unusable startup time (timeouts)

Secondary Issue

The consumer stopped after an arbitrary 1-second delay:

await new Promise(r => setTimeout(r, 1000));

This caused:

• ❌ Missed messages under load
• ❌ Wasted time when idle

---

Root Cause

Two independent bugs:

1. Non-stable group ID

• A new group ID on every restart
• Kafka cannot commit offsets
• Always replays from the beginning

2. Arbitrary sleep as stop condition

• No guarantee all messages are consumed
• Not tied to actual Kafka state

---

Fix

1. Stable Consumer Group ID

// Before
groupId: `warmup-consumer-${Date.now()}`

// After
groupId: 'ascent-warmup-stable'

Behavior

• First run → same as fromBeginning: true
• Subsequent runs → resume from last committed offset
• Only new messages are processed

---

2. Offset-aware Catch-up using Admin API

Before starting the consumer, fetch the latest offsets (high-water mark) for each partition:

const latestOffsets = await fetchLatestOffsets(admin);

Inside eachMessage, track catch-up:

if (currentOffset >= latestOffset - 1n) {
  caughtUp.add(key);
}

if (caughtUp.size === latestOffsets.size) {
  resolve(); // all partitions caught up
}

Key Improvements

• Deterministic completion
• No arbitrary delays
• Works under load and idle conditions

---

Additional Notes

• Partitions with no messages (offset 0) are skipped
• autoCommit: true ensures durability across restarts/crashes

---

Behaviour Comparison

Scenario	Before	After
First startup	Replays all messages	Replays all messages (identical)
Restart with no new messages	Replays all messages again	Exits immediately
Restart with 100 new messages	Replays all messages	Replays only 100 new messages
1M messages in Kafka	30+ min startup	Seconds
Fresh cluster (no messages)	1s sleep then done	Exits immediately
Stop condition	Arbitrary 1s sleep	Deterministic

---

Files Changed

• backend/src/utils/warmupKafka.ts - full rewrite of warmup logic