Real-Time Data Template for Engineering Teams

Ask: what happens if the data is 60 seconds old instead of 2 seconds old? If the user experience is meaningfully worse (collaborative editing, fraud detection, live bidding), you need streaming. If the data is consumed in dashboards refreshed every few minutes, hourly batch or micro-batch (every 30-60 seconds) is simpler, cheaper, and easier to maintain. Most "real-time" requests can be satisfied with near-real-time batch processing.

At-least-once means every event is delivered at least one time but may be delivered multiple times (duplicates possible). Exactly-once means every event is delivered precisely one time with no duplicates. Exactly-once requires idempotent consumers or transactional processing, which adds latency and complexity. Most systems use at-least-once with consumer-side dedup (e.g. check event ID before processing). True exactly-once is expensive and rarely necessary.

Consumer lag is the delay between when a message is produced and when it is consumed. Monitor lag continuously. Set alert thresholds (e.g. alert if lag exceeds 30 seconds). When lag grows: (1) scale consumers horizontally (add instances or partitions), (2) optimize processing logic to reduce per-event latency, (3) as a last resort, skip to the latest offset and accept data loss for the gap. The [data pipeline specification template](/templates/data-pipeline-spec-template) covers SLA definitions that apply to streaming consumer lag.

Kafka is the most flexible and powerful but requires operational expertise (or Confluent Cloud). Kinesis integrates tightly with AWS and is simpler to operate but has shard-based scaling limits. Google Pub/Sub scales automatically and integrates with GCP services. Redis Streams works for lower-volume use cases where you already run Redis. Choose based on your cloud provider, team expertise, and throughput requirements. For most teams under 10,000 events/second, any of these options works.

Three approaches: (1) Load testing with synthetic events at 2-3x peak rate using tools like Gatling or k6. (2) Chaos testing: kill a consumer instance mid-stream and verify recovery. (3) Latency testing: inject timestamps into events and measure end-to-end delivery time under load. Test failure scenarios (broker down, slow consumer, schema change) before launch. Streaming bugs found in production are significantly harder to fix than batch pipeline bugs. ---