Batch Processing System Template

Use batch when latency of minutes to hours is acceptable, when you need to process the full dataset (not just new events), or when the business logic requires joins across large tables. Use real-time (streaming) when users expect data within seconds, when event ordering matters, or when each event triggers an immediate action. Many systems use both: real-time for alerts and dashboards, batch for aggregations and reports. The [Product Analytics Handbook](/analytics-guide) covers data architecture decisions for product teams.

Small enough to process within your per-partition timeout and fit in worker memory. Large enough to avoid scheduling overhead dominating processing time. For most systems, 10K-500K records per partition works well. Start with a size that gives you 50-200 partitions and adjust based on observed processing times.

Define a "late arrival window" (e.g., 48 hours). Run the main job on the expected data, then run a separate reconciliation job that catches late arrivals within the window. For data arriving after the window, require manual reprocessing. Track late arrival rates as a [data quality metric](/glossary/aarrr-pirate-metrics).

Airflow is the most mature with the largest community and plugin ecosystem. Prefect offers a more Pythonic API and better local development experience. Temporal is strongest for complex workflows with branching and human-in-the-loop steps. For straightforward ETL pipelines, any of the three works. Choose based on your team's existing skills and infrastructure.

Run batch jobs during low-traffic windows. Use read replicas instead of hitting the primary database. Set connection pool limits on batch workers (separate from production pools). Use resource quotas in Kubernetes to prevent batch jobs from consuming compute meant for production workloads. Monitor production latency during batch windows to catch resource contention early. ---