Real-time data pipeline: ingest, process and act on events
A real-time pipeline handles each event within seconds of it happening, instead of waiting for tonight's batch. Here is how the pieces fit, which service plays each part on AWS, Azure and Google Cloud, and what to settle first.
Quick answer
A real-time data pipeline publishes events from apps, devices or databases to a streaming service. A stream processor then filters, enriches and aggregates them within seconds, using time windows. Results go to dashboards and alerts, while a raw copy lands in a data lake for replay and history.
| Best for | Fraud checks, live dashboards, clickstream analysis and monitoring, where seconds matter |
|---|---|
| Less suited when | Daily or hourly reports are fast enough, so a scheduled batch job is simpler and cheaper |
| Start with | One event source, one stream, one windowed count and one dashboard, then add a raw archive |
| Main services on AWS | Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, AWS Lambda, Amazon S3 + AWS Lake Formation, Amazon Quick Sight, Amazon SNS |
| Main services on Azure | Azure Event Hubs, Azure Stream Analytics, Azure Functions, Microsoft Fabric OneLake, Power BI, Azure Service Bus topics |
| Main services on Google Cloud | Pub/Sub, Dataflow, Cloud Run functions, BigQuery lakehouse, Looker |
The idea in plain words
Batch pipelines collect data and process it on a schedule, so answers are hours old. A streaming pipeline treats each change as an event, a small record of something that happened, and handles it within seconds. That makes it possible to block a fraudulent payment, update a live dashboard or alert on a failing machine while it still matters.
Events enter a streaming service that holds them in order for a set time, so several readers can consume them at their own pace. A stream processor reads the events, filters and enriches them and computes figures over time windows, such as orders per minute. The results go to dashboards, alerts and storage, and a raw copy is kept so you can replay events after a mistake.
Streaming adds moving parts that run all day and must cope with late, duplicated and out-of-order events. Choose it only for the questions that really need fresh answers, and keep the rest in batch.
The architecture
Boxes show the job each part does. Pick a cloud to see the service for each job.
Step by step
- Events are producedAn app, a device or a database change creates a small record of what happened, such as a click, a payment or a temperature reading, with a time and an identifier.
- The stream takes them inA streaming service accepts events from many producers, spreads them across partitions for speed and keeps them in order for a set time, so slow readers can catch up.
- A raw copy is savedThe same events are written to a data lake, unchanged. That gives you history for reports, and a way to replay events when a rule or a bug needs correcting.
- The processor works on the flowA stream processor filters out noise, adds reference data such as customer details, and calculates figures over time windows, for example payments per minute in each country.
- Late and repeated events are handledThe processor waits a short, chosen time for late events and drops or merges duplicates, so the same event never counts twice.
- Results are usedFresh figures go to dashboards or to a fast store behind your app. When a rule fires, a function runs your code, and a notification service tells people and other systems.
- Everything is watchedMonitor how far readers lag behind the newest event, the error rate and the queue of failed events, and alert before users notice delays.
Services side by side
The main service for each job on each cloud, with a second option where one exists. Names checked on 23 October 2026.
| Job | AWS | Azure | Google Cloud |
|---|---|---|---|
| Real-time data streams (ingest events at scale) |
|
|
|
| Real-time stream processing and analytics |
|
|
|
| Serverless functions |
|
|
|
| Data lake and lakehouse |
|
|
|
| Dashboards and business intelligence |
|
|
|
| Publish/subscribe and fan-out notifications |
|
|
|
Some streaming services can write events straight to storage or a warehouse without extra code, which covers the raw copy and simple loads. Add a stream processor when you need joins, time windows or rules.
Common variations
- You need exact, one-time results. Use a processor and destinations that support exactly-once processing, and make your own writes safe to repeat. Test by stopping and restarting jobs on purpose.
- Events come from existing databases. Use change data capture, which turns each insert, update and delete into an event, instead of polling tables on a timer.
- Many teams read the same events. Give each team its own consumer group or subscription, and publish a schema so producers can change events without breaking readers.
- Quiet periods then big bursts. Use a service that scales on demand, or add partitions before known peaks, and let the stream absorb the burst while processors catch up.
Cloud-native stream service or managed Kafka?
Each cloud has its own streaming service, and each also offers Apache Kafka, the open-source standard, as a managed service or a Kafka-compatible endpoint, shown in the table.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
| Managed Apache Kafka |
|
|
|
Cloud-native stream service
Good
- Simplest to set up and scale
- Tight links to the cloud's other services
- Little to tune or patch
Watch out
- Its own interfaces, so moving away takes work
- Limits on message size, retention and throughput
- Fewer tools from the Kafka ecosystem
Managed Apache Kafka
Good
- Open standard with a large ecosystem of tools and skills
- Easier to move between clouds or run on-premises
- Rich options for replay and many independent readers
Watch out
- More settings to learn and tune
- Costs more at small scale
- Still needs capacity and partition planning
Keep in mind
- Do you really need real time?. Streaming costs more to build and run than a scheduled load. If a decision can wait an hour, a batch job is simpler. Ask what a minute earlier is worth before you commit.
- Ordering, duplicates and late data. Events can arrive twice, out of order or late. Design every step to be safe to repeat, put an event id on each record and choose how long you will wait for stragglers.
- Schemas will change. Producers change their events over time. Agree a schema, add new fields without removing old ones, and keep a registry so readers do not break.
- Failed events need a home. Send events that cannot be processed to a separate queue instead of dropping them or blocking the stream. Alert when it grows, fix the cause and replay them.
- Sizing and cost. Cost follows throughput, retention and how long processors run, and they run all the time. Set retention to what replay needs, shut down test streams and set budget alerts.
- Security and privacy. Encrypt events in transit and at rest, limit who can publish or read each stream, and keep personal data out of event payloads where you can. The GDPR, the DPDP Act and similar laws still apply to data in motion.
- Test with real traffic. Replay a day of recorded events through the pipeline before launch. It shows speed, lag and cost better than any estimate, and it exposes hidden duplicates.
Words used on this page
Services in this pattern that have changed
Cloud providers rename and retire services often. These affect the parts above. Checked on 23 October 2026.
Sources and further reading
Check details such as features, limits, prices and regions in the providers' own documentation.
- AWS documentation (opens in a new tab)
- Microsoft Azure documentation (opens in a new tab)
- Google Cloud documentation (opens in a new tab)
Announcements about the changed services above
Frequently asked questions
What is a real-time data pipeline?
A real-time data pipeline moves and processes data within seconds of it being created, instead of in scheduled batches. Events flow through a streaming service to a processor, then to dashboards, alerts and storage, so decisions can use data that is only seconds old.
What is the difference between batch and stream processing?
Batch processing collects data and handles it in large chunks on a schedule, so results are hours old. Stream processing handles each event as it arrives, so results are seconds old. Batch is simpler and cheaper, and streaming suits cases where fresher data changes the decision.
Do I need Apache Kafka?
No. Every major cloud has its own streaming service that does the job without Kafka. Choose Kafka when your team knows it, when you need its ecosystem, or when you want the option of moving between clouds or running it yourself.
What is a time window in stream processing?
A time window groups events by time so you can count or average them, for example orders per minute. Windows can be fixed and separate, overlapping, or based on gaps in activity. The window length decides how fresh and how steady the figures are.
How is latency measured in a streaming pipeline?
Measure it end to end, from when an event happens to when its result is visible. Also watch consumer lag, the gap between the newest event and the one a reader has handled. Rising lag means a processor cannot keep up and needs more capacity.
Want this reviewed for your case?
CloudBrum architects can adapt this pattern to your documents, users, rules and budget, and estimate the cost. Nothing is sent until you choose.
Building something else?
Describe it in plain words and get matching services on all three clouds.
Service suggester → Cloud consulting →