Real-time data sounds unambiguously better than data that is an hour old. That intuition leads a lot of teams to build streaming architectures they do not need — and to pay for that decision every day in complexity and operational load. Streaming is a powerful tool. It is also one of the easiest to over-apply.

This article is a practical test for when streaming genuinely earns its cost, and when a well-built batch pipeline is the better engineering decision.

What streaming actually costs

A batch pipeline runs on a schedule, processes a defined set of data, and either succeeds or fails in a way that is easy to see and rerun. A streaming pipeline runs continuously and processes events as they arrive. That continuous nature brings real, recurring costs.

  • Operational complexity — a system that runs continuously must handle late events, out-of-order events, duplicates, and partial failures without a tidy start and stop.
  • Harder debugging — there is no single run to inspect. Reproducing an issue in a flowing system is meaningfully harder than rerunning a batch job.
  • Continuous infrastructure — streaming systems consume resources around the clock, where batch consumes them only while running.
  • Scarcer skills — fewer engineers are fluent in streaming systems, so building and operating one is a larger organisational commitment.

None of this means streaming is wrong. It means streaming should be a deliberate choice that buys something worth more than these costs.

Streaming should buy something worth more than its costs — not just a smaller number on a latency chart.

The test: does latency change a decision?

Here is the question that cuts through it. Between the moment data is created and the moment someone or something acts on it — does the delay change what action is taken?

If a fraud signal arrives while a transaction can still be blocked, latency changes the outcome. If a sensor reading arrives in time to halt a machine before damage, latency changes the outcome. If a recommendation reflects what a user did seconds ago rather than yesterday, latency may change the outcome.

But if the data feeds a dashboard reviewed each morning, an hour of latency changes nothing. If it feeds a model retrained weekly, freshness within the day is irrelevant. If a human acts on the information once a day, sub-second delivery is invisible to them. In all of those cases, streaming buys a smaller number on a latency chart and no actual outcome.

A straightforward way to decide

We work through four questions with clients before committing to an architecture.

  1. What is the slowest acceptable delay? Name the real tolerance — seconds, minutes, hours. Be honest; the answer is often far more generous than the first instinct.
  2. Who or what consumes the data, and how often do they act? Data delivered faster than its consumer acts is wasted freshness.
  3. Does a faster decision produce measurable value? If you cannot point to an outcome that improves, streaming is buying latency for its own sake.
  4. Can the team operate a streaming system well? A streaming pipeline that is unreliable because nobody can support it is worse than a dependable batch one.

If the honest answers point to genuine value from low latency and a team able to run the system, streaming is the right call and worth doing properly. If they do not, a well-built batch pipeline will be more reliable, cheaper, and easier to live with.

Build for the requirement, not the trend

The best data architecture is the simplest one that meets the actual requirement. Sometimes that is streaming, and when it is, the value is real. Often it is a solid batch pipeline that does its job quietly and rarely pages anyone. Choosing between them on the merits — not on which sounds more modern — is one of the higher-leverage decisions in a data platform.