Real-Time and Streaming Analytics

What it is

Event streaming is the practice of capturing data in real time from sources such as databases, sensors and applications as streams of events, storing those streams durably, and processing and reacting to them as they occur or afterwards.

A stream may be unbounded: it has a start but no defined end, and provides data as it is generated. Processing it continuously is different from loading a finished file.

Streaming analytics is the analysis done on those streams: aggregating, joining and detecting patterns over time windows, and feeding the result to a dashboard, an alert or another system.

Why it matters

Some decisions lose their value if they wait for the next batch. The documented uses include processing payments, tracking shipments, analysing sensor data and reacting to customer interactions.

It is not always needed. DII has read no independent evidence of how widely enterprises use streaming for analysis, and the engineering cost is higher than batch.

The problem it addresses

DII editorial view

Batch processing answers a question hours after the event. For a payment, a shipment in transit or a customer interaction in progress, that is too late. Streaming analyses events as they arrive, at a higher engineering cost.

Business and organisational relevance

DII editorial view

The business question is how fresh an answer needs to be and what that freshness is worth. Streaming costs more to build and run than batch, so the case rests on decisions that lose value by waiting.

Practical use cases

• Processing payments and financial transactions in real time
• Tracking vehicles and shipments
• Capturing and analysing sensor data
• Reacting immediately to customer interactions

Benefits and opportunities

• Decisions while the event still matters
• One record of events that several consumers can read

Risks, limitations and dependencies

• Operating streaming systems is harder than operating batch jobs
• Late and out-of-order events complicate correctness
• Real-time is often asked for where a short batch would do

Prerequisites for adoption

DII editorial view

• A decision whose value depends on freshness
• Engineering capacity to operate streaming systems
• A design for late and out-of-order events

Implementation considerations

• Ask how fresh the answer needs to be. If a decision can wait fifteen minutes, a short batch may be simpler and cheaper.
• Decide what happens to late events. Events arrive late and out of order; the handling must be designed, not discovered.
• Govern the stream as a data store. It holds events durably and needs retention, access and personal-data controls.

The DII Radar assessment

Catalogue status

Not yet assessed. DII has not published a Radar assessment for Real-Time and Streaming Analytics. It is listed in the catalogue so that it can be explained. No relevance, direction, strategic impact or enterprise adoption is given for it, and it is not plotted on the Radar.

What to consider next

DII editorial view

• Ask how fresh the answer really needs to be
• Decide what happens to late events
• Govern the stream as a data store, with retention and access controls