ACE Journal

Streaming Feature Stores for Real-Time ML

Who this is for: ML engineers and data platform engineers who already run a model behind a real-time inference service and a batch feature pipeline, and are hitting the freshness ceiling. The assumption is that you know what a feature is, you have a model in production, and you have at least one feature that is now too stale to be useful. You do not yet have a stream processor wired into your feature serving path. If that is you, this is the architecture that turns “yesterday’s aggregate” into “the last few seconds.”

The feature that was already too old

A fraud team ships a model that scores card transactions. In offline evaluation it is excellent. In production it misses a class of fraud that the offline data clearly contained: rapid bursts of small charges across merchants in the span of a minute. The post-mortem finds the cause, and it is not the model. The feature txn_count_last_60s was materialized by a batch job that ran every six hours. At inference time the model read a value that was, on average, three hours stale. The burst pattern the feature was designed to catch had always already finished by the time the feature reflected it.

This is the failure that pushes teams from batch to streaming feature stores. A batch pipeline computes features on a schedule, and the online model reads whatever the last run produced. That is fine for slow-moving features such as a customer’s lifetime order count, and fatal for features whose entire predictive value lives in a window measured in seconds. Freshness is capped by the cadence of the job, and no amount of model tuning recovers signal the feature never carried.

The fix is to stop computing on a schedule and start computing on arrival. A stream processor consumes the event the moment it lands, updates the affected feature values, and pushes them to a store the model can read in single-digit milliseconds. The architectural cost of that shift is the subject of this article, because making features fresh is the easy half. The hard half is making the fresh value you serve and the historical value you train on agree.

The dual-store architecture

Strip a streaming feature store down and it is two storage tiers fed by one transformation layer, coordinated by a registry that defines what every feature means.

Figure 1 shows the shape. Raw events flow into a stream processor, which applies feature transformations and fans the results out to both stores.

flowchart LR
    Src[event sources<br/>Kafka / Kinesis] --> SP[stream processor<br/>Flink / Spark SS]
    Reg[(feature registry)] -. defines transforms .-> SP
    SP -->|synchronous, low latency| Online[(online store<br/>Redis / DynamoDB)]
    SP -->|async, batched| Offline[(offline store<br/>warehouse / object store)]
    Online --> Infer[inference service]
    Offline --> Train[training + backfill]

Figure 1. The dual-store architecture of a streaming feature store. Note the two write paths out of the stream processor: the synchronous online write minimizes serving staleness, while the asynchronous batched offline write feeds training, and the latency gap between them is the source of the skew problem covered later.

The online store is built for point lookups. Redis, DynamoDB, and Bigtable are the common choices, and the requirement is a sub-millisecond read of “the current feature vector for entity X” at inference time. It holds only the latest value per entity and feature; it is not a history.

The offline store is built for scans. A columnar format such as Parquet on object storage, or a data warehouse, holds the full timestamped history of every feature value. Training reads from here, because training needs not the current value but the value as it was at some point in the past, for every label in the dataset.

The stream processor is the engine. Flink, Spark Structured Streaming, and Kafka Streams all fit. It reads raw events from a log such as Kafka or Kinesis, applies the windowed aggregations and transformations the registry defines, and writes computed values to both stores. The two writes have deliberately different characteristics. The online write is synchronous and latency-sensitive, because every millisecond it lags is a millisecond of staleness the model inherits. The offline write is asynchronous and batched, because training tolerates minutes of delay and batching is cheaper.

The feature registry is what keeps the two paths describing the same feature. It is the single declaration of what txn_count_last_60s means: its entity, its data type, its source, and the transformation that produces it. Both the streaming path and any backfill job read the same definition, which is the only thing that makes the online and offline values comparable at all.

Worked example: a windowed feature, defined and served

Make it concrete. Below is a feature definition in Feast, the open-source feature store, declaring a streaming feature view for transaction counts keyed on a card entity. The definition is what both the online serving path and the offline training path resolve against.

from datetime import timedelta
from feast import Entity, FeatureView, Field, PushSource
from feast.types import Int64

card = Entity(name="card", join_keys=["card_id"])

txn_push = PushSource(
    name="txn_push_source",
    batch_source=txn_parquet_source,  # offline backfill source
)

txn_features = FeatureView(
    name="txn_features",
    entities=[card],
    ttl=timedelta(minutes=10),
    schema=[Field(name="txn_count_last_60s", dtype=Int64)],
    online=True,
    source=txn_push,
)

The PushSource is the streaming entry point. It says values for this view can be pushed in from an external stream processor rather than only materialized from batch. The ttl of 10 minutes bounds how long a value stays servable before it is considered expired, which matters because an online store holds only the latest write and a card that stops transacting should not keep returning a stale count forever.

The stream processor computes the aggregate and pushes it. Here is the push side, the part that runs for every micro-batch the processor emits:

import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path=".")

def push_batch(agg_df: pd.DataFrame) -> None:
    # agg_df columns: card_id, txn_count_last_60s, event_timestamp
    store.push(
        push_source_name="txn_push_source",
        df=agg_df,
        to=store.PushMode.ONLINE_AND_OFFLINE,
    )

PushMode.ONLINE_AND_OFFLINE is the design decision that earns its place. It writes the same computed value to both tiers in one call, which is what gives you any hope of the online and offline records agreeing later. The event_timestamp column travels with the value, and that timestamp is not bookkeeping. It is the join key for correctness, the thing that lets a training job ask “what was this feature at this exact moment” instead of “what is this feature now.” Tecton’s stream ingest API exposes the same shape under a managed runtime: you hand it a record with an event timestamp and it owns the dual write.

At inference time the read is trivial by comparison, a point lookup against the online store:

features = store.get_online_features(
    features=["txn_features:txn_count_last_60s"],
    entity_rows=[{"card_id": "card_8842"}],
).to_dict()
score = model.predict(features)

The asymmetry is the whole point. Writes are continuous, windowed, and fan out to two stores; reads are a single fast key lookup. The complexity lives in the write path so the serving path can stay simple and fast.

The hard part: training-serving skew is a timestamp problem

The non-obvious thing, the lesson you only learn after a model underperforms in production for reasons that do not show up in offline metrics, is this: the value you serve and the value you train on are computed from different event windows, and reconciling them is a timestamp problem, not a storage problem.

Here is why they diverge. Events arrive continuously and out of order. A transaction that physically occurred at 12:00:03 might reach the stream processor at 12:00:05 because of network and broker delay. At inference time, at 12:00:04, the model reads a txn_count_last_60s that does not yet include that late event, because it has not arrived. Later, when the offline store materializes the window for training, the late event is present, and the offline value for that same instant is higher. The model is now trained on a version of history that was never actually served. Offline evaluation looks great precisely because offline data is the clean, complete, after-the-fact version. Production is worse because production saw the incomplete, in-flight version. That divergence is training-serving skew, and the cleaner your offline data looks, the more dangerous it is.

Figure 2 traces a single late event through the two paths to show exactly where the two values part.

sequenceDiagram
    participant E as event (occurs 12:00:03)
    participant SP as stream processor
    participant On as online store
    participant Off as offline store
    participant M as model (infers 12:00:04)
    M->>On: read txn_count_last_60s
    On-->>M: value WITHOUT late event
    E->>SP: arrives late at 12:00:05
    SP->>On: update (now includes event)
    SP->>Off: materialize window (includes event)
    Note over Off,M: offline value != value model saw

Figure 2. How a late-arriving event creates training-serving skew. Watch the order: the model reads at 12:00:04, before the 12:00:03 event arrives at the processor, so the offline window later materialized for training contains an event the served value never did.

You do not fix this by making the stores faster or more consistent. You fix it by logging what was actually served and treating that log as ground truth for training. The pattern has two halves. First, at inference time, log the exact feature vector the model consumed along with the request timestamp, not a value re-read later. Second, generate training data not by re-aggregating raw events, which reproduces the clean offline version, but by joining labels against those served-feature logs. The feature value in your training set then is, by construction, the value the model saw.

# log the served vector at request time, not after the fact
served = store.get_online_features(
    features=["txn_features:txn_count_last_60s"],
    entity_rows=[{"card_id": card_id}],
).to_dict()

feature_log.write({
    "card_id": card_id,
    "request_ts": now,           # the instant the model decided
    "txn_count_last_60s": served["txn_count_last_60s"][0],
    "model_version": MODEL_VERSION,
})

This served-feature log is the single most important artifact in a streaming feature store, and it is the one teams most often skip because the system appears to work without it. It works right up until you retrain on offline data and silently widen the gap between your evaluation and your production reality. Build the log before you build the second model, not after the first one disappoints.

Backfill and point-in-time joins

The other place the timestamp does the heavy lifting is generating the initial training set, before you have accumulated enough served-feature logs to train from. This needs a point-in-time correct join: for each labeled example, retrieve the feature value as of the label’s timestamp, never the current value.

The reason this is not a plain join is leakage. A naive join of labels to the latest feature value attaches information from after the label was generated, the model trains on the future, and offline metrics become a fantasy. A point-in-time join instead asks, for each label at time T, for the most recent feature value whose event_timestamp is at or before T.

entity_df = pd.DataFrame({
    "card_id": ["card_8842", "card_3310"],
    "event_timestamp": ["2026-05-09 12:00:04", "2026-05-09 12:01:17"],
    "is_fraud": [1, 0],   # the labels
})

training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["txn_features:txn_count_last_60s"],
).to_df()

get_historical_features runs the point-in-time join against the offline store, using the event_timestamp recorded with every value. For each row it finds the feature value that would have been current at that row’s timestamp, respecting the view’s ttl. Both Feast and Tecton implement this against the offline history, and Tecton additionally enforces the same window definition across the streaming and historical paths so the backfilled value matches the streaming logic by construction.

The correctness of this join is bounded by offline materialization latency. If the offline store lags the online store by several minutes, then for recent labels the most-recent-at-or-before value may simply not be in the offline store yet, and the join returns a staler value than the one served. The practical consequence: point-in-time joins are trustworthy for labels older than your offline lag and increasingly imprecise as labels approach the present. Set an explicit materialization-latency SLA, validate it in system testing, and treat very recent labels with suspicion until the offline store has caught up.

Integration into a real workflow

In day-to-day operation the feature store sits between three teams that otherwise speak different languages. Data engineers own the stream processor and the event sources. ML engineers own the feature definitions in the registry and the model that reads them. The platform owns the two stores and their SLAs. The registry is the contract that lets these three move independently: a feature definition checked into version control, reviewed like code, is what stops the streaming aggregation and the training-time backfill from drifting apart.

The daily loop looks like this. A feature is defined once in the registry and applied (feast apply), which provisions the online and offline schemas. The stream processor picks up the definition and begins pushing live values. Training jobs pull point-in-time-correct datasets via get_historical_features, and once the model ships, the inference service reads via get_online_features and writes the served-feature log. Monitoring closes the loop by comparing logged served values against materialized offline values for the same entity and window, which is the only direct measurement of skew you will get.

That skew monitor is the integration step teams underbuild. It ships no feature, but it is the smoke detector for the exact failure that does not show up in offline evaluation. A rising divergence between served and materialized values means your online and offline paths have drifted, usually because someone changed an aggregation on one side and not the other, and catching it in a dashboard is far cheaper than catching it in a degraded model weeks later.

The streaming feature store space sorts along one main axis: how much of the transformation layer the platform owns versus how much you run yourself.

Feast is the open-source, bring-your-own-compute end. It is deliberately agnostic about how features are computed: it ingests pre-computed values through its push API from whatever stream processor you already run, and it owns the registry, the dual-store coordination, and the point-in-time join. That is the right trade for a team already operating Flink or Spark, because it adds a thin, inspectable layer over infrastructure you control rather than a second compute engine. The cost is that the transformation logic, and therefore the consistency between streaming and backfill aggregations, is your responsibility to keep aligned.

Tecton is the managed, owns-the-transformation end. You declare features and Tecton runs the streaming and batch computation, the dual write, and the historical join behind one SDK. That collapses the number of moving parts and removes the most common source of skew, two separately maintained aggregations, by generating both from a single definition. The trade is the usual managed-platform trade: you couple to the SDK and runtime, and you give up direct control over the compute. For a team that wants real-time features without standing up and babysitting a stream processor, that is often worth it; for a team with an existing, heavily customized streaming stack, the coupling bites.

Roll-your-own on Kafka Streams or Flink plus Redis is the third option, more common than the vendor framing suggests. It is the right call when your feature logic is simple enough that a feature-store abstraction adds more concept than it removes, and the wrong call the moment you need point-in-time-correct training data, because that is precisely the hard, easy-to-get-subtly-wrong machinery the platforms exist to provide. Many teams arrive at a feature store only after building this by hand and discovering that skew is harder than the serving problem they set out to solve.

None of these removes the underlying discipline. The registry still has to be the single source of truth, the served-feature log still has to be built, and the materialization SLA still has to be validated. The tool decides who operates the stream processor; it does not decide whether you have to think about timestamps.

Where it breaks

Streaming feature stores fail in recognizable ways, and most production incidents trace to one of these.

Silent training-serving skew. The headline failure, covered above: serving from the online store while training on re-aggregated offline data, with no served-feature log to reconcile them. The model evaluates well and underperforms in production, and because nothing errors, the gap can persist for months. The fix is the served-feature log and an explicit skew monitor, built before the second model, not after the first disappoints.

Online store staleness from a slow processor. The online value is only as fresh as the processor’s end-to-end latency. Backpressure, an undersized Flink cluster, or a slow online write turns “last 60 seconds” into “last several minutes” with no error surfacing, because the store returns whatever was last written. The freshness gain that justified the whole architecture quietly evaporates. Monitor processing lag as a first-class SLA.

Out-of-order and late events handled implicitly. If the windowing does not explicitly account for event-time versus processing-time and bound lateness with watermarks, late events either get dropped, undercounting the feature, or get folded into the wrong window, corrupting it. A streaming feature whose window semantics are left to defaults is a feature whose values you cannot reason about. Define event-time windows and a watermark policy deliberately.

TTL mismatch with feature semantics. The online store holds only the latest value, governed by a TTL. Set it too long and a dormant entity keeps returning a stale aggregate as if it were current; too short and an entity with sparse-but-real activity returns nulls the model is not built to expect. The TTL must match what the feature actually means, and it is routinely set once and forgotten.

Registry drift between paths. When the streaming aggregation and the backfill aggregation are maintained as two separate pieces of code, they drift. Someone changes the window on the streaming side, forgets the batch side, and now historical training and live serving compute subtly different things, which is training-serving skew reintroduced through the back door. A single registry-driven definition, or a managed platform that generates both, is the structural defense; code review of feature definitions is the human one.

Offline lag breaking recent point-in-time joins. As covered in the backfill section, point-in-time joins are only as correct as the offline store is current. Train on very recent labels while the offline store still lags, and the join hands you values staler than what was served, corrupting exactly the freshest, often most valuable, examples. Respect the materialization SLA and hold recent labels out until the offline store has caught up.

Further reading

Start with the Feast documentation on its online and offline store architecture and its push-based streaming ingestion [1][2], which together establish the dual-store model and the registry contract this article is built on. The Feast point-in-time join and training data retrieval guide [3] is the canonical source for the leakage-safe join. For the managed, owns-the-transformation approach, the Tecton documentation on stream ingestion [4] and on online-offline consistency [5] cover how a single definition drives both paths. On the stream-processing engines underneath, the Apache Flink event-time and watermarks documentation [6] and the Spark Structured Streaming programming guide [7] are the references for the windowing and lateness semantics that the “where it breaks” section depends on. The Uber Michelangelo engineering write-up [8] remains the clearest industrial account of why training-serving skew is the central problem, and Chip Huyen’s “Designing Machine Learning Systems” [9] places feature stores in the broader real-time ML context. For the online store choices, the Redis [10] and DynamoDB [11] documentation cover the low-latency lookup requirements.

Written with AI assistance, editorially reviewed.

References

[1] Feast, “Feast Architecture Overview.” https://docs.feast.dev/getting-started/architecture-and-components/overview

[2] Feast, “Push Source and Streaming Ingestion.” https://docs.feast.dev/reference/data-sources/push

[3] Feast, “Getting Training Features (Point-in-Time Joins).” https://docs.feast.dev/getting-started/concepts/feature-retrieval

[4] Tecton, “Stream Ingest API.” https://docs.tecton.ai/docs/defining-features/feature-views/stream-feature-view

[5] Tecton, “Online and Offline Consistency.” https://docs.tecton.ai/docs/monitoring/production-readiness

[6] Apache Flink, “Event Time and Watermarks.” https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/

[7] Apache Spark, “Structured Streaming Programming Guide.” https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

[8] Uber Engineering, “Meet Michelangelo: Uber’s Machine Learning Platform.” https://www.uber.com/blog/michelangelo-machine-learning-platform/

[9] C. Huyen, “Designing Machine Learning Systems.” https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/

[10] Redis, “Redis as a Feature Store / Low-Latency Data Store.” https://redis.io/docs/latest/develop/

[11] Amazon, “Amazon DynamoDB Developer Guide.” https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html