Databricks Feature Store 如何实现亚秒级特征新鲜度。

Machine learning models are only as good as the signals they receive. A fraud detection use case must decide in milliseconds within a user pressing purchase whether to allow the transaction. Making the right call depends on seeing a suspicious transaction happening only seconds ago. Combining a user’s average transactions for the last 30 days along with the total transaction amount from the last 10 minutes highlights the potential fraud. The long-range aggregations set a baseline profile of the user to determine what is normal, while the most recent data helps surface any abnormal behavior right as it’s happening. Personalization faces the same pressure: the freshest signals are what capture a user's current intent and drive engagement.
Spark pipelines are an established way to process bulk data in the Lakehouse for historic baseline features. Running these batch jobs on a regular schedule is well understood, but introduces minutes to hours of lag. For baseline signals about users, this lag is an acceptable price for simpler infrastructure. When models require fresh signals, this infrastructure breaks down; getting down to seconds or milliseconds is not possible in existing feature store platforms. To deliver the value of fresh features, data scientists are forced to implement complex, streaming specific logic to handle these aggregations and stand up custom hosted infrastructure.
Databricks Feature Store lets you author a feature once and use it everywhere: the same definition drives large-scale batch flows offline and highly fresh feature pipelines online. The framework removes the infrastructure burden, orchestrating Spark Real-Time Mode (RTM) for continuous stream processing, Lakebase for streaming-optimized online storage, and Model Serving for retrieval at scale. And once authored, that feature is served in milliseconds: end-to-end p99 latency of 200ms, from an event arriving in Kafka to availability in the online feature store.
Architecture: Kafka to Feature Store in 200ms
Let’s take a look under the hood to see how the Databricks Feature Store takes an infrastructure agnostic Feature definition and builds a pipeline to consistently compute it within milliseconds. The end-to-end path for a streaming feature looks like this:

- Events land in Kafka - raw data like credit card transactions, ad impressions, or clickstream events

- A Spark RTM pipeline on serverless Lakeflow Spark Delta Pipelines continuously processes events, computing rolling aggregations in real time

- Updated aggregates are written to Lakebase via a new streaming JDBC sink, landing in the online feature store

- Model Serving endpoints retrieve the latest features from Lakebase at inference time, feeding them into the model automatically
Let’s tie this to our fraud feature, the sum of a user’s transaction amount over the last 10 minutes. Each incoming event carries the transaction details - amount, location, user id, merchant information - and is routed to a stateful pipeline. The pipeline consults a local RocksDB instance holding the user's running transaction total, with expiry times that keep the window to the last 10 minutes. The pipeline reads and increments the value locally, then writes the updated feature value to Lakebase. So when a query comes to the model to approve a new transaction, an up-to-date transaction sum is available with sub-second freshness in the feature store. This sum feature will be fetched along with the user’s historic purchasing baseline to inform approval. A sum well above the historic baseline is a strong indicator to the model of potential fraud.
Each component in this pipeline has been optimized so incoming events are routed, aggregations are calculated, and features are written to the online store as quickly as possible.
Rolling window: update aggregations in milliseconds
Before going deeper into the infrastructure, let’s talk about aggregation features and shifting from a paradigm of batch sync to real-time updates.
Aggregation features over a time window - for example counts, sums, or averages - are powerful and flexible signals for real-time ML. A long term batch feature sets a historical baseline for the user over a period of time which allows the model to adapt and understand behavior of each user. A short, fresh feature reacts quickly to changing situations to distinguish new user interest or fraudulent activity. Time windows define a time range (e.g. 10 minutes) as well as how those time ranges should evolve over time (e.g. overlap or disjoint).
Databricks Feature Store supports 3 different time windows:

- Tumbling windows are aligned to wall-clock intervals and begin as soon as the last interval ends. A 10-minute tumbling window might cover 12:00–12:10, then 12:10–12:20. Events are batched into these fixed intervals with a feature value being emitted at the end of an interval. This means the aggregate is only fresh at interval boundaries

- Sliding windows are also aligned to wall-clock intervals but allow for overlap in intervals. A 10-minute sliding window with a 5 minute slide interval might cover 12:00–12:10, then 12:05–12:15, and then 12:10-12:20.

- Rolling windows are not aligned to wall-clock but look backward from each event's timestamp with millisecond resolution. "The sum of transactions in the last 10 minutes as of the current wall-clock" is always up to date, because the window moves with each new event. This makes RollingWindow the natural fit for real-time serving where "now" is always changing.
Tumbling and sliding windows remain useful when a feature doesn’t change frequently: they emit fewer updates, are cheaper to maintain, and fit naturally into simpler scheduled pipelines. Rolling windows trade that efficiency for maximum freshness, which is most valuable for signals where every new event should immediately affect the value served to the model.
Here's how simple it is to define a rolling window feature with the Feature Store declarative API: