What is a feature store and why do I need one?

A feature store is a centralized system for managing ML features, the input variables your models consume. It handles storage (online for low-latency serving, offline for training), transformation (computing features from raw data), and consistency (ensuring training and serving use identical feature logic). You need one when feature duplication across teams, training-serving skew, or feature freshness becomes a bottleneck.

How do point-in-time joins prevent data leakage?

Point-in-time joins retrieve feature values as they existed at the time of each training example, not as they exist now. If you're training a fraud detector on a transaction from January 15th, the join fetches the user's average spend as of January 15th, not today's value, which includes months of data that wouldn't have been available at prediction time.

Is Feast production-ready for real-time serving?

Yes, with the right online store backend. Feast with Redis serves features at sub-5ms p99 latency for moderate feature set sizes. For very high throughput (100K+ requests/second), you'll want to tune Redis cluster configuration or consider DynamoDB as the online store. The main limitation is that Feast's streaming ingestion requires you to manage the Spark/Flink job yourself.

MLOps Feature Stores: Feast, Tecton, Real-Time Features, and Production ML Pipeline Architecture

The Core Problem

ML models consume features, derived variables computed from raw data. A simple recommendation model might need:

user_avg_session_duration_7d, from clickstream
user_purchase_count_30d, from transactions
item_view_count_24h, from event stream
user_item_category_affinity, from purchase history

Each of these has to be computed in at least two contexts:

Training time: from historical data, with correct point-in-time semantics
Serving time: from live data, at low latency (usually <50ms)

If the computation logic differs between these two contexts, even slightly, you get training-serving skew, and your model performs worse in production than in evaluation. This is not a theoretical concern. A 15% accuracy drop from a date timezone mismatch between a Spark batch job and a Python serving function is a documented pattern.

Feast: The Open-Source Foundation

Feast (Feature Store) is the standard open-source option. It handles feature registration, offline/online storage, and materialization (copying features from offline to online store).

Architecture

 ┌─────────────┐
 │ Feature │
 │ Repository │
 │ (Python) │
 └──────┬──────┘
 │
 ┌────────────┼────────────┐
 │ │ │
 ┌────────▼───┐ ┌────▼─────┐ ┌──▼──────────┐
 │ Offline │ │ Online │ │ Feature │
 │ Store │ │ Store │ │ Server │
 │ (BigQuery/ │ │ (Redis/ │ │ (gRPC/HTTP) │
 │ Snowflake/ │ │ DynamoDB)│ │ │
 │ S3+Parquet)│ │ │ │ │
 └────────────┘ └──────────┘ └─────────────┘

Defining Features

# feature_repo/features.py
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

user = Entity(name="user_id", join_keys=["user_id"])

user_transaction_stats = FeatureView(
 name="user_transaction_stats",
 entities=[user],
 schema=[
 Field(name="avg_transaction_amount_30d", dtype=Float32),
 Field(name="transaction_count_30d", dtype=Int64),
 Field(name="max_transaction_amount_30d", dtype=Float32),
 Field(name="days_since_last_transaction", dtype=Int64),
 ],
 source=FileSource(
 path="s3://ml-features/user_transaction_stats.parquet",
 timestamp_field="event_timestamp",
 ),
 ttl=timedelta(days=1),
 online=True,
)

Point-in-Time Joins

This is where feature stores earn their keep. When you fetch training data, you need features as they existed at the time of each training example:

from feast import FeatureStore
import pandas as pd

store = FeatureStore(repo_path="feature_repo/")

# Training entity dataframe, each row is a training example
# with the timestamp at which we need features
entity_df = pd.DataFrame({
 "user_id": [1001, 1002, 1001, 1003],
 "event_timestamp": [
 "2025-11-15 10:30:00",
 "2025-11-15 14:00:00",
 "2025-12-01 09:00:00", # Same user, different time
 "2025-12-02 16:45:00",
 ],
})

# Feast joins features with correct temporal semantics
training_df = store.get_historical_features(
 entity_df=entity_df,
 features=[
 "user_transaction_stats:avg_transaction_amount_30d",
 "user_transaction_stats:transaction_count_30d",
 "user_session_stats:avg_session_duration_7d",
 ],
).to_df()

The event_timestamp column is the key. For user 1001 on November 15th, Feast returns the feature values computed from data available up to November 15th. For the same user on December 1st, it returns different values, reflecting the additional two weeks of data. Without point-in-time joins, you'd use the latest feature values and unknowingly leak future information into training.

Online Serving

# Real-time feature retrieval for inference
features = store.get_online_features(
 features=[
 "user_transaction_stats:avg_transaction_amount_30d",
 "user_transaction_stats:transaction_count_30d",
 ],
 entity_rows=[{"user_id": 1001}],
).to_dict()

# Returns: {'avg_transaction_amount_30d': [142.50], 'transaction_count_30d': [23]}

With Redis as the online store, this returns in under 5ms at p99. Benchmarks show 2,000 requests/second on a single Feast server instance with 50 features per request.

Materialization Pipeline

# Push features from offline to online store
feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")

This runs a batch job that reads the latest features from your offline store and writes them to the online store. This typically runs on a cron schedule, hourly for most features, every 5 minutes for features that need higher freshness.

Tecton: The Managed Platform

Tecton is the commercial feature platform built by former Uber Michelangelo engineers. It handles everything Feast does, plus real-time feature computation, streaming ingestion, and monitoring.

The killer feature is unified transformation definitions:

# Tecton feature definition, single logic, multiple execution contexts
@batch_feature_view(
 sources=[transactions_batch],
 entities=[user_entity],
 mode="pandas",
 batch_schedule=timedelta(days=1),
)
def user_transaction_stats(transactions):
 return transactions.groupby("user_id").agg(
 avg_amount_30d=("amount", "mean"),
 count_30d=("amount", "count"),
 max_amount_30d=("amount", "max"),
 )

@stream_feature_view(
 source=transactions_stream,
 entities=[user_entity],
 mode="pandas",
 aggregation_interval=timedelta(minutes=5),
 aggregations=[
 Aggregation(column="amount", function="mean", time_window=timedelta(days=30)),
 Aggregation(column="amount", function="count", time_window=timedelta(days=30)),
 ],
)
def user_transaction_stats_realtime(transactions):
 return transactions

The same feature logic, applied both as a batch computation (for training) and as a streaming computation (for serving). Tecton guarantees consistency between them.

Real-Time Features: The Hard Part

Batch features (updated hourly or daily) cover maybe 60% of production ML use cases. The remaining 40% need features that reflect events from the last few minutes or seconds.

A proven approach for real-time features with Feast (since Feast does not natively handle streaming):

# Kafka consumer that computes and pushes real-time features
from feast import FeatureStore
from kafka import KafkaConsumer
import json

store = FeatureStore(repo_path="feature_repo/")
consumer = KafkaConsumer("user-events", bootstrap_servers="kafka:9092")

# In-memory sliding window for aggregation
from collections import defaultdict
import time

windows = defaultdict(list)
WINDOW_SIZE = 300 # 5 minutes

for message in consumer:
 event = json.loads(message.value)
 user_id = event["user_id"]
 timestamp = event["timestamp"]

 # Update sliding window
 windows[user_id].append({"ts": timestamp, "value": event["amount"]})
 # Evict old entries
 cutoff = time.time() - WINDOW_SIZE
 windows[user_id] = [e for e in windows[user_id] if e["ts"] > cutoff]

 # Compute real-time features
 values = [e["value"] for e in windows[user_id]]
 features = {
 "user_id": user_id,
 "event_count_5m": len(values),
 "avg_amount_5m": sum(values) / len(values) if values else 0,
 }

 # Push to online store
 store.push("user_realtime_stats", pd.DataFrame([features]))

This is production-viable but operationally heavy. You're managing a Kafka consumer, in-memory state, and the push to Feast. Tecton handles all of this natively, which is why it costs what it costs.

Feature Drift Detection

Features drift. User behavior changes, data sources get modified, upstream schemas evolve. If you're not monitoring feature distributions, your model degrades silently.

# Feature drift monitoring with evidently
from evidently.metric_preset import DataDriftPreset
from evidently.report import Report
import pandas as pd

# Reference: feature distributions from training data
reference_data = pd.read_parquet("training_features_snapshot.parquet")

# Current: latest features from the online store
current_data = fetch_current_features(sample_size=10000)

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference_data, current_data=current_data)
result = report.as_dict()

# Alert on drifted features
for feature_name, drift_info in result["metrics"][0]["result"]["drift_by_columns"].items():
 if drift_info["drift_detected"]:
 alert(
 f"Feature drift detected: {feature_name}",
 f"Drift score: {drift_info['drift_score']:.4f}, "
 f"Method: {drift_info['stattest_name']}"
 )

This typically runs daily as a scheduled job. When drift is detected, automatic retraining is not advisable, investigation comes first. Drift in user_avg_session_duration might mean a UI change, not a data quality issue. Context matters.

Production Architecture

Here is the full architecture for a fraud detection system with ~200 features:

Raw Data Sources
├── PostgreSQL (transactions, accounts)
├── Kafka (real-time events)
└── S3 (clickstream logs)
 │
 ▼
Feature Computation
├── Spark (batch features, daily)
├── Flink (streaming features, real-time)
└── Python transforms (on-demand features)
 │
 ▼
Feast Feature Store
├── Offline: S3 Parquet (training)
├── Online: Redis Cluster (serving)
└── Registry: PostgreSQL (metadata)
 │
 ├──► Training Pipeline
 │ └── get_historical_features() → Model Training
 │
 └──► Serving Pipeline
 └── get_online_features() → Model Inference (p99 < 10ms)

The Spark jobs run daily on EMR and produce Parquet files in S3. The Flink jobs run continuously, consuming from Kafka and pushing to Redis via Feast's push API. The model serving layer calls Feast for features, runs inference, and returns predictions.

Total latency budget for the fraud detection path: 50ms. Feature retrieval takes 4-8ms (Redis), model inference takes 2-5ms (ONNX Runtime), and the rest is network overhead and serialization.

Wrapping Up

Feature stores are unsexy infrastructure. Nobody gets promoted for deploying Feast. But the alternative, every data scientist maintaining their own feature computation pipelines, with different code for training vs serving, with no monitoring, and with point-in-time semantics implemented ad hoc (or not at all), is how you end up with ML systems that work in notebooks and fail in production.

Start with Feast. Use Redis for online, the existing warehouse for offline. Define features once, use them everywhere. Monitor distributions. Graduate to Tecton when the operational burden of managing streaming features independently exceeds the cost of the platform.

The model is the easy part. The features are the hard part. And the feature store is what keeps the hard part from becoming impossible.

MLOps Feature Stores: Feast, Tecton, Real-Time Features, and Production ML Pipeline Architecture