AI/ML # MLOps Feature Stores: Feast, Tecton, Real-Time Features, and Production ML Pipeline Architecture 2025-11-307 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The Core Problem ML models consume features, derived variables computed from raw data. A simple recommendation model might need: ``` user_avg_session_duration_7d, from clickstream user_purchase_count_30d, from transactions item_view_count_24h, from event stream user_item_category_affinity, from purchase history ``` Each of these has to be computed in at least two contexts: 1. **Training time**: from historical data, with correct point-in-time semantics 2. **Serving time**: from live data, at low latency (usually \<50ms) If the computation logic differs between these two contexts, even slightly, you get training-serving skew, and your model performs worse in production than in evaluation. This is not a theoretical concern. A 15% accuracy drop from a date timezone mismatch between a Spark batch job and a Python serving function is a documented pattern. ## Feast: The Open-Source Foundation [Feast]() (Feature Store) is the standard open-source option. It handles feature registration, offline/online storage, and materialization (copying features from offline to online store). ### Architecture ``` ┌─────────────┐ │ Feature │ │ Repository │ │ (Python) │ └──────┬──────┘ │ ┌────────────┼────────────┐ │ │ │ ┌────────▼───┐ ┌────▼─────┐ ┌──▼──────────┐ │ Offline │ │ Online │ │ Feature │ │ Store │ │ Store │ │ Server │ │ (BigQuery/ │ │ (Redis/ │ │ (gRPC/HTTP) │ │ Snowflake/ │ │ DynamoDB)│ │ │ │ S3+Parquet)│ │ │ │ │ └────────────┘ └──────────┘ └─────────────┘ ``` ### Defining Features ```python # feature_repo/features.py from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64 from datetime import timedelta user = Entity(name="user_id", join_keys=["user_id"]) user_transaction_stats = FeatureView( name="user_transaction_stats", entities=[user], schema=[ Field(name="avg_transaction_amount_30d", dtype=Float32), Field(name="transaction_count_30d", dtype=Int64), Field(name="max_transaction_amount_30d", dtype=Float32), Field(name="days_since_last_transaction", dtype=Int64), ], source=FileSource( path="s3://ml-features/user_transaction_stats.parquet", timestamp_field="event_timestamp", ), ttl=timedelta(days=1), online=True, ) ``` ### Point-in-Time Joins This is where feature stores earn their keep. When you fetch training data, you need features as they existed at the time of each training example: ```python from feast import FeatureStore import pandas as pd store = FeatureStore(repo_path="feature_repo/") # Training entity dataframe, each row is a training example # with the timestamp at which we need features entity_df = pd.DataFrame({ "user_id": [1001, 1002, 1001, 1003], "event_timestamp": [ "2025-11-15 10:30:00", "2025-11-15 14:00:00", "2025-12-01 09:00:00", # Same user, different time "2025-12-02 16:45:00", ], }) # Feast joins features with correct temporal semantics training_df = store.get_historical_features( entity_df=entity_df, features=[ "user_transaction_stats:avg_transaction_amount_30d", "user_transaction_stats:transaction_count_30d", "user_session_stats:avg_session_duration_7d", ], ).to_df() ``` The ` event_timestamp ` column is the key. For user 1001 on November 15th, Feast returns the feature values computed from data available up to November 15th. For the same user on December 1st, it returns different values, reflecting the additional two weeks of data. Without point-in-time joins, you'd use the latest feature values and unknowingly leak future information into training. ### Online Serving ```python # Real-time feature retrieval for inference features = store.get_online_features( features=[ "user_transaction_stats:avg_transaction_amount_30d", "user_transaction_stats:transaction_count_30d", ], entity_rows=[{"user_id": 1001}], ).to_dict() # Returns: {'avg_transaction_amount_30d': [142.50], 'transaction_count_30d': [23]} ``` With Redis as the online store, this returns in under 5ms at p99. Benchmarks show 2,000 requests/second on a single Feast server instance with 50 features per request. ### Materialization Pipeline ```bash # Push features from offline to online store feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S") ``` This runs a batch job that reads the latest features from your offline store and writes them to the online store. This typically runs on a cron schedule, hourly for most features, every 5 minutes for features that need higher freshness. ## Tecton: The Managed Platform [Tecton]() is the commercial feature platform built by former Uber Michelangelo engineers. It handles everything Feast does, plus real-time feature computation, streaming ingestion, and monitoring. The killer feature is unified transformation definitions: ```python # Tecton feature definition, single logic, multiple execution contexts @batch_feature_view( sources=[transactions_batch], entities=[user_entity], mode="pandas", batch_schedule=timedelta(days=1), ) def user_transaction_stats(transactions): return transactions.groupby("user_id").agg( avg_amount_30d=("amount", "mean"), count_30d=("amount", "count"), max_amount_30d=("amount", "max"), ) @stream_feature_view( source=transactions_stream, entities=[user_entity], mode="pandas", aggregation_interval=timedelta(minutes=5), aggregations=[ Aggregation(column="amount", function="mean", time_window=timedelta(days=30)), Aggregation(column="amount", function="count", time_window=timedelta(days=30)), ], ) def user_transaction_stats_realtime(transactions): return transactions ``` The same feature logic, applied both as a batch computation (for training) and as a streaming computation (for serving). Tecton guarantees consistency between them. ## Real-Time Features: The Hard Part Batch features (updated hourly or daily) cover maybe 60% of production ML use cases. The remaining 40% need features that reflect events from the last few minutes or seconds. A proven approach for real-time features with Feast (since Feast does not natively handle streaming): ```python # Kafka consumer that computes and pushes real-time features from feast import FeatureStore from kafka import KafkaConsumer import json store = FeatureStore(repo_path="feature_repo/") consumer = KafkaConsumer("user-events", bootstrap_servers="kafka:9092") # In-memory sliding window for aggregation from collections import defaultdict import time windows = defaultdict(list) WINDOW_SIZE = 300 # 5 minutes for message in consumer: event = json.loads(message.value) user_id = event["user_id"] timestamp = event["timestamp"] # Update sliding window windows[user_id].append({"ts": timestamp, "value": event["amount"]}) # Evict old entries cutoff = time.time() - WINDOW_SIZE windows[user_id] = [e for e in windows[user_id] if e["ts"] > cutoff] # Compute real-time features values = [e["value"] for e in windows[user_id]] features = { "user_id": user_id, "event_count_5m": len(values), "avg_amount_5m": sum(values) / len(values) if values else 0, } # Push to online store store.push("user_realtime_stats", pd.DataFrame([features])) ``` This is production-viable but operationally heavy. You're managing a Kafka consumer, in-memory state, and the push to Feast. Tecton handles all of this natively, which is why it costs what it costs. ## Feature Drift Detection Features drift. User behavior changes, data sources get modified, upstream schemas evolve. If you're not monitoring feature distributions, your model degrades silently. ```python # Feature drift monitoring with evidently from evidently.metric_preset import DataDriftPreset from evidently.report import Report import pandas as pd # Reference: feature distributions from training data reference_data = pd.read_parquet("training_features_snapshot.parquet") # Current: latest features from the online store current_data = fetch_current_features(sample_size=10000) report = Report(metrics=[DataDriftPreset()]) report.run(reference_data=reference_data, current_data=current_data) result = report.as_dict() # Alert on drifted features for feature_name, drift_info in result["metrics"][0]["result"]["drift_by_columns"].items(): if drift_info["drift_detected"]: alert( f"Feature drift detected: {feature_name}", f"Drift score: {drift_info['drift_score']:.4f}, " f"Method: {drift_info['stattest_name']}" ) ``` This typically runs daily as a scheduled job. When drift is detected, automatic retraining is not advisable, investigation comes first. Drift in ` user_avg_session_duration ` might mean a UI change, not a data quality issue. Context matters. ## Production Architecture Here is the full architecture for a fraud detection system with ~200 features: ``` Raw Data Sources ├── PostgreSQL (transactions, accounts) ├── Kafka (real-time events) └── S3 (clickstream logs) │ ▼ Feature Computation ├── Spark (batch features, daily) ├── Flink (streaming features, real-time) └── Python transforms (on-demand features) │ ▼ Feast Feature Store ├── Offline: S3 Parquet (training) ├── Online: Redis Cluster (serving) └── Registry: PostgreSQL (metadata) │ ├──► Training Pipeline │ └── get_historical_features() → Model Training │ └──► Serving Pipeline └── get_online_features() → Model Inference (p99 < 10ms) ``` The Spark jobs run daily on EMR and produce Parquet files in S3. The Flink jobs run continuously, consuming from Kafka and pushing to Redis via Feast's push API. The model serving layer calls Feast for features, runs inference, and returns predictions. Total latency budget for the fraud detection path: 50ms. Feature retrieval takes 4-8ms (Redis), model inference takes 2-5ms (ONNX Runtime), and the rest is network overhead and serialization. ## Wrapping Up Feature stores are unsexy infrastructure. Nobody gets promoted for deploying Feast. But the alternative, every data scientist maintaining their own feature computation pipelines, with different code for training vs serving, with no monitoring, and with point-in-time semantics implemented ad hoc (or not at all), is how you end up with ML systems that work in notebooks and fail in production. Start with Feast. Use Redis for online, the existing warehouse for offline. Define features once, use them everywhere. Monitor distributions. Graduate to Tecton when the operational burden of managing streaming features independently exceeds the cost of the platform. The model is the easy part. The features are the hard part. And the feature store is what keeps the hard part from becoming impossible. ## Frequently asked questions ### What is a feature store and why do I need one? A feature store is a centralized system for managing ML features, the input variables your models consume. It handles storage (online for low-latency serving, offline for training), transformation (computing features from raw data), and consistency (ensuring training and serving use identical feature logic). You need one when feature duplication across teams, training-serving skew, or feature freshness becomes a bottleneck. ### How do point-in-time joins prevent data leakage? Point-in-time joins retrieve feature values as they existed at the time of each training example, not as they exist now. If you're training a fraud detector on a transaction from January 15th, the join fetches the user's average spend as of January 15th, not today's value, which includes months of data that wouldn't have been available at prediction time. ### Is Feast production-ready for real-time serving? Yes, with the right online store backend. Feast with Redis serves features at sub-5ms p99 latency for moderate feature set sizes. For very high throughput (100K+ requests/second), you'll want to tune Redis cluster configuration or consider DynamoDB as the online store. The main limitation is that Feast's streaming ingestion requires you to manage the Spark/Flink job yourself. ## Tips 1. **Tip:** Always compute training features with point-in-time joins, never with current-state lookups. Training a fraud model with 'current account balance' instead of 'balance at transaction time' produces a model that looks amazing in training and is useless in production. Classic leakage. 2. **Tip:** For real-time features, keep your transformation logic in one place. If your batch pipeline computes ` avg_transactions_7d ` differently than your streaming pipeline, you've got a training-serving skew that will silently destroy your model's performance. 3. **Tip:** Start with Feast + Redis for online serving and Feast + the existing data warehouse for offline. Tecton becomes worthwhile once feature count crosses 500 and a dedicated ML platform team is in place. The operational overhead of a managed service only pays off at scale. - mlops - feature-stores - feast - tecton - machine-learning - real-time-features - point-in-time-joins - feature-drift - ml-pipelines - production-ml [Back to Blog]()