{"schemaVersion":1,"id":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust","title":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","description":"A hands-on guide to Apache Arrow DataFusion, building custom query engines in Rust with columnar execution, push-based vs pull-based models, custom table providers, user-defined functions, and real-world performance tuning for analytical workloads.","language":"en","datePublished":"2026-01-05","dateModified":null,"contentFormat":"text/markdown","content":"Data Engineering\n\n# Apache Arrow DataFusion: Building Custom Query Engines in Rust\n\n2026-01-057 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/arrow-datafusion-query-engine-rust>)\n\n## DataFusion Overview and Architecture\n\n[Apache Arrow DataFusion](<https://datafusion.apache.org/>) is a query engine, not a database. It provides:\n\n- A SQL parser and planner\n- A cost-based query optimizer (with 60+ optimization rules)\n- A parallel execution engine using Arrow RecordBatches\n- An extension API for custom everything\n\nWhat it does NOT give you: storage management, transactions, replication, user management. It's a library, not a server.\n\n```rust\nuse datafusion::prelude::*;\n\n#[tokio::main]\nasync fn main() -> datafusion::error::Result<()> {\n let ctx = SessionContext::new();\n\n // Register a Parquet file as a table\n ctx.register_parquet(\n \"events\",\n \"s3://mybucket/events/*.parquet\",\n ParquetReadOptions::default(),\n ).await?;\n\n // SQL just works\n let df = ctx.sql(\"\n SELECT date_trunc('hour', timestamp) as hour,\n count(*) as event_count,\n approx_percentile_cont(latency_ms, 0.99) as p99\n FROM events\n WHERE timestamp > '2026-02-01'\n GROUP BY 1\n ORDER BY 1\n \").await?;\n\n df.show().await?;\n Ok(())\n}\n```\n\nThat's a fully parallel query with predicate pushdown to Parquet row groups, columnar execution on Arrow arrays, and approximate percentile computation. In about 15 lines.\n\n## The Execution Model: How Queries Actually Run\n\nDataFusion uses a pull-based streaming model. Here's the flow:\n\n```\nSQL String\n ↓ (parse)\nLogical Plan (AST of relational operators)\n ↓ (optimize, 60+ rules)\nOptimized Logical Plan\n ↓ (physical planning)\nPhysical Plan (concrete execution operators)\n ↓ (execute)\nStream of RecordBatches\n```\n\nEach physical operator implements ` ExecutionPlan `:\n\n```rust\npub trait ExecutionPlan: Send + Sync {\n /// Output schema\n fn schema(&self) -> SchemaRef;\n\n /// How the output is partitioned\n fn output_partitioning(&self) -> Partitioning;\n\n /// Required input ordering\n fn required_input_ordering(&self) -> Vec<Option<Vec<PhysicalSortExpr>>>;\n\n /// Create a stream of RecordBatches for partition `partition`\n fn execute(\n &self,\n partition: usize,\n context: Arc<TaskContext>,\n ) -> Result<SendableRecordBatchStream>;\n\n /// Children of this node\n fn children(&self) -> Vec<Arc<dyn ExecutionPlan>>;\n}\n```\n\nThe key insight: ` execute() ` returns a ` Stream `, not a ` Vec `. Data flows in batches (default 8,192 rows), and each partition runs on its own tokio task. This means a query over 16 Parquet files naturally parallelizes to 16 tasks, each streaming batches through the operator pipeline.\n\n## Custom Table Providers: Your Data, Your Rules\n\nThis is where DataFusion gets really interesting. The ` TableProvider ` trait lets you wrap any data source:\n\n```rust\nuse arrow::datatypes::SchemaRef;\nuse datafusion::datasource::TableProvider;\nuse datafusion::logical_expr::TableType;\nuse datafusion::execution::context::SessionState;\nuse datafusion::physical_plan::ExecutionPlan;\nuse datafusion::logical_expr::Expr;\n\npub struct MetricsTableProvider {\n schema: SchemaRef,\n metrics_client: Arc<MetricsClient>,\n}\n\n#[async_trait]\nimpl TableProvider for MetricsTableProvider {\n fn as_any(&self) -> &dyn Any { self }\n\n fn schema(&self) -> SchemaRef {\n self.schema.clone()\n }\n\n fn table_type(&self) -> TableType {\n TableType::Base\n }\n\n async fn scan(\n &self,\n state: &SessionState,\n projection: Option<&Vec<usize>>,\n filters: &[Expr],\n limit: Option<usize>,\n ) -> Result<Arc<dyn ExecutionPlan>> {\n // Extract time range from filters for pushdown\n let time_range = extract_time_range(filters);\n\n // Only project requested columns\n let projected_schema = project_schema(&self.schema, projection);\n\n Ok(Arc::new(MetricsScanExec {\n schema: projected_schema,\n client: self.metrics_client.clone(),\n time_range,\n limit,\n }))\n }\n\n fn supports_filters_pushdown(\n &self,\n filters: &[&Expr],\n ) -> Result<Vec<TableProviderFilterPushDown>> {\n filters.iter().map(|f| {\n if is_time_filter(f) {\n Ok(TableProviderFilterPushDown::Exact)\n } else {\n Ok(TableProviderFilterPushDown::Unsupported)\n }\n }).collect()\n }\n}\n```\n\nThe ` supports_filters_pushdown ` method tells the optimizer which predicates your provider can handle natively. Returning ` Exact ` means the optimizer won't apply that filter again after the scan, your provider handled it completely. This eliminates redundant filtering and can cut metric queries from ~800ms to ~200ms.\n\n## User-Defined Functions: Extending the SQL Surface\n\nDataFusion supports scalar UDFs, aggregate UDFs, and window UDFs:\n\n```rust\nuse datafusion::logical_expr::{create_udf, Volatility};\nuse datafusion::arrow::datatypes::DataType;\nuse arrow::array::{ArrayRef, StringArray, Float64Array};\n\n// Scalar UDF: custom haversine distance\nfn haversine_distance(args: &[ArrayRef]) -> Result<ArrayRef> {\n let lat1 = args[0].as_any().downcast_ref::<Float64Array>().unwrap();\n let lon1 = args[1].as_any().downcast_ref::<Float64Array>().unwrap();\n let lat2 = args[2].as_any().downcast_ref::<Float64Array>().unwrap();\n let lon2 = args[3].as_any().downcast_ref::<Float64Array>().unwrap();\n\n let result: Float64Array = lat1.iter().zip(lon1.iter())\n .zip(lat2.iter().zip(lon2.iter()))\n .map(|((la1, lo1), (la2, lo2))| {\n match (la1, lo1, la2, lo2) {\n (Some(la1), Some(lo1), Some(la2), Some(lo2)) => {\n Some(haversine(la1, lo1, la2, lo2))\n }\n _ => None,\n }\n })\n .collect();\n\n Ok(Arc::new(result))\n}\n\n// Register it\nlet udf = create_udf(\n \"haversine\",\n vec![DataType::Float64, DataType::Float64,\n DataType::Float64, DataType::Float64],\n Arc::new(DataType::Float64),\n Volatility::Immutable,\n Arc::new(haversine_distance),\n);\n\nctx.register_udf(udf);\n\n// Now use it in SQL\nlet df = ctx.sql(\"\n SELECT city_name,\n haversine(lat, lon, -34.6037, -58.3816) as distance_to_bsas\n FROM cities\n ORDER BY distance_to_bsas\n LIMIT 10\n\").await?;\n```\n\nFor aggregate UDFs, you implement the ` Accumulator ` trait which handles partial aggregation across partitions automatically:\n\n```rust\nuse datafusion::logical_expr::Accumulator;\nuse datafusion::scalar::ScalarValue;\n\n#[derive(Debug)]\nstruct ApproxCountDistinctAccumulator {\n hll: HyperLogLog,\n}\n\nimpl Accumulator for ApproxCountDistinctAccumulator {\n fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {\n let array = values[0].as_any().downcast_ref::<StringArray>().unwrap();\n for value in array.iter().flatten() {\n self.hll.insert(value);\n }\n Ok(())\n }\n\n fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {\n // Merge HLL sketches from other partitions\n let state_array = states[0].as_any()\n .downcast_ref::<BinaryArray>().unwrap();\n for state in state_array.iter().flatten() {\n let other: HyperLogLog = deserialize(state);\n self.hll.merge(&other);\n }\n Ok(())\n }\n\n fn state(&mut self) -> Result<Vec<ScalarValue>> {\n Ok(vec![ScalarValue::Binary(Some(serialize(&self.hll)))])\n }\n\n fn evaluate(&mut self) -> Result<ScalarValue> {\n Ok(ScalarValue::UInt64(Some(self.hll.count())))\n }\n}\n```\n\n## Push-Based vs Pull-Based: The Architecture Decision\n\nDataFusion uses pull-based execution (the Volcano model): each operator calls ` next() ` on its child to get the next batch. This has natural backpressure, a slow consumer automatically slows down the producer.\n\nPush-based systems (like Velox or some parts of ClickHouse) invert this: a source pushes data downstream through a chain of operators. This can be faster because it avoids virtual function call overhead per batch and enables better SIMD vectorization.\n\n```\nPull-based (DataFusion):\n FilterExec.next()\n → calls ProjectionExec.next()\n → calls ParquetExec.next()\n → reads batch from Parquet file\n → returns RecordBatch upstream\n → applies projection\n → returns RecordBatch upstream\n → applies filter\n → returns RecordBatch to caller\n\nPush-based (Velox-style):\n ParquetSource.produce()\n → reads batch from Parquet file\n → pushes to ProjectionOp.process(batch)\n → applies projection\n → pushes to FilterOp.process(batch)\n → applies filter\n → pushes to OutputSink.consume(batch)\n```\n\nIn practice, the performance difference on analytical queries is minimal (5-15%) because the batch size (8K rows) amortizes the per-call overhead. Where pull-based shines is composability: you can easily build complex plans with joins, subqueries, and correlated expressions without managing explicit data flow.\n\n## Performance Tuning: Real Numbers\n\nHere are the optimizations that make the biggest difference in production deployments:\n\n### 1. Partition pruning\n\n```rust\n// Register partitioned data\nctx.register_parquet(\n \"events\",\n \"s3://bucket/events/year=*/month=*/day=*/*.parquet\",\n ParquetReadOptions::default()\n .table_partition_cols(vec![\n (\"year\".to_string(), DataType::Int32),\n (\"month\".to_string(), DataType::Int32),\n (\"day\".to_string(), DataType::Int32),\n ]),\n).await?;\n\n// Query with partition filter, only reads relevant files\n// Before: scans 365 files (~50 GB)\n// After: scans 7 files (~960 MB)\nlet df = ctx.sql(\"\n SELECT count(*)\n FROM events\n WHERE year = 2026 AND month = 2 AND day BETWEEN 18 AND 24\n\").await?;\n```\n\n### 2. Memory management\n\n```rust\nlet config = SessionConfig::new()\n .with_batch_size(16384) // Larger batches = less overhead\n .with_target_partitions(16) // Match CPU cores\n .with_repartition_joins(true) // Repartition for hash joins\n .with_repartition_aggregations(true);\n\nlet runtime = RuntimeConfig::new()\n .with_memory_limit(4 * 1024 * 1024 * 1024, 0.8); // 4 GB, spill at 80%\n\nlet ctx = SessionContext::new_with_config_rt(\n config,\n Arc::new(RuntimeEnv::new(runtime)?),\n);\n```\n\n### 3. Benchmark results\n\nOn a 32-core machine, querying 50 GB of Parquet data (1.2 billion rows):\n\n```\nQuery: SELECT date_trunc('hour', ts), count(*), avg(value)\n FROM events WHERE ts > '2026-02-01' GROUP BY 1\n\nDuckDB (in-process): 1.8s\nDataFusion (default): 2.4s\nDataFusion (tuned): 1.1s\nSpark (local[32]): 14.2s\n\nTuning applied:\n - batch_size: 16384 (up from 8192)\n - target_partitions: 32\n - Predicate pushdown to row groups\n - Custom Parquet reader with async prefetch\n```\n\nThe tuned DataFusion was actually faster than DuckDB on this specific workload because the custom Parquet reader overlapped I/O with computation, while DuckDB blocked on reads, DataFusion was processing the previous batch.\n\n## A Production Architecture\n\nA mature DataFusion deployment architecture:\n\n```\n┌─────────────────────────────────────────────┐\n│ SQL / DataFrame API │\n├─────────────────────────────────────────────┤\n│ DataFusion Query Engine │\n│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │\n│ │ Parquet │ │ Redis │ │ Metrics │ │\n│ │ Provider │ │ Provider │ │ Provider │ │\n│ └──────────┘ └──────────┘ └──────────┘ │\n├─────────────────────────────────────────────┤\n│ Custom UDFs │\n│ - haversine(), geo_contains() │\n│ - percentile_tdigest() aggregate │\n│ - session_window() window function │\n├─────────────────────────────────────────────┤\n│ Object Store Layer (S3, local, cache) │\n└─────────────────────────────────────────────┘\n```\n\nCustom providers for each data source, a shared UDF library, and an object store abstraction that handles S3 with local caching. The whole thing compiles to a single binary that starts in under 100ms.\n\nFor analytics on heterogeneous data without a managed database server, DataFusion is one of the strongest options available. The Rust ecosystem around Arrow is maturing fast, and the community is very active. The extension APIs are powerful but not trivial, solid Rust knowledge is a prerequisite.\n\n## Frequently asked questions\n\n### What is Apache Arrow DataFusion?\n\nDataFusion is an extensible query engine written in Rust, built on top of Apache Arrow's columnar memory format. It provides a SQL and DataFrame API, a cost-based query optimizer, and a parallel execution engine that you can embed in your own applications or extend with custom data sources, functions, and execution strategies.\n\n### When should I use DataFusion instead of an existing database?\n\nDataFusion excels when you need embedded analytics without a separate database server, custom data sources that don't fit traditional databases, or when you want to build domain-specific query engines. If you need transactions, multi-user concurrency, or full ACID guarantees, a traditional database is better.\n\n### What's the difference between push-based and pull-based execution?\n\nIn pull-based (Volcano) execution, each operator pulls batches from its children by calling next(). In push-based execution, data flows downstream without operators needing to request it. DataFusion uses a pull-based streaming model with RecordBatch as the unit of work, which naturally supports backpressure and integrates well with Rust's async model.\n\n## Tips\n\n1. **Tip:** DataFusion's ` TableProvider ` trait is deceptively powerful. You can wrap literally any data source, S3 Parquet files, REST APIs, even gRPC streams, behind it, and the query optimizer just works. A custom provider for an internal metrics store can be built in about 200 lines of Rust.\n2. **Tip:** If your queries filter on a specific column 90% of the time, implement ` supports_filters_pushdown ` in your ` TableProvider `. The optimizer will push predicates down to your scan, and you can skip reading entire row groups. This alone can cut query latency by 4x.\n3. **Tip:** Don't write custom aggregation functions from scratch. Use DataFusion's ` Accumulator ` trait, it handles partitioning and merging across partitions automatically. Rolling a custom parallel HyperLogLog from scratch is unnecessary when the trait handles all the hard parts automatically.\n\n- arrow\n- datafusion\n- rust\n- query-engine\n- columnar\n- data-engineering\n- analytics\n- performance\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#title"},{"level":2,"text":"DataFusion Overview and Architecture","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#datafusion-overview-and-architecture"},{"level":2,"text":"The Execution Model: How Queries Actually Run","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#the-execution-model-how-queries-actually-run"},{"level":2,"text":"Custom Table Providers: Your Data, Your Rules","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#custom-table-providers-your-data-your-rules"},{"level":2,"text":"User-Defined Functions: Extending the SQL Surface","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#user-defined-functions-extending-the-sql-surface"},{"level":2,"text":"Push-Based vs Pull-Based: The Architecture Decision","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#push-based-vs-pull-based-the-architecture-decision"},{"level":2,"text":"Performance Tuning: Real Numbers","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#performance-tuning-real-numbers"},{"level":3,"text":"1. Partition pruning","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#1-partition-pruning"},{"level":3,"text":"2. Memory management","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#2-memory-management"},{"level":3,"text":"3. Benchmark results","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#3-benchmark-results"},{"level":2,"text":"A Production Architecture","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#a-production-architecture"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#questions"},{"level":3,"text":"What is Apache Arrow DataFusion?","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-1"},{"level":3,"text":"When should I use DataFusion instead of an existing database?","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-2"},{"level":3,"text":"What's the difference between push-based and pull-based execution?","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#article-tips"}],"sections":[{"heading":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","level":1,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust","content":"Data Engineering"},{"heading":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","level":1,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#title","content":"2026-01-057 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/arrow-datafusion-query-engine-rust>)"},{"heading":"DataFusion Overview and Architecture","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#datafusion-overview-and-architecture","content":"[Apache Arrow DataFusion](<https://datafusion.apache.org/>) is a query engine, not a database. It provides:\n\n- A SQL parser and planner\n- A cost-based query optimizer (with 60+ optimization rules)\n- A parallel execution engine using Arrow RecordBatches\n- An extension API for custom everything\n\nWhat it does NOT give you: storage management, transactions, replication, user management. It's a library, not a server.\n\n```rust\nuse datafusion::prelude::*;\n\n#[tokio::main]\nasync fn main() -> datafusion::error::Result<()> {\n let ctx = SessionContext::new();\n\n // Register a Parquet file as a table\n ctx.register_parquet(\n \"events\",\n \"s3://mybucket/events/*.parquet\",\n ParquetReadOptions::default(),\n ).await?;\n\n // SQL just works\n let df = ctx.sql(\"\n SELECT date_trunc('hour', timestamp) as hour,\n count(*) as event_count,\n approx_percentile_cont(latency_ms, 0.99) as p99\n FROM events\n WHERE timestamp > '2026-02-01'\n GROUP BY 1\n ORDER BY 1\n \").await?;\n\n df.show().await?;\n Ok(())\n}\n```\n\nThat's a fully parallel query with predicate pushdown to Parquet row groups, columnar execution on Arrow arrays, and approximate percentile computation. In about 15 lines."},{"heading":"The Execution Model: How Queries Actually Run","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#the-execution-model-how-queries-actually-run","content":"DataFusion uses a pull-based streaming model. Here's the flow:\n\n```\nSQL String\n ↓ (parse)\nLogical Plan (AST of relational operators)\n ↓ (optimize, 60+ rules)\nOptimized Logical Plan\n ↓ (physical planning)\nPhysical Plan (concrete execution operators)\n ↓ (execute)\nStream of RecordBatches\n```\n\nEach physical operator implements ` ExecutionPlan `:\n\n```rust\npub trait ExecutionPlan: Send + Sync {\n /// Output schema\n fn schema(&self) -> SchemaRef;\n\n /// How the output is partitioned\n fn output_partitioning(&self) -> Partitioning;\n\n /// Required input ordering\n fn required_input_ordering(&self) -> Vec<Option<Vec<PhysicalSortExpr>>>;\n\n /// Create a stream of RecordBatches for partition `partition`\n fn execute(\n &self,\n partition: usize,\n context: Arc<TaskContext>,\n ) -> Result<SendableRecordBatchStream>;\n\n /// Children of this node\n fn children(&self) -> Vec<Arc<dyn ExecutionPlan>>;\n}\n```\n\nThe key insight: ` execute() ` returns a ` Stream `, not a ` Vec `. Data flows in batches (default 8,192 rows), and each partition runs on its own tokio task. This means a query over 16 Parquet files naturally parallelizes to 16 tasks, each streaming batches through the operator pipeline."},{"heading":"Custom Table Providers: Your Data, Your Rules","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#custom-table-providers-your-data-your-rules","content":"This is where DataFusion gets really interesting. The ` TableProvider ` trait lets you wrap any data source:\n\n```rust\nuse arrow::datatypes::SchemaRef;\nuse datafusion::datasource::TableProvider;\nuse datafusion::logical_expr::TableType;\nuse datafusion::execution::context::SessionState;\nuse datafusion::physical_plan::ExecutionPlan;\nuse datafusion::logical_expr::Expr;\n\npub struct MetricsTableProvider {\n schema: SchemaRef,\n metrics_client: Arc<MetricsClient>,\n}\n\n#[async_trait]\nimpl TableProvider for MetricsTableProvider {\n fn as_any(&self) -> &dyn Any { self }\n\n fn schema(&self) -> SchemaRef {\n self.schema.clone()\n }\n\n fn table_type(&self) -> TableType {\n TableType::Base\n }\n\n async fn scan(\n &self,\n state: &SessionState,\n projection: Option<&Vec<usize>>,\n filters: &[Expr],\n limit: Option<usize>,\n ) -> Result<Arc<dyn ExecutionPlan>> {\n // Extract time range from filters for pushdown\n let time_range = extract_time_range(filters);\n\n // Only project requested columns\n let projected_schema = project_schema(&self.schema, projection);\n\n Ok(Arc::new(MetricsScanExec {\n schema: projected_schema,\n client: self.metrics_client.clone(),\n time_range,\n limit,\n }))\n }\n\n fn supports_filters_pushdown(\n &self,\n filters: &[&Expr],\n ) -> Result<Vec<TableProviderFilterPushDown>> {\n filters.iter().map(|f| {\n if is_time_filter(f) {\n Ok(TableProviderFilterPushDown::Exact)\n } else {\n Ok(TableProviderFilterPushDown::Unsupported)\n }\n }).collect()\n }\n}\n```\n\nThe ` supports_filters_pushdown ` method tells the optimizer which predicates your provider can handle natively. Returning ` Exact ` means the optimizer won't apply that filter again after the scan, your provider handled it completely. This eliminates redundant filtering and can cut metric queries from ~800ms to ~200ms."},{"heading":"User-Defined Functions: Extending the SQL Surface","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#user-defined-functions-extending-the-sql-surface","content":"DataFusion supports scalar UDFs, aggregate UDFs, and window UDFs:\n\n```rust\nuse datafusion::logical_expr::{create_udf, Volatility};\nuse datafusion::arrow::datatypes::DataType;\nuse arrow::array::{ArrayRef, StringArray, Float64Array};\n\n// Scalar UDF: custom haversine distance\nfn haversine_distance(args: &[ArrayRef]) -> Result<ArrayRef> {\n let lat1 = args[0].as_any().downcast_ref::<Float64Array>().unwrap();\n let lon1 = args[1].as_any().downcast_ref::<Float64Array>().unwrap();\n let lat2 = args[2].as_any().downcast_ref::<Float64Array>().unwrap();\n let lon2 = args[3].as_any().downcast_ref::<Float64Array>().unwrap();\n\n let result: Float64Array = lat1.iter().zip(lon1.iter())\n .zip(lat2.iter().zip(lon2.iter()))\n .map(|((la1, lo1), (la2, lo2))| {\n match (la1, lo1, la2, lo2) {\n (Some(la1), Some(lo1), Some(la2), Some(lo2)) => {\n Some(haversine(la1, lo1, la2, lo2))\n }\n _ => None,\n }\n })\n .collect();\n\n Ok(Arc::new(result))\n}\n\n// Register it\nlet udf = create_udf(\n \"haversine\",\n vec![DataType::Float64, DataType::Float64,\n DataType::Float64, DataType::Float64],\n Arc::new(DataType::Float64),\n Volatility::Immutable,\n Arc::new(haversine_distance),\n);\n\nctx.register_udf(udf);\n\n// Now use it in SQL\nlet df = ctx.sql(\"\n SELECT city_name,\n haversine(lat, lon, -34.6037, -58.3816) as distance_to_bsas\n FROM cities\n ORDER BY distance_to_bsas\n LIMIT 10\n\").await?;\n```\n\nFor aggregate UDFs, you implement the ` Accumulator ` trait which handles partial aggregation across partitions automatically:\n\n```rust\nuse datafusion::logical_expr::Accumulator;\nuse datafusion::scalar::ScalarValue;\n\n#[derive(Debug)]\nstruct ApproxCountDistinctAccumulator {\n hll: HyperLogLog,\n}\n\nimpl Accumulator for ApproxCountDistinctAccumulator {\n fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {\n let array = values[0].as_any().downcast_ref::<StringArray>().unwrap();\n for value in array.iter().flatten() {\n self.hll.insert(value);\n }\n Ok(())\n }\n\n fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {\n // Merge HLL sketches from other partitions\n let state_array = states[0].as_any()\n .downcast_ref::<BinaryArray>().unwrap();\n for state in state_array.iter().flatten() {\n let other: HyperLogLog = deserialize(state);\n self.hll.merge(&other);\n }\n Ok(())\n }\n\n fn state(&mut self) -> Result<Vec<ScalarValue>> {\n Ok(vec![ScalarValue::Binary(Some(serialize(&self.hll)))])\n }\n\n fn evaluate(&mut self) -> Result<ScalarValue> {\n Ok(ScalarValue::UInt64(Some(self.hll.count())))\n }\n}\n```"},{"heading":"Push-Based vs Pull-Based: The Architecture Decision","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#push-based-vs-pull-based-the-architecture-decision","content":"DataFusion uses pull-based execution (the Volcano model): each operator calls ` next() ` on its child to get the next batch. This has natural backpressure, a slow consumer automatically slows down the producer.\n\nPush-based systems (like Velox or some parts of ClickHouse) invert this: a source pushes data downstream through a chain of operators. This can be faster because it avoids virtual function call overhead per batch and enables better SIMD vectorization.\n\n```\nPull-based (DataFusion):\n FilterExec.next()\n → calls ProjectionExec.next()\n → calls ParquetExec.next()\n → reads batch from Parquet file\n → returns RecordBatch upstream\n → applies projection\n → returns RecordBatch upstream\n → applies filter\n → returns RecordBatch to caller\n\nPush-based (Velox-style):\n ParquetSource.produce()\n → reads batch from Parquet file\n → pushes to ProjectionOp.process(batch)\n → applies projection\n → pushes to FilterOp.process(batch)\n → applies filter\n → pushes to OutputSink.consume(batch)\n```\n\nIn practice, the performance difference on analytical queries is minimal (5-15%) because the batch size (8K rows) amortizes the per-call overhead. Where pull-based shines is composability: you can easily build complex plans with joins, subqueries, and correlated expressions without managing explicit data flow."},{"heading":"Performance Tuning: Real Numbers","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#performance-tuning-real-numbers","content":"Here are the optimizations that make the biggest difference in production deployments:"},{"heading":"1. Partition pruning","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#1-partition-pruning","content":"```rust\n// Register partitioned data\nctx.register_parquet(\n \"events\",\n \"s3://bucket/events/year=*/month=*/day=*/*.parquet\",\n ParquetReadOptions::default()\n .table_partition_cols(vec![\n (\"year\".to_string(), DataType::Int32),\n (\"month\".to_string(), DataType::Int32),\n (\"day\".to_string(), DataType::Int32),\n ]),\n).await?;\n\n// Query with partition filter, only reads relevant files\n// Before: scans 365 files (~50 GB)\n// After: scans 7 files (~960 MB)\nlet df = ctx.sql(\"\n SELECT count(*)\n FROM events\n WHERE year = 2026 AND month = 2 AND day BETWEEN 18 AND 24\n\").await?;\n```"},{"heading":"2. Memory management","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#2-memory-management","content":"```rust\nlet config = SessionConfig::new()\n .with_batch_size(16384) // Larger batches = less overhead\n .with_target_partitions(16) // Match CPU cores\n .with_repartition_joins(true) // Repartition for hash joins\n .with_repartition_aggregations(true);\n\nlet runtime = RuntimeConfig::new()\n .with_memory_limit(4 * 1024 * 1024 * 1024, 0.8); // 4 GB, spill at 80%\n\nlet ctx = SessionContext::new_with_config_rt(\n config,\n Arc::new(RuntimeEnv::new(runtime)?),\n);\n```"},{"heading":"3. Benchmark results","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#3-benchmark-results","content":"On a 32-core machine, querying 50 GB of Parquet data (1.2 billion rows):\n\n```\nQuery: SELECT date_trunc('hour', ts), count(*), avg(value)\n FROM events WHERE ts > '2026-02-01' GROUP BY 1\n\nDuckDB (in-process): 1.8s\nDataFusion (default): 2.4s\nDataFusion (tuned): 1.1s\nSpark (local[32]): 14.2s\n\nTuning applied:\n - batch_size: 16384 (up from 8192)\n - target_partitions: 32\n - Predicate pushdown to row groups\n - Custom Parquet reader with async prefetch\n```\n\nThe tuned DataFusion was actually faster than DuckDB on this specific workload because the custom Parquet reader overlapped I/O with computation, while DuckDB blocked on reads, DataFusion was processing the previous batch."},{"heading":"A Production Architecture","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#a-production-architecture","content":"A mature DataFusion deployment architecture:\n\n```\n┌─────────────────────────────────────────────┐\n│ SQL / DataFrame API │\n├─────────────────────────────────────────────┤\n│ DataFusion Query Engine │\n│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │\n│ │ Parquet │ │ Redis │ │ Metrics │ │\n│ │ Provider │ │ Provider │ │ Provider │ │\n│ └──────────┘ └──────────┘ └──────────┘ │\n├─────────────────────────────────────────────┤\n│ Custom UDFs │\n│ - haversine(), geo_contains() │\n│ - percentile_tdigest() aggregate │\n│ - session_window() window function │\n├─────────────────────────────────────────────┤\n│ Object Store Layer (S3, local, cache) │\n└─────────────────────────────────────────────┘\n```\n\nCustom providers for each data source, a shared UDF library, and an object store abstraction that handles S3 with local caching. The whole thing compiles to a single binary that starts in under 100ms.\n\nFor analytics on heterogeneous data without a managed database server, DataFusion is one of the strongest options available. The Rust ecosystem around Arrow is maturing fast, and the community is very active. The extension APIs are powerful but not trivial, solid Rust knowledge is a prerequisite."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#questions","content":""},{"heading":"What is Apache Arrow DataFusion?","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-1","content":"DataFusion is an extensible query engine written in Rust, built on top of Apache Arrow's columnar memory format. It provides a SQL and DataFrame API, a cost-based query optimizer, and a parallel execution engine that you can embed in your own applications or extend with custom data sources, functions, and execution strategies."},{"heading":"When should I use DataFusion instead of an existing database?","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-2","content":"DataFusion excels when you need embedded analytics without a separate database server, custom data sources that don't fit traditional databases, or when you want to build domain-specific query engines. If you need transactions, multi-user concurrency, or full ACID guarantees, a traditional database is better."},{"heading":"What's the difference between push-based and pull-based execution?","level":3,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#question-3","content":"In pull-based (Volcano) execution, each operator pulls batches from its children by calling next(). In push-based execution, data flows downstream without operators needing to request it. DataFusion uses a pull-based streaming model with RecordBatch as the unit of work, which naturally supports backpressure and integrates well with Rust's async model."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#article-tips","content":"1. **Tip:** DataFusion's ` TableProvider ` trait is deceptively powerful. You can wrap literally any data source, S3 Parquet files, REST APIs, even gRPC streams, behind it, and the query optimizer just works. A custom provider for an internal metrics store can be built in about 200 lines of Rust.\n2. **Tip:** If your queries filter on a specific column 90% of the time, implement ` supports_filters_pushdown ` in your ` TableProvider `. The optimizer will push predicates down to your scan, and you can skip reading entire row groups. This alone can cut query latency by 4x.\n3. **Tip:** Don't write custom aggregation functions from scratch. Use DataFusion's ` Accumulator ` trait, it handles partitioning and merging across partitions automatically. Rolling a custom parallel HyperLogLog from scratch is unnecessary when the trait handles all the hard parts automatically.\n\n- arrow\n- datafusion\n- rust\n- query-engine\n- columnar\n- data-engineering\n- analytics\n- performance\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/arrow-datafusion-query-engine-rust"},{"text":"Apache Arrow DataFusion","url":"https://datafusion.apache.org/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust"},"headline":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","description":"A hands-on guide to Apache Arrow DataFusion, building custom query engines in Rust with columnar execution, push-based vs pull-based models, custom table providers, user-defined functions, and real-world performance tuning for analytical workloads.","datePublished":"2026-01-05","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Data Engineering","keywords":"arrow, datafusion, rust, query-engine, columnar, data-engineering, analytics, performance","wordCount":1305,"timeRequired":"PT7M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/arrow-datafusion-query-engine-rust"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Apache Arrow DataFusion: Building Custom Query Engines in Rust","item":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is Apache Arrow DataFusion?","acceptedAnswer":{"@type":"Answer","text":"DataFusion is an extensible query engine written in Rust, built on top of Apache Arrow's columnar memory format. It provides a SQL and DataFrame API, a cost-based query optimizer, and a parallel execution engine that you can embed in your own applications or extend with custom data sources, functions, and execution strategies."}},{"@type":"Question","name":"When should I use DataFusion instead of an existing database?","acceptedAnswer":{"@type":"Answer","text":"DataFusion excels when you need embedded analytics without a separate database server, custom data sources that don't fit traditional databases, or when you want to build domain-specific query engines. If you need transactions, multi-user concurrency, or full ACID guarantees, a traditional database is better."}},{"@type":"Question","name":"What's the difference between push-based and pull-based execution?","acceptedAnswer":{"@type":"Answer","text":"In pull-based (Volcano) execution, each operator pulls batches from its children by calling next(). In push-based execution, data flows downstream without operators needing to request it. DataFusion uses a pull-based streaming model with RecordBatch as the unit of work, which naturally supports backpressure and integrates well with Rust's async model."}}]}]}],"contentHash":"df16ecda767f0c5b223571c553396b798f69574fafcd88f04d9470c471da2d91","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/arrow-datafusion-query-engine-rust","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":67260,"contentBytes":14472}}