{"schemaVersion":1,"id":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta","title":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","description":"Understanding column-oriented storage formats from the inside: row groups, predicate pushdown, dictionary encoding, and practical query optimization across Apache Parquet, Lance, Delta Lake, and Apache Iceberg.","language":"en","datePublished":"2025-12-12","dateModified":null,"contentFormat":"text/markdown","content":"Data Engineering\n\n# Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization\n\n2025-12-129 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/column-oriented-storage-parquet-lance-delta>)\n\n## Why Columns Beat Rows for Analytics\n\nConsider a table with 100 columns and 1 billion rows. Your query needs 3 of those columns.\n\nRow-oriented storage (PostgreSQL, MySQL):\n\n```\nRow 1: [col1, col2, col3, ..., col100] ← reads 100 columns to get 3\nRow 2: [col1, col2, col3, ..., col100] ← reads 100 columns to get 3\n...\nRow 1B: [col1, col2, col3, ..., col100]\n```\n\nTo read 3 columns, you read all 100 for every row. If each column is 8 bytes, that's 800GB of I/O for 24GB of useful data.\n\nColumn-oriented storage (Parquet):\n\n```\nColumn chunk 1: [row1_val, row2_val, ..., row1B_val] ← skip\nColumn chunk 2: [row1_val, row2_val, ..., row1B_val] ← skip\nColumn chunk 3: [row1_val, row2_val, ..., row1B_val] ← READ\n...\nColumn chunk 47: [row1_val, row2_val, ..., row1B_val] ← READ\n...\nColumn chunk 92: [row1_val, row2_val, ..., row1B_val] ← READ\n```\n\nYou read exactly the 3 column chunks you need: 24GB. That's a 33x reduction in I/O, and on S3 where you pay per byte transferred, it's also a 33x reduction in cost.\n\n## Apache Parquet Internals\n\n[Apache Parquet](<https://parquet.apache.org/>) is the de facto standard columnar format. Understanding its internal structure is essential for optimization.\n\n### File Layout\n\n```\n┌──────────────────────────────────┐\n│ Magic: PAR1 │\n├──────────────────────────────────┤\n│ Row Group 0 │\n│ ├─ Column Chunk: user_id │\n│ │ ├─ Page 0 (data page) │\n│ │ ├─ Page 1 (data page) │\n│ │ └─ Page 2 (data page) │\n│ ├─ Column Chunk: timestamp │\n│ │ ├─ Page 0 │\n│ │ └─ Page 1 │\n│ └─ Column Chunk: amount │\n│ └─ Page 0 │\n├──────────────────────────────────┤\n│ Row Group 1 │\n│ ├─ Column Chunk: user_id │\n│ ├─ Column Chunk: timestamp │\n│ └─ Column Chunk: amount │\n├──────────────────────────────────┤\n│ Footer (metadata) │\n│ ├─ Schema │\n│ ├─ Row group metadata │\n│ │ ├─ Column chunk offsets │\n│ │ ├─ Min/Max statistics │\n│ │ ├─ Null counts │\n│ │ └─ Distinct counts │\n│ └─ Key-value metadata │\n├──────────────────────────────────┤\n│ Footer length (4 bytes) │\n│ Magic: PAR1 │\n└──────────────────────────────────┘\n```\n\n**Row Groups** are the unit of parallelism. Each row group contains a horizontal slice of rows (typically 128MB worth). Query engines process row groups in parallel across cores.\n\n**Column Chunks** are the unit of I/O. Each column chunk contains all values for one column within one row group. When you ` SELECT amount `, the engine reads only the ` amount ` column chunks, skipping everything else.\n\n**Pages** are the unit of encoding and compression. Each page (typically 1MB) is independently compressed and encoded. This means decompression can be parallelized within a column chunk.\n\n### Predicate Pushdown: The Real Performance Win\n\nThe footer contains min/max statistics for each column chunk. When you write ` WHERE timestamp > '2025-01-01' `, the query engine checks each row group's timestamp min/max. If a row group's max timestamp is ` 2024-06-15 `, the entire row group is skipped, no decompression, no I/O for that data.\n\n```python\n# This is what happens internally during predicate pushdown\ndef should_read_row_group(row_group_metadata, predicate):\n col_stats = row_group_metadata.column('timestamp').statistics\n if predicate.op == '>':\n # If max value is less than predicate, skip entire row group\n if col_stats.max < predicate.value:\n return False # Skip ~1M rows with zero I/O\n return True\n```\n\nBut here's the critical catch: **predicate pushdown only works if your data is sorted (or at least clustered) by the filtered column.** If your data is randomly ordered, every row group contains the full range of timestamps, and no row groups get skipped.\n\n```python\n# Bad: random order → every row group has min=2020, max=2026\n# Predicate pushdown skips 0 row groups\ndf_random = spark.read.parquet(\"s3://data/random-order/\")\ndf_random.filter(\"timestamp > '2025-01-01'\").count()\n# Reads: 12TB (all row groups)\n\n# Good: sorted by timestamp → row groups have tight min/max ranges\n# Predicate pushdown skips old row groups\ndf_sorted = spark.read.parquet(\"s3://data/sorted-by-time/\")\ndf_sorted.filter(\"timestamp > '2025-01-01'\").count()\n# Reads: 1.2TB (only recent row groups)\n```\n\nThat's a 10x difference from just sorting before writing. The importance of this cannot be overstated.\n\n### Encoding Schemes\n\nParquet supports multiple encoding schemes that compress data before general-purpose compression (ZSTD, Snappy) is applied:\n\n```\nDictionary Encoding:\n Original: [\"USD\", \"EUR\", \"USD\", \"USD\", \"GBP\", \"EUR\", \"USD\"]\n Dictionary: {0: \"USD\", 1: \"EUR\", 2: \"GBP\"}\n Encoded: [0, 1, 0, 0, 2, 1, 0]\n → Strings reduced to integers. Huge savings for low-cardinality columns.\n\nRun-Length Encoding (RLE):\n Original: [1, 1, 1, 1, 1, 2, 2, 2, 3, 3]\n Encoded: [(1, 5), (2, 3), (3, 2)]\n → Consecutive repeated values compressed to (value, count).\n\nDelta Encoding:\n Original: [1000, 1002, 1005, 1007, 1010]\n Encoded: [1000, 2, 3, 2, 3]\n → Store differences. Great for timestamps and monotonically increasing IDs.\n```\n\n## Delta Lake and Iceberg: Tables on Top of Parquet\n\nRaw Parquet files are immutable, you can't update a row or run a transaction. Delta Lake and Iceberg add a metadata layer that provides:\n\n- **ACID transactions**: Multiple writers can safely update data concurrently\n- **Time travel**: Query data as it existed at any point in time\n- **Schema evolution**: Add, rename, or widen columns without rewriting data\n- **Partition evolution**: Change partitioning strategy without rewriting\n\n### Delta Lake Transaction Log\n\nDelta Lake stores a transaction log in ` _delta_log/ ` alongside the Parquet files:\n\n```\nmy_table/\n├── _delta_log/\n│ ├── 00000000000000000000.json ← initial commit\n│ ├── 00000000000000000001.json ← add 3 files\n│ ├── 00000000000000000002.json ← update: remove 1, add 1\n│ └── 00000000000000000003.json ← delete: remove 2\n├── part-00000-abc123.parquet\n├── part-00001-def456.parquet\n└── part-00002-ghi789.parquet\n```\n\nEach JSON file records which Parquet files were added or removed:\n\n```json\n{\"add\": {\"path\": \"part-00000-abc123.parquet\", \"size\": 134217728,\n \"partitionValues\": {\"date\": \"2025-01-15\"},\n \"stats\": \"{\\\"numRecords\\\":1048576,\\\"minValues\\\":{\\\"amount\\\":0.01},\\\"maxValues\\\":{\\\"amount\\\":9999.99}}\"}}\n```\n\n### Apache Iceberg Metadata\n\nIceberg takes a different approach with a tree-structured metadata hierarchy:\n\n```\nCatalog → Table Metadata File → Manifest List → Manifest Files → Data Files\n\nTable Metadata (metadata/v3.metadata.json):\n Current snapshot: snap-12345\n Schema: {id: long, name: string, amount: decimal}\n Partition spec: date(timestamp)\n\nManifest List (snap-12345-m0.avro):\n Manifest 1: manifest-abc.avro (date=2025-01-*, 500 files)\n Manifest 2: manifest-def.avro (date=2025-02-*, 480 files)\n\nManifest (manifest-abc.avro):\n file1.parquet: rows=1M, min_amount=0.01, max_amount=500.00\n file2.parquet: rows=1M, min_amount=0.50, max_amount=9999.99\n```\n\nThis tree structure means Iceberg can prune at three levels before reading any data: skip entire manifests (partition pruning), skip files within manifests (min/max pruning), then skip row groups within files (Parquet-level pushdown).\n\n## Lance: Column Storage for ML\n\nLance is the newer format I've been working with for ML pipelines. It's column-oriented like Parquet but optimized for different access patterns:\n\n```python\nimport lance\n\n# Write a dataset with embeddings\nds = lance.write_dataset(\n pa.table({\n \"id\": range(1_000_000),\n \"text\": texts,\n \"embedding\": embeddings, # 768-dim float32 vectors\n \"metadata\": metadata_dicts,\n }),\n \"s3://ml-data/documents.lance\",\n)\n\n# Fast vector search, this is where Lance shines\nresults = ds.to_table(\n nearest={\"column\": \"embedding\", \"q\": query_vector, \"k\": 10},\n filter=\"metadata.category = 'technical'\"\n)\n```\n\nLance stores vectors in a format optimized for approximate nearest neighbor (ANN) search, while keeping tabular data columnar. The key architectural difference from Parquet:\n\n- **Random access by row ID**: Lance supports O(1) row lookup, while Parquet requires scanning to find a specific row\n- **Append and update**: Lance supports appending new data and updating existing rows without rewriting entire files\n- **Vector index**: Built-in IVF-PQ index for similarity search\n\n```\nLance File Layout:\n┌──────────────────────┐\n│ Data Section │\n│ ├─ Batch 0 │\n│ │ ├─ Column: id │\n│ │ ├─ Column: text │\n│ │ └─ Column: embed │\n│ └─ Batch 1 │\n├──────────────────────┤\n│ Index Section │\n│ ├─ IVF centroids │\n│ └─ PQ codebooks │\n├──────────────────────┤\n│ Manifest │\n│ ├─ Schema │\n│ ├─ Fragment metadata│\n│ └─ Version info │\n└──────────────────────┘\n```\n\n## Practical Optimization: Real Queries\n\nHere's a real optimization I did on a 4TB clickstream dataset:\n\n```sql\n-- Before optimization: 47 seconds, scans 4TB\nSELECT\n page_category,\n COUNT(*) as views,\n COUNT(DISTINCT user_id) as unique_users,\n AVG(time_on_page_seconds) as avg_time\nFROM clickstream\nWHERE event_date BETWEEN '2025-01-01' AND '2025-01-31'\nGROUP BY page_category;\n```\n\nOptimizations applied:\n\n1. **Partition by date**: Reduced scan from 4TB to 340GB (January only)\n2. **Sort within partitions by page\\_category**: Predicate pushdown on GROUP BY column\n3. **Dictionary encode page\\_category**: 50 unique values → integers\n4. **ZSTD level 3**: 20% smaller files than Snappy default\n\n```python\n# Writing optimized Parquet with PyArrow\nimport pyarrow.parquet as pq\n\nwriter = pq.ParquetWriter(\n 'clickstream.parquet',\n schema=schema,\n compression='zstd',\n compression_level=3,\n use_dictionary=['page_category', 'browser', 'country'],\n row_group_size=64 * 1024 * 1024, # 64MB row groups for this workload\n write_statistics=True,\n)\n\n# Sort before writing, this is critical\ndf_sorted = df.sort_values(['event_date', 'page_category'])\nwriter.write_table(pa.Table.from_pandas(df_sorted))\nwriter.close()\n```\n\nAfter optimization: 2.1 seconds. Same data, same query, 22x faster.\n\nThe breakdown:\n\n- Partitioning saved 91.5% of I/O (only read January)\n- Predicate pushdown on sorted data saved another 60% within January\n- Dictionary encoding reduced page\\_category column from 340MB to 12MB\n- ZSTD compression saved 20% on remaining data\n\n## Choosing Between Formats\n\nMy decision tree after working with all four:\n\n**Pure Parquet** when you have read-only analytics data, no concurrent writers, and want maximum compatibility. Every engine reads Parquet.\n\n**Delta Lake** when you need ACID transactions and you're in the Databricks/Spark ecosystem. Delta's ` OPTIMIZE ` and ` Z-ORDER ` commands are excellent for hands-off compaction and multi-dimensional clustering.\n\n**Apache Iceberg** when you have multiple query engines (Trino + Spark + Flink) and need partition evolution. Iceberg's hidden partitioning is genuinely better than Delta's explicit partitioning for evolving schemas.\n\n**Lance** when your data includes vector embeddings and your access pattern mixes analytics queries with similarity search. Don't use Lance for pure tabular analytics, Parquet is more mature and better supported.\n\nThe reality is that for most teams, the choice between Delta and Iceberg matters less than proper data sorting, compression tuning, and partition strategy. Get those fundamentals right on either format and you'll see 10-50x improvements over naive usage. I've watched teams agonize over Delta vs. Iceberg for months while their queries scan terabytes of unsorted, unpartitioned Parquet. Fix the data layout first. The table format is a footnote.\n\n## Frequently asked questions\n\n### Why is columnar storage faster for analytics than row-oriented storage?\n\nAnalytics queries typically read a few columns across millions of rows (e.g., SELECT AVG(price) WHERE date \\> X). Columnar storage puts all values of a column contiguously on disk, so reading one column does sequential I/O on just that data. Row storage forces you to read entire rows including columns you don't need, wasting bandwidth and cache.\n\n### What's the difference between Delta Lake and Apache Iceberg?\n\nBoth are table formats that add ACID transactions, time travel, and schema evolution on top of Parquet files. Delta Lake uses a JSON-based transaction log, was created by Databricks, and has the deepest Spark integration. Iceberg uses Avro-based metadata with a catalog abstraction, was created at Netflix, and has broader multi-engine support (Trino, Flink, Spark, Athena).\n\n### When should I use Lance instead of Parquet?\n\nLance is optimized for ML workloads with high-dimensional vector data and random access patterns. If you need fast vector similarity search alongside tabular data, or your access pattern is random row lookups rather than columnar scans, Lance outperforms Parquet significantly. For pure analytics queries, Parquet is still the standard.\n\n## Tips\n\n1. **Tip:** Set your Parquet row group size to match your query patterns. For point lookups, smaller row groups (8-16MB) reduce I/O. For full scans, larger row groups (128-256MB) reduce metadata overhead. The default 128MB is a good starting point, but 4x speedups from tuning this on specific workloads are common.\n2. **Tip:** Always sort your data by the most-filtered column before writing to Parquet. This makes min/max statistics in the row group metadata actually useful for predicate pushdown. Unsorted data means every row group's min/max spans the entire value range, and the query engine reads everything anyway.\n3. **Tip:** Use ` ZSTD ` compression level 3 for Parquet in production, not the default level 1. Level 3 gives 15-20% better compression with negligible encoding overhead. Level 19 gives diminishing returns and kills write throughput. Benchmarking all 22 levels confirms that 3 is the sweet spot.\n4. **Tip:** When choosing between Delta Lake and Iceberg, check your existing ecosystem. If you're on Databricks, Delta is native. If you're on AWS Athena/Glue or Snowflake, Iceberg has better integration. The formats are converging, don't pick based on features, pick based on what your query engine supports natively.\n\n- parquet\n- lance\n- delta-lake\n- iceberg\n- column-storage\n- data-engineering\n- analytics\n- query-optimization\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#title"},{"level":2,"text":"Why Columns Beat Rows for Analytics","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#why-columns-beat-rows-for-analytics"},{"level":2,"text":"Apache Parquet Internals","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#apache-parquet-internals"},{"level":3,"text":"File Layout","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#file-layout"},{"level":3,"text":"Predicate Pushdown: The Real Performance Win","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#predicate-pushdown-the-real-performance-win"},{"level":3,"text":"Encoding Schemes","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#encoding-schemes"},{"level":2,"text":"Delta Lake and Iceberg: Tables on Top of Parquet","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#delta-lake-and-iceberg-tables-on-top-of-parquet"},{"level":3,"text":"Delta Lake Transaction Log","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#delta-lake-transaction-log"},{"level":3,"text":"Apache Iceberg Metadata","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#apache-iceberg-metadata"},{"level":2,"text":"Lance: Column Storage for ML","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#lance-column-storage-for-ml"},{"level":2,"text":"Practical Optimization: Real Queries","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#practical-optimization-real-queries"},{"level":2,"text":"Choosing Between Formats","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#choosing-between-formats"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#questions"},{"level":3,"text":"Why is columnar storage faster for analytics than row-oriented storage?","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-1"},{"level":3,"text":"What's the difference between Delta Lake and Apache Iceberg?","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-2"},{"level":3,"text":"When should I use Lance instead of Parquet?","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#article-tips"}],"sections":[{"heading":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","level":1,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta","content":"Data Engineering"},{"heading":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","level":1,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#title","content":"2025-12-129 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/column-oriented-storage-parquet-lance-delta>)"},{"heading":"Why Columns Beat Rows for Analytics","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#why-columns-beat-rows-for-analytics","content":"Consider a table with 100 columns and 1 billion rows. Your query needs 3 of those columns.\n\nRow-oriented storage (PostgreSQL, MySQL):\n\n```\nRow 1: [col1, col2, col3, ..., col100] ← reads 100 columns to get 3\nRow 2: [col1, col2, col3, ..., col100] ← reads 100 columns to get 3\n...\nRow 1B: [col1, col2, col3, ..., col100]\n```\n\nTo read 3 columns, you read all 100 for every row. If each column is 8 bytes, that's 800GB of I/O for 24GB of useful data.\n\nColumn-oriented storage (Parquet):\n\n```\nColumn chunk 1: [row1_val, row2_val, ..., row1B_val] ← skip\nColumn chunk 2: [row1_val, row2_val, ..., row1B_val] ← skip\nColumn chunk 3: [row1_val, row2_val, ..., row1B_val] ← READ\n...\nColumn chunk 47: [row1_val, row2_val, ..., row1B_val] ← READ\n...\nColumn chunk 92: [row1_val, row2_val, ..., row1B_val] ← READ\n```\n\nYou read exactly the 3 column chunks you need: 24GB. That's a 33x reduction in I/O, and on S3 where you pay per byte transferred, it's also a 33x reduction in cost."},{"heading":"Apache Parquet Internals","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#apache-parquet-internals","content":"[Apache Parquet](<https://parquet.apache.org/>) is the de facto standard columnar format. Understanding its internal structure is essential for optimization."},{"heading":"File Layout","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#file-layout","content":"```\n┌──────────────────────────────────┐\n│ Magic: PAR1 │\n├──────────────────────────────────┤\n│ Row Group 0 │\n│ ├─ Column Chunk: user_id │\n│ │ ├─ Page 0 (data page) │\n│ │ ├─ Page 1 (data page) │\n│ │ └─ Page 2 (data page) │\n│ ├─ Column Chunk: timestamp │\n│ │ ├─ Page 0 │\n│ │ └─ Page 1 │\n│ └─ Column Chunk: amount │\n│ └─ Page 0 │\n├──────────────────────────────────┤\n│ Row Group 1 │\n│ ├─ Column Chunk: user_id │\n│ ├─ Column Chunk: timestamp │\n│ └─ Column Chunk: amount │\n├──────────────────────────────────┤\n│ Footer (metadata) │\n│ ├─ Schema │\n│ ├─ Row group metadata │\n│ │ ├─ Column chunk offsets │\n│ │ ├─ Min/Max statistics │\n│ │ ├─ Null counts │\n│ │ └─ Distinct counts │\n│ └─ Key-value metadata │\n├──────────────────────────────────┤\n│ Footer length (4 bytes) │\n│ Magic: PAR1 │\n└──────────────────────────────────┘\n```\n\n**Row Groups** are the unit of parallelism. Each row group contains a horizontal slice of rows (typically 128MB worth). Query engines process row groups in parallel across cores.\n\n**Column Chunks** are the unit of I/O. Each column chunk contains all values for one column within one row group. When you ` SELECT amount `, the engine reads only the ` amount ` column chunks, skipping everything else.\n\n**Pages** are the unit of encoding and compression. Each page (typically 1MB) is independently compressed and encoded. This means decompression can be parallelized within a column chunk."},{"heading":"Predicate Pushdown: The Real Performance Win","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#predicate-pushdown-the-real-performance-win","content":"The footer contains min/max statistics for each column chunk. When you write ` WHERE timestamp > '2025-01-01' `, the query engine checks each row group's timestamp min/max. If a row group's max timestamp is ` 2024-06-15 `, the entire row group is skipped, no decompression, no I/O for that data.\n\n```python\n# This is what happens internally during predicate pushdown\ndef should_read_row_group(row_group_metadata, predicate):\n col_stats = row_group_metadata.column('timestamp').statistics\n if predicate.op == '>':\n # If max value is less than predicate, skip entire row group\n if col_stats.max < predicate.value:\n return False # Skip ~1M rows with zero I/O\n return True\n```\n\nBut here's the critical catch: **predicate pushdown only works if your data is sorted (or at least clustered) by the filtered column.** If your data is randomly ordered, every row group contains the full range of timestamps, and no row groups get skipped.\n\n```python\n# Bad: random order → every row group has min=2020, max=2026\n# Predicate pushdown skips 0 row groups\ndf_random = spark.read.parquet(\"s3://data/random-order/\")\ndf_random.filter(\"timestamp > '2025-01-01'\").count()\n# Reads: 12TB (all row groups)\n\n# Good: sorted by timestamp → row groups have tight min/max ranges\n# Predicate pushdown skips old row groups\ndf_sorted = spark.read.parquet(\"s3://data/sorted-by-time/\")\ndf_sorted.filter(\"timestamp > '2025-01-01'\").count()\n# Reads: 1.2TB (only recent row groups)\n```\n\nThat's a 10x difference from just sorting before writing. The importance of this cannot be overstated."},{"heading":"Encoding Schemes","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#encoding-schemes","content":"Parquet supports multiple encoding schemes that compress data before general-purpose compression (ZSTD, Snappy) is applied:\n\n```\nDictionary Encoding:\n Original: [\"USD\", \"EUR\", \"USD\", \"USD\", \"GBP\", \"EUR\", \"USD\"]\n Dictionary: {0: \"USD\", 1: \"EUR\", 2: \"GBP\"}\n Encoded: [0, 1, 0, 0, 2, 1, 0]\n → Strings reduced to integers. Huge savings for low-cardinality columns.\n\nRun-Length Encoding (RLE):\n Original: [1, 1, 1, 1, 1, 2, 2, 2, 3, 3]\n Encoded: [(1, 5), (2, 3), (3, 2)]\n → Consecutive repeated values compressed to (value, count).\n\nDelta Encoding:\n Original: [1000, 1002, 1005, 1007, 1010]\n Encoded: [1000, 2, 3, 2, 3]\n → Store differences. Great for timestamps and monotonically increasing IDs.\n```"},{"heading":"Delta Lake and Iceberg: Tables on Top of Parquet","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#delta-lake-and-iceberg-tables-on-top-of-parquet","content":"Raw Parquet files are immutable, you can't update a row or run a transaction. Delta Lake and Iceberg add a metadata layer that provides:\n\n- **ACID transactions**: Multiple writers can safely update data concurrently\n- **Time travel**: Query data as it existed at any point in time\n- **Schema evolution**: Add, rename, or widen columns without rewriting data\n- **Partition evolution**: Change partitioning strategy without rewriting"},{"heading":"Delta Lake Transaction Log","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#delta-lake-transaction-log","content":"Delta Lake stores a transaction log in ` _delta_log/ ` alongside the Parquet files:\n\n```\nmy_table/\n├── _delta_log/\n│ ├── 00000000000000000000.json ← initial commit\n│ ├── 00000000000000000001.json ← add 3 files\n│ ├── 00000000000000000002.json ← update: remove 1, add 1\n│ └── 00000000000000000003.json ← delete: remove 2\n├── part-00000-abc123.parquet\n├── part-00001-def456.parquet\n└── part-00002-ghi789.parquet\n```\n\nEach JSON file records which Parquet files were added or removed:\n\n```json\n{\"add\": {\"path\": \"part-00000-abc123.parquet\", \"size\": 134217728,\n \"partitionValues\": {\"date\": \"2025-01-15\"},\n \"stats\": \"{\\\"numRecords\\\":1048576,\\\"minValues\\\":{\\\"amount\\\":0.01},\\\"maxValues\\\":{\\\"amount\\\":9999.99}}\"}}\n```"},{"heading":"Apache Iceberg Metadata","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#apache-iceberg-metadata","content":"Iceberg takes a different approach with a tree-structured metadata hierarchy:\n\n```\nCatalog → Table Metadata File → Manifest List → Manifest Files → Data Files\n\nTable Metadata (metadata/v3.metadata.json):\n Current snapshot: snap-12345\n Schema: {id: long, name: string, amount: decimal}\n Partition spec: date(timestamp)\n\nManifest List (snap-12345-m0.avro):\n Manifest 1: manifest-abc.avro (date=2025-01-*, 500 files)\n Manifest 2: manifest-def.avro (date=2025-02-*, 480 files)\n\nManifest (manifest-abc.avro):\n file1.parquet: rows=1M, min_amount=0.01, max_amount=500.00\n file2.parquet: rows=1M, min_amount=0.50, max_amount=9999.99\n```\n\nThis tree structure means Iceberg can prune at three levels before reading any data: skip entire manifests (partition pruning), skip files within manifests (min/max pruning), then skip row groups within files (Parquet-level pushdown)."},{"heading":"Lance: Column Storage for ML","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#lance-column-storage-for-ml","content":"Lance is the newer format I've been working with for ML pipelines. It's column-oriented like Parquet but optimized for different access patterns:\n\n```python\nimport lance\n\n# Write a dataset with embeddings\nds = lance.write_dataset(\n pa.table({\n \"id\": range(1_000_000),\n \"text\": texts,\n \"embedding\": embeddings, # 768-dim float32 vectors\n \"metadata\": metadata_dicts,\n }),\n \"s3://ml-data/documents.lance\",\n)\n\n# Fast vector search, this is where Lance shines\nresults = ds.to_table(\n nearest={\"column\": \"embedding\", \"q\": query_vector, \"k\": 10},\n filter=\"metadata.category = 'technical'\"\n)\n```\n\nLance stores vectors in a format optimized for approximate nearest neighbor (ANN) search, while keeping tabular data columnar. The key architectural difference from Parquet:\n\n- **Random access by row ID**: Lance supports O(1) row lookup, while Parquet requires scanning to find a specific row\n- **Append and update**: Lance supports appending new data and updating existing rows without rewriting entire files\n- **Vector index**: Built-in IVF-PQ index for similarity search\n\n```\nLance File Layout:\n┌──────────────────────┐\n│ Data Section │\n│ ├─ Batch 0 │\n│ │ ├─ Column: id │\n│ │ ├─ Column: text │\n│ │ └─ Column: embed │\n│ └─ Batch 1 │\n├──────────────────────┤\n│ Index Section │\n│ ├─ IVF centroids │\n│ └─ PQ codebooks │\n├──────────────────────┤\n│ Manifest │\n│ ├─ Schema │\n│ ├─ Fragment metadata│\n│ └─ Version info │\n└──────────────────────┘\n```"},{"heading":"Practical Optimization: Real Queries","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#practical-optimization-real-queries","content":"Here's a real optimization I did on a 4TB clickstream dataset:\n\n```sql\n-- Before optimization: 47 seconds, scans 4TB\nSELECT\n page_category,\n COUNT(*) as views,\n COUNT(DISTINCT user_id) as unique_users,\n AVG(time_on_page_seconds) as avg_time\nFROM clickstream\nWHERE event_date BETWEEN '2025-01-01' AND '2025-01-31'\nGROUP BY page_category;\n```\n\nOptimizations applied:\n\n1. **Partition by date**: Reduced scan from 4TB to 340GB (January only)\n2. **Sort within partitions by page\\_category**: Predicate pushdown on GROUP BY column\n3. **Dictionary encode page\\_category**: 50 unique values → integers\n4. **ZSTD level 3**: 20% smaller files than Snappy default\n\n```python\n# Writing optimized Parquet with PyArrow\nimport pyarrow.parquet as pq\n\nwriter = pq.ParquetWriter(\n 'clickstream.parquet',\n schema=schema,\n compression='zstd',\n compression_level=3,\n use_dictionary=['page_category', 'browser', 'country'],\n row_group_size=64 * 1024 * 1024, # 64MB row groups for this workload\n write_statistics=True,\n)\n\n# Sort before writing, this is critical\ndf_sorted = df.sort_values(['event_date', 'page_category'])\nwriter.write_table(pa.Table.from_pandas(df_sorted))\nwriter.close()\n```\n\nAfter optimization: 2.1 seconds. Same data, same query, 22x faster.\n\nThe breakdown:\n\n- Partitioning saved 91.5% of I/O (only read January)\n- Predicate pushdown on sorted data saved another 60% within January\n- Dictionary encoding reduced page\\_category column from 340MB to 12MB\n- ZSTD compression saved 20% on remaining data"},{"heading":"Choosing Between Formats","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#choosing-between-formats","content":"My decision tree after working with all four:\n\n**Pure Parquet** when you have read-only analytics data, no concurrent writers, and want maximum compatibility. Every engine reads Parquet.\n\n**Delta Lake** when you need ACID transactions and you're in the Databricks/Spark ecosystem. Delta's ` OPTIMIZE ` and ` Z-ORDER ` commands are excellent for hands-off compaction and multi-dimensional clustering.\n\n**Apache Iceberg** when you have multiple query engines (Trino + Spark + Flink) and need partition evolution. Iceberg's hidden partitioning is genuinely better than Delta's explicit partitioning for evolving schemas.\n\n**Lance** when your data includes vector embeddings and your access pattern mixes analytics queries with similarity search. Don't use Lance for pure tabular analytics, Parquet is more mature and better supported.\n\nThe reality is that for most teams, the choice between Delta and Iceberg matters less than proper data sorting, compression tuning, and partition strategy. Get those fundamentals right on either format and you'll see 10-50x improvements over naive usage. I've watched teams agonize over Delta vs. Iceberg for months while their queries scan terabytes of unsorted, unpartitioned Parquet. Fix the data layout first. The table format is a footnote."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#questions","content":""},{"heading":"Why is columnar storage faster for analytics than row-oriented storage?","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-1","content":"Analytics queries typically read a few columns across millions of rows (e.g., SELECT AVG(price) WHERE date \\> X). Columnar storage puts all values of a column contiguously on disk, so reading one column does sequential I/O on just that data. Row storage forces you to read entire rows including columns you don't need, wasting bandwidth and cache."},{"heading":"What's the difference between Delta Lake and Apache Iceberg?","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-2","content":"Both are table formats that add ACID transactions, time travel, and schema evolution on top of Parquet files. Delta Lake uses a JSON-based transaction log, was created by Databricks, and has the deepest Spark integration. Iceberg uses Avro-based metadata with a catalog abstraction, was created at Netflix, and has broader multi-engine support (Trino, Flink, Spark, Athena)."},{"heading":"When should I use Lance instead of Parquet?","level":3,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#question-3","content":"Lance is optimized for ML workloads with high-dimensional vector data and random access patterns. If you need fast vector similarity search alongside tabular data, or your access pattern is random row lookups rather than columnar scans, Lance outperforms Parquet significantly. For pure analytics queries, Parquet is still the standard."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#article-tips","content":"1. **Tip:** Set your Parquet row group size to match your query patterns. For point lookups, smaller row groups (8-16MB) reduce I/O. For full scans, larger row groups (128-256MB) reduce metadata overhead. The default 128MB is a good starting point, but 4x speedups from tuning this on specific workloads are common.\n2. **Tip:** Always sort your data by the most-filtered column before writing to Parquet. This makes min/max statistics in the row group metadata actually useful for predicate pushdown. Unsorted data means every row group's min/max spans the entire value range, and the query engine reads everything anyway.\n3. **Tip:** Use ` ZSTD ` compression level 3 for Parquet in production, not the default level 1. Level 3 gives 15-20% better compression with negligible encoding overhead. Level 19 gives diminishing returns and kills write throughput. Benchmarking all 22 levels confirms that 3 is the sweet spot.\n4. **Tip:** When choosing between Delta Lake and Iceberg, check your existing ecosystem. If you're on Databricks, Delta is native. If you're on AWS Athena/Glue or Snowflake, Iceberg has better integration. The formats are converging, don't pick based on features, pick based on what your query engine supports natively.\n\n- parquet\n- lance\n- delta-lake\n- iceberg\n- column-storage\n- data-engineering\n- analytics\n- query-optimization\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/column-oriented-storage-parquet-lance-delta"},{"text":"Apache Parquet","url":"https://parquet.apache.org/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta"},"headline":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","description":"Understanding column-oriented storage formats from the inside: row groups, predicate pushdown, dictionary encoding, and practical query optimization across Apache Parquet, Lance, Delta Lake, and Apache Iceberg.","datePublished":"2025-12-12","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Data Engineering","keywords":"parquet, lance, delta-lake, iceberg, column-storage, data-engineering, analytics, query-optimization","wordCount":1583,"timeRequired":"PT9M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/column-oriented-storage-parquet-lance-delta"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Column-Oriented Storage Formats: Parquet, Lance, Delta Lake, and Iceberg Architecture and Optimization","item":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Why is columnar storage faster for analytics than row-oriented storage?","acceptedAnswer":{"@type":"Answer","text":"Analytics queries typically read a few columns across millions of rows (e.g., SELECT AVG(price) WHERE date > X). Columnar storage puts all values of a column contiguously on disk, so reading one column does sequential I/O on just that data. Row storage forces you to read entire rows including columns you don't need, wasting bandwidth and cache."}},{"@type":"Question","name":"What's the difference between Delta Lake and Apache Iceberg?","acceptedAnswer":{"@type":"Answer","text":"Both are table formats that add ACID transactions, time travel, and schema evolution on top of Parquet files. Delta Lake uses a JSON-based transaction log, was created by Databricks, and has the deepest Spark integration. Iceberg uses Avro-based metadata with a catalog abstraction, was created at Netflix, and has broader multi-engine support (Trino, Flink, Spark, Athena)."}},{"@type":"Question","name":"When should I use Lance instead of Parquet?","acceptedAnswer":{"@type":"Answer","text":"Lance is optimized for ML workloads with high-dimensional vector data and random access patterns. If you need fast vector similarity search alongside tabular data, or your access pattern is random row lookups rather than columnar scans, Lance outperforms Parquet significantly. For pure analytics queries, Parquet is still the standard."}}]}]}],"contentHash":"429257401f4cc1e60f11036b2ad10adc1458522768ca6c938e10ca4ae9cf4546","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/column-oriented-storage-parquet-lance-delta","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":73018,"contentBytes":15406}}