{"schemaVersion":1,"id":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx","title":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","description":"Building a real IoT data pipeline that handles 1M inserts/sec, comparing TimescaleDB hypertables, QuestDB's columnar engine, and InfluxDB IOx with actual benchmarks and production gotchas.","language":"en","datePublished":"2026-01-10","dateModified":null,"contentFormat":"text/markdown","content":"Databases\n\n# Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx\n\n2026-01-109 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/time-series-databases-timescaledb-questdb-influx>)\n\n## The Problem with General-Purpose Databases for Time Series\n\nTime series data has specific characteristics that make it hostile to traditional RDBMS:\n\n1. **Append-mostly writes**: Data arrives chronologically and rarely gets updated.\n2. **Time-range queries dominate**: \"Give me the last hour of data\" is 95% of queries.\n3. **Massive cardinality**: 50K sensors x 4 metrics x 1 sample/sec = 200K rows/sec.\n4. **Old data loses value**: Last hour is hot, last month is warm, last year is cold.\n\nPostgreSQL's B-tree indexes are designed for random access. When you have 4 billion rows sorted by time, inserting new data at the end of the index while querying the most recent partition creates enormous I/O contention. The WAL amplification alone was killing their disk throughput.\n\n## TimescaleDB: PostgreSQL's Time Series Extension\n\n[TimescaleDB](<https://www.timescaledb.com/>) is the pragmatic choice. It's a PostgreSQL extension, so you keep your SQL, your connection libraries, your JOINs with relational data, your backup tooling, everything. The magic is in hypertables.\n\n### Hypertables and Chunks\n\nA hypertable looks like a regular PostgreSQL table, but under the hood it's automatically partitioned into chunks by time interval:\n\n```sql\n-- Create the hypertable\nCREATE TABLE sensor_data (\n time TIMESTAMPTZ NOT NULL,\n sensor_id TEXT NOT NULL,\n temperature DOUBLE PRECISION,\n humidity DOUBLE PRECISION,\n pressure DOUBLE PRECISION,\n vibration DOUBLE PRECISION\n);\n\n-- Convert to hypertable with 6-hour chunks\nSELECT create_hypertable('sensor_data', 'time',\n chunk_time_interval => INTERVAL '6 hours'\n);\n\n-- Add a space partition for parallel inserts\nSELECT add_dimension('sensor_data', 'sensor_id',\n number_partitions => 16\n);\n\n-- Compression policy: compress chunks older than 24 hours\nALTER TABLE sensor_data SET (\n timescaledb.compress,\n timescaledb.compress_segmentby = 'sensor_id',\n timescaledb.compress_orderby = 'time DESC'\n);\n\nSELECT add_compression_policy('sensor_data', INTERVAL '24 hours');\n\n-- Retention policy: drop chunks older than 90 days\nSELECT add_retention_policy('sensor_data', INTERVAL '90 days');\n```\n\nThe chunk time interval is critical. Each chunk is its own PostgreSQL table with its own indexes and its own storage. When you query ` WHERE time > now() - INTERVAL '1 hour' `, TimescaleDB only touches the relevant chunks. On a 50K sensor deployment, that turned a full table scan of 4 billion rows into a scan of about 720K rows.\n\n### Continuous Aggregates\n\nThis is TimescaleDB's killer feature for dashboards. Instead of recomputing aggregates on every query, you materialize them:\n\n```sql\n-- Create a materialized view that auto-refreshes\nCREATE MATERIALIZED VIEW sensor_hourly\nWITH (timescaledb.continuous) AS\nSELECT\n time_bucket('1 hour', time) AS bucket,\n sensor_id,\n avg(temperature) AS avg_temp,\n min(temperature) AS min_temp,\n max(temperature) AS max_temp,\n avg(humidity) AS avg_humidity,\n count(*) AS sample_count\nFROM sensor_data\nGROUP BY bucket, sensor_id;\n\n-- Refresh policy: recompute last 2 hours every 30 minutes\nSELECT add_continuous_aggregate_policy('sensor_hourly',\n start_offset => INTERVAL '2 hours',\n end_offset => INTERVAL '30 minutes',\n schedule_interval => INTERVAL '30 minutes'\n);\n\n-- Query the aggregate instead of raw data\nSELECT bucket, avg_temp, min_temp, max_temp\nFROM sensor_hourly\nWHERE sensor_id = 'sensor-42'\n AND bucket > now() - INTERVAL '7 days'\nORDER BY bucket;\n```\n\nDashboard queries that took 8 seconds on raw data now return in 12ms. That's not a typo.\n\n## QuestDB: Built for Ingestion Speed\n\n[QuestDB](<https://questdb.io/>) is a columnar time series database written in Java and C++. It speaks PostgreSQL wire protocol and InfluxDB Line Protocol, but internally it's a completely different engine optimized for append-only time series workloads.\n\n### The Ingestion Pipeline\n\nQuestDB's InfluxDB Line Protocol (ILP) interface over TCP is the fastest way to ingest data:\n\n```python\n# questdb_ingest.py - high-throughput ingestion\nfrom questdb.ingress import Sender, IngressError, TimestampNanos\nimport time\nimport random\n\ndef generate_sensor_batch(batch_size: int, sensors: list[str]):\n \"\"\"Generate a batch of sensor readings.\"\"\"\n now_ns = time.time_ns()\n readings = []\n for i in range(batch_size):\n readings.append({\n 'sensor_id': random.choice(sensors),\n 'temperature': 20.0 + random.gauss(0, 5),\n 'humidity': 50.0 + random.gauss(0, 10),\n 'pressure': 1013.25 + random.gauss(0, 2),\n 'vibration': abs(random.gauss(0, 0.5)),\n 'timestamp': now_ns + (i * 1_000_000) # 1ms apart\n })\n return readings\n\ndef ingest_batch(sender: Sender, readings: list[dict]):\n for r in readings:\n sender.row(\n 'sensor_data',\n symbols={'sensor_id': r['sensor_id']},\n columns={\n 'temperature': r['temperature'],\n 'humidity': r['humidity'],\n 'pressure': r['pressure'],\n 'vibration': r['vibration'],\n },\n at=TimestampNanos(r['timestamp'])\n )\n sender.flush()\n\n# Main ingestion loop\nsensors = [f'sensor-{i:05d}' for i in range(50_000)]\n\nconf = 'http::addr=localhost:9000;auto_flush_rows=50000;auto_flush_interval=1000;'\nwith Sender.from_conf(conf) as sender:\n while True:\n batch = generate_sensor_batch(50_000, sensors)\n ingest_batch(sender, batch)\n # On a 4-core machine, this saturates at ~1.2M rows/sec\n```\n\n### QuestDB's SQL Extensions\n\nQuestDB adds time series-specific SQL syntax that's genuinely useful:\n\n```sql\n-- SAMPLE BY: time-bucketed aggregation (like time_bucket but native)\nSELECT\n sensor_id,\n avg(temperature) avg_temp,\n min(temperature) min_temp,\n max(temperature) max_temp,\n count() sample_count\nFROM sensor_data\nWHERE sensor_id = 'sensor-00042'\nSAMPLE BY 1h\nORDER BY timestamp;\n\n-- LATEST ON: get the most recent value per sensor (extremely fast)\nSELECT * FROM sensor_data\nLATEST ON timestamp PARTITION BY sensor_id;\n\n-- ASOF JOIN: join time series without exact timestamp matches\nSELECT\n s.timestamp,\n s.sensor_id,\n s.temperature,\n a.alert_type,\n a.threshold\nFROM sensor_data s\nASOF JOIN alerts a ON (s.sensor_id = a.sensor_id);\n```\n\nThe ` LATEST ON ` query is worth the price of admission alone. Getting the current state of 50K sensors in PostgreSQL requires either a subquery with ` DISTINCT ON ` (slow) or maintaining a separate \"current state\" table (complex). QuestDB does it in single-digit milliseconds because it tracks the latest row per partition natively.\n\n## InfluxDB IOx: The Storage Engine Rewrite\n\nInfluxDB 3.x (IOx engine) is a ground-up rewrite in Rust using Apache Arrow and DataFusion. It replaces the TSM storage engine with a columnar Parquet-based system that's fundamentally different from InfluxDB 1.x/2.x.\n\n```python\n# influxdb3_ingest.py\nfrom influxdb_client_3 import InfluxDBClient3\nimport pandas as pd\nimport numpy as np\n\nclient = InfluxDBClient3(\n host=\"https://us-east-1-1.aws.cloud2.influxdata.com\",\n token=\"your-token\",\n org=\"your-org\",\n database=\"iot_pipeline\"\n)\n\n# Batch ingestion with line protocol\ndef write_batch(sensors: list[str], batch_size: int = 50_000):\n lines = []\n for _ in range(batch_size):\n sensor = np.random.choice(sensors)\n temp = 20.0 + np.random.normal(0, 5)\n humidity = 50.0 + np.random.normal(0, 10)\n pressure = 1013.25 + np.random.normal(0, 2)\n vibration = abs(np.random.normal(0, 0.5))\n ts = int(time.time() * 1e9)\n lines.append(\n f'sensor_data,sensor_id={sensor} '\n f'temperature={temp:.2f},humidity={humidity:.2f},'\n f'pressure={pressure:.4f},vibration={vibration:.4f} {ts}'\n )\n client.write('\\n'.join(lines))\n\n# Query with SQL (backed by DataFusion)\ndf = client.query(\"\"\"\n SELECT\n date_bin('1 hour', time) AS bucket,\n sensor_id,\n avg(temperature) AS avg_temp,\n approx_percentile_cont(temperature, 0.95) AS p95_temp,\n count(*) AS samples\n FROM sensor_data\n WHERE time > now() - INTERVAL '24 hours'\n AND sensor_id IN ('sensor-00001', 'sensor-00042', 'sensor-01337')\n GROUP BY bucket, sensor_id\n ORDER BY bucket DESC\n\"\"\").to_pandas()\n```\n\nThe big deal with IOx is ` approx_percentile_cont `. Computing percentiles on time series data is something you need all the time for SLO monitoring, but it's expensive in traditional databases. IOx uses t-digest sketches under the hood, giving you approximate percentiles in constant memory.\n\n## The Benchmark: 1M Inserts/sec\n\nA proper benchmark was set up on three identical [AWS](<https://aws.com/>) r6g.2xlarge instances (8 vCPU, 64GB RAM, gp3 SSD at 16,000 IOPS). Each database got the same workload: 50K simulated sensors, 4 float64 metrics each, 1-second reporting interval. Target: 200K rows/sec sustained, burst to 1M.\n\n```\nSustained ingestion (200K rows/sec, 24h test):\n┌──────────────┬─────────────┬───────────┬────────────┬─────────────┐\n│ Database │ Avg Latency │ p99 Lat │ CPU Usage │ Disk Write │\n├──────────────┼─────────────┼───────────┼────────────┼─────────────┤\n│ TimescaleDB │ 4.2ms/batch │ 18ms │ 45% │ 380 MB/hr │\n│ QuestDB │ 1.1ms/batch │ 4.8ms │ 28% │ 210 MB/hr │\n│ InfluxDB IOx │ 3.8ms/batch │ 22ms │ 38% │ 290 MB/hr │\n└──────────────┴─────────────┴───────────┴────────────┴─────────────┘\n\nBurst ingestion (1M rows/sec, 10-minute burst):\n┌──────────────┬──────────────┬─────────────┬──────────────────────┐\n│ Database │ Sustained? │ Peak Actual │ Notes │\n├──────────────┼──────────────┼─────────────┼──────────────────────┤\n│ TimescaleDB │ Yes (w/ tuning) │ 1.05M │ Needed 16 space │\n│ │ │ │ partitions + 8 │\n│ │ │ │ parallel writers │\n│ QuestDB │ Yes │ 1.4M │ Single node, WAL │\n│ │ │ │ mode, no special │\n│ │ │ │ tuning │\n│ InfluxDB IOx │ Mostly │ 920K │ Hit compaction │\n│ │ │ │ pressure at ~950K │\n└──────────────┴──────────────┴─────────────┴──────────────────────┘\n```\n\nQuestDB won the raw ingestion benchmark by a wide margin. On a single node with no special configuration beyond enabling WAL mode, it sustained 1.4M inserts/sec. TimescaleDB needed careful tuning of space partitions and parallel writers but got there. InfluxDB IOx hit compaction backpressure around 950K rows/sec, though this may improve as the IOx engine matures.\n\n### Query Performance\n\nIngestion speed is only half the story. Here's query performance on the same dataset after 24 hours of sustained ingestion (about 17 billion rows):\n\n```\nQuery: \"Average temperature by sensor, last 1 hour\"\n┌──────────────┬──────────────┬───────────────────────────┐\n│ Database │ Response Time│ Notes │\n├──────────────┼──────────────┼───────────────────────────┤\n│ TimescaleDB │ 340ms │ Using continuous aggregate│\n│ │ 2.8s │ Raw query, no aggregate │\n│ QuestDB │ 180ms │ SAMPLE BY, native │\n│ InfluxDB IOx │ 220ms │ DataFusion, columnar scan │\n└──────────────┴──────────────┴───────────────────────────┘\n\nQuery: \"Latest value per sensor (50K sensors)\"\n┌──────────────┬──────────────┬───────────────────────────┐\n│ Database │ Response Time│ Notes │\n├──────────────┼──────────────┼───────────────────────────┤\n│ TimescaleDB │ 4.2s │ DISTINCT ON, painful │\n│ QuestDB │ 8ms │ LATEST ON, instant │\n│ InfluxDB IOx │ 1.8s │ GROUP BY + last() │\n└──────────────┴──────────────┴───────────────────────────┘\n```\n\nThat ` LATEST ON ` result from QuestDB is real. 8ms to get the current state of 50,000 sensors from 17 billion rows. It maintains an internal index of the latest row per partition key, so the query doesn't scan data at all.\n\n## The Production Pipeline\n\nHere is the final architecture for a representative IoT project:\n\n```\nSensors (50K)\n → MQTT Broker (EMQX cluster)\n → Kafka (3-node, 12 partitions)\n → QuestDB (ingestion, hot data, last 7 days)\n → TimescaleDB (analytics, warm data, last 90 days, continuous aggregates)\n → S3 Parquet export (cold data, historical archive)\n```\n\n```python\n# kafka_to_questdb.py - consumer that bridges Kafka to QuestDB\nfrom confluent_kafka import Consumer\nfrom questdb.ingress import Sender, TimestampNanos\nimport json\n\nconsumer = Consumer({\n 'bootstrap.servers': 'kafka:9092',\n 'group.id': 'questdb-ingestor',\n 'auto.offset.reset': 'latest',\n 'fetch.min.bytes': 65536,\n 'fetch.wait.max.ms': 100,\n})\nconsumer.subscribe(['sensor-data'])\n\nconf = 'tcp::addr=questdb:9009;init_buf_size=65536;'\nwith Sender.from_conf(conf) as sender:\n while True:\n messages = consumer.consume(num_messages=5000, timeout=0.1)\n for msg in messages:\n if msg.error():\n continue\n data = json.loads(msg.value())\n sender.row(\n 'sensor_data',\n symbols={'sensor_id': data['sensor_id']},\n columns={\n 'temperature': data['temperature'],\n 'humidity': data['humidity'],\n 'pressure': data['pressure'],\n 'vibration': data['vibration'],\n },\n at=TimestampNanos(data['timestamp_ns'])\n )\n sender.flush()\n consumer.commit()\n```\n\nQuestDB handles hot data (last 7 days) because of its ingestion speed and ` LATEST ON ` performance for real-time dashboards. TimescaleDB handled the analytical layer with continuous aggregates feeding Grafana dashboards. Old data got exported to Parquet on S3 for cost efficiency.\n\n## What Would Be Different Today\n\nStarting this project today, it would be worth seriously considering running only QuestDB with its newer enterprise features for the hot path, and using InfluxDB IOx for the analytical layer since the DataFusion engine handles complex analytics better than QuestDB's SQL subset. TimescaleDB remains the recommendation when joins are needed with relational data, it's PostgreSQL, after all, but for pure time series workloads, the purpose-built engines have a clear performance advantage.\n\nIn one deployment, the pipeline ran for fourteen months without a major incident. The DBA sleeps through the night. That is the best benchmark there is.\n\n## Frequently asked questions\n\n### Can I use PostgreSQL instead of a dedicated time series database?\n\nFor low-volume use cases (under 10K inserts/sec), PostgreSQL with proper partitioning and indexes works fine. TimescaleDB is literally PostgreSQL with extensions, so you're not abandoning the ecosystem. But once you hit high ingestion rates, the columnar storage engines in QuestDB and InfluxDB IOx genuinely outperform row-oriented stores.\n\n### Which time series database is best for IoT?\n\nQuestDB for raw ingestion speed on a single node. TimescaleDB if you need SQL compatibility and the PostgreSQL ecosystem. InfluxDB if your team already uses the TICK stack or you need built-in downsampling and retention policies.\n\n### How do I handle backfilling historical data?\n\nTimescaleDB handles this natively since hypertable chunks are just partitions. QuestDB requires data to be roughly time-ordered for optimal performance, use the 'o3' (out-of-order) commit lag setting. InfluxDB IOx handles out-of-order writes well but at the cost of more compaction work.\n\n## Tips\n\n1. **Tip:** TimescaleDB's ` chunk_time_interval ` should match your most common query range. If you query last 24h mostly, set chunks to 12h, not 1 week.\n2. **Tip:** QuestDB's ` WAL ` mode is mandatory for concurrent writes. Without it, you're limited to a single writer thread. Set ` cairo.wal.enabled.default=true ` in server.conf.\n3. **Tip:** Never use InfluxDB tags for high-cardinality fields. If your tag has more than 100K unique values, it should be a field. This single mistake has killed more InfluxDB deployments than anything else.\n4. **Tip:** For all three databases, batch your inserts. Going from row-at-a-time to batches of 5000 typically gives you a ` 50-100x ` throughput improvement.\n\n- time-series\n- timescaledb\n- questdb\n- influxdb\n- iot\n- data-pipeline\n- performance\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#title"},{"level":2,"text":"The Problem with General-Purpose Databases for Time Series","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-problem-with-general-purpose-databases-for-time-series"},{"level":2,"text":"TimescaleDB: PostgreSQL's Time Series Extension","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#timescaledb-postgresqls-time-series-extension"},{"level":3,"text":"Hypertables and Chunks","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#hypertables-and-chunks"},{"level":3,"text":"Continuous Aggregates","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#continuous-aggregates"},{"level":2,"text":"QuestDB: Built for Ingestion Speed","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questdb-built-for-ingestion-speed"},{"level":3,"text":"The Ingestion Pipeline","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-ingestion-pipeline"},{"level":3,"text":"QuestDB's SQL Extensions","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questdbs-sql-extensions"},{"level":2,"text":"InfluxDB IOx: The Storage Engine Rewrite","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#influxdb-iox-the-storage-engine-rewrite"},{"level":2,"text":"The Benchmark: 1M Inserts/sec","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-benchmark-1m-insertssec"},{"level":3,"text":"Query Performance","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#query-performance"},{"level":2,"text":"The Production Pipeline","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-production-pipeline"},{"level":2,"text":"What Would Be Different Today","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#what-would-be-different-today"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questions"},{"level":3,"text":"Can I use PostgreSQL instead of a dedicated time series database?","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-1"},{"level":3,"text":"Which time series database is best for IoT?","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-2"},{"level":3,"text":"How do I handle backfilling historical data?","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#article-tips"}],"sections":[{"heading":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","level":1,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx","content":"Databases"},{"heading":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","level":1,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#title","content":"2026-01-109 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/time-series-databases-timescaledb-questdb-influx>)"},{"heading":"The Problem with General-Purpose Databases for Time Series","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-problem-with-general-purpose-databases-for-time-series","content":"Time series data has specific characteristics that make it hostile to traditional RDBMS:\n\n1. **Append-mostly writes**: Data arrives chronologically and rarely gets updated.\n2. **Time-range queries dominate**: \"Give me the last hour of data\" is 95% of queries.\n3. **Massive cardinality**: 50K sensors x 4 metrics x 1 sample/sec = 200K rows/sec.\n4. **Old data loses value**: Last hour is hot, last month is warm, last year is cold.\n\nPostgreSQL's B-tree indexes are designed for random access. When you have 4 billion rows sorted by time, inserting new data at the end of the index while querying the most recent partition creates enormous I/O contention. The WAL amplification alone was killing their disk throughput."},{"heading":"TimescaleDB: PostgreSQL's Time Series Extension","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#timescaledb-postgresqls-time-series-extension","content":"[TimescaleDB](<https://www.timescaledb.com/>) is the pragmatic choice. It's a PostgreSQL extension, so you keep your SQL, your connection libraries, your JOINs with relational data, your backup tooling, everything. The magic is in hypertables."},{"heading":"Hypertables and Chunks","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#hypertables-and-chunks","content":"A hypertable looks like a regular PostgreSQL table, but under the hood it's automatically partitioned into chunks by time interval:\n\n```sql\n-- Create the hypertable\nCREATE TABLE sensor_data (\n time TIMESTAMPTZ NOT NULL,\n sensor_id TEXT NOT NULL,\n temperature DOUBLE PRECISION,\n humidity DOUBLE PRECISION,\n pressure DOUBLE PRECISION,\n vibration DOUBLE PRECISION\n);\n\n-- Convert to hypertable with 6-hour chunks\nSELECT create_hypertable('sensor_data', 'time',\n chunk_time_interval => INTERVAL '6 hours'\n);\n\n-- Add a space partition for parallel inserts\nSELECT add_dimension('sensor_data', 'sensor_id',\n number_partitions => 16\n);\n\n-- Compression policy: compress chunks older than 24 hours\nALTER TABLE sensor_data SET (\n timescaledb.compress,\n timescaledb.compress_segmentby = 'sensor_id',\n timescaledb.compress_orderby = 'time DESC'\n);\n\nSELECT add_compression_policy('sensor_data', INTERVAL '24 hours');\n\n-- Retention policy: drop chunks older than 90 days\nSELECT add_retention_policy('sensor_data', INTERVAL '90 days');\n```\n\nThe chunk time interval is critical. Each chunk is its own PostgreSQL table with its own indexes and its own storage. When you query ` WHERE time > now() - INTERVAL '1 hour' `, TimescaleDB only touches the relevant chunks. On a 50K sensor deployment, that turned a full table scan of 4 billion rows into a scan of about 720K rows."},{"heading":"Continuous Aggregates","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#continuous-aggregates","content":"This is TimescaleDB's killer feature for dashboards. Instead of recomputing aggregates on every query, you materialize them:\n\n```sql\n-- Create a materialized view that auto-refreshes\nCREATE MATERIALIZED VIEW sensor_hourly\nWITH (timescaledb.continuous) AS\nSELECT\n time_bucket('1 hour', time) AS bucket,\n sensor_id,\n avg(temperature) AS avg_temp,\n min(temperature) AS min_temp,\n max(temperature) AS max_temp,\n avg(humidity) AS avg_humidity,\n count(*) AS sample_count\nFROM sensor_data\nGROUP BY bucket, sensor_id;\n\n-- Refresh policy: recompute last 2 hours every 30 minutes\nSELECT add_continuous_aggregate_policy('sensor_hourly',\n start_offset => INTERVAL '2 hours',\n end_offset => INTERVAL '30 minutes',\n schedule_interval => INTERVAL '30 minutes'\n);\n\n-- Query the aggregate instead of raw data\nSELECT bucket, avg_temp, min_temp, max_temp\nFROM sensor_hourly\nWHERE sensor_id = 'sensor-42'\n AND bucket > now() - INTERVAL '7 days'\nORDER BY bucket;\n```\n\nDashboard queries that took 8 seconds on raw data now return in 12ms. That's not a typo."},{"heading":"QuestDB: Built for Ingestion Speed","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questdb-built-for-ingestion-speed","content":"[QuestDB](<https://questdb.io/>) is a columnar time series database written in Java and C++. It speaks PostgreSQL wire protocol and InfluxDB Line Protocol, but internally it's a completely different engine optimized for append-only time series workloads."},{"heading":"The Ingestion Pipeline","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-ingestion-pipeline","content":"QuestDB's InfluxDB Line Protocol (ILP) interface over TCP is the fastest way to ingest data:\n\n```python\n# questdb_ingest.py - high-throughput ingestion\nfrom questdb.ingress import Sender, IngressError, TimestampNanos\nimport time\nimport random\n\ndef generate_sensor_batch(batch_size: int, sensors: list[str]):\n \"\"\"Generate a batch of sensor readings.\"\"\"\n now_ns = time.time_ns()\n readings = []\n for i in range(batch_size):\n readings.append({\n 'sensor_id': random.choice(sensors),\n 'temperature': 20.0 + random.gauss(0, 5),\n 'humidity': 50.0 + random.gauss(0, 10),\n 'pressure': 1013.25 + random.gauss(0, 2),\n 'vibration': abs(random.gauss(0, 0.5)),\n 'timestamp': now_ns + (i * 1_000_000) # 1ms apart\n })\n return readings\n\ndef ingest_batch(sender: Sender, readings: list[dict]):\n for r in readings:\n sender.row(\n 'sensor_data',\n symbols={'sensor_id': r['sensor_id']},\n columns={\n 'temperature': r['temperature'],\n 'humidity': r['humidity'],\n 'pressure': r['pressure'],\n 'vibration': r['vibration'],\n },\n at=TimestampNanos(r['timestamp'])\n )\n sender.flush()\n\n# Main ingestion loop\nsensors = [f'sensor-{i:05d}' for i in range(50_000)]\n\nconf = 'http::addr=localhost:9000;auto_flush_rows=50000;auto_flush_interval=1000;'\nwith Sender.from_conf(conf) as sender:\n while True:\n batch = generate_sensor_batch(50_000, sensors)\n ingest_batch(sender, batch)\n # On a 4-core machine, this saturates at ~1.2M rows/sec\n```"},{"heading":"QuestDB's SQL Extensions","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questdbs-sql-extensions","content":"QuestDB adds time series-specific SQL syntax that's genuinely useful:\n\n```sql\n-- SAMPLE BY: time-bucketed aggregation (like time_bucket but native)\nSELECT\n sensor_id,\n avg(temperature) avg_temp,\n min(temperature) min_temp,\n max(temperature) max_temp,\n count() sample_count\nFROM sensor_data\nWHERE sensor_id = 'sensor-00042'\nSAMPLE BY 1h\nORDER BY timestamp;\n\n-- LATEST ON: get the most recent value per sensor (extremely fast)\nSELECT * FROM sensor_data\nLATEST ON timestamp PARTITION BY sensor_id;\n\n-- ASOF JOIN: join time series without exact timestamp matches\nSELECT\n s.timestamp,\n s.sensor_id,\n s.temperature,\n a.alert_type,\n a.threshold\nFROM sensor_data s\nASOF JOIN alerts a ON (s.sensor_id = a.sensor_id);\n```\n\nThe ` LATEST ON ` query is worth the price of admission alone. Getting the current state of 50K sensors in PostgreSQL requires either a subquery with ` DISTINCT ON ` (slow) or maintaining a separate \"current state\" table (complex). QuestDB does it in single-digit milliseconds because it tracks the latest row per partition natively."},{"heading":"InfluxDB IOx: The Storage Engine Rewrite","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#influxdb-iox-the-storage-engine-rewrite","content":"InfluxDB 3.x (IOx engine) is a ground-up rewrite in Rust using Apache Arrow and DataFusion. It replaces the TSM storage engine with a columnar Parquet-based system that's fundamentally different from InfluxDB 1.x/2.x.\n\n```python\n# influxdb3_ingest.py\nfrom influxdb_client_3 import InfluxDBClient3\nimport pandas as pd\nimport numpy as np\n\nclient = InfluxDBClient3(\n host=\"https://us-east-1-1.aws.cloud2.influxdata.com\",\n token=\"your-token\",\n org=\"your-org\",\n database=\"iot_pipeline\"\n)\n\n# Batch ingestion with line protocol\ndef write_batch(sensors: list[str], batch_size: int = 50_000):\n lines = []\n for _ in range(batch_size):\n sensor = np.random.choice(sensors)\n temp = 20.0 + np.random.normal(0, 5)\n humidity = 50.0 + np.random.normal(0, 10)\n pressure = 1013.25 + np.random.normal(0, 2)\n vibration = abs(np.random.normal(0, 0.5))\n ts = int(time.time() * 1e9)\n lines.append(\n f'sensor_data,sensor_id={sensor} '\n f'temperature={temp:.2f},humidity={humidity:.2f},'\n f'pressure={pressure:.4f},vibration={vibration:.4f} {ts}'\n )\n client.write('\\n'.join(lines))\n\n# Query with SQL (backed by DataFusion)\ndf = client.query(\"\"\"\n SELECT\n date_bin('1 hour', time) AS bucket,\n sensor_id,\n avg(temperature) AS avg_temp,\n approx_percentile_cont(temperature, 0.95) AS p95_temp,\n count(*) AS samples\n FROM sensor_data\n WHERE time > now() - INTERVAL '24 hours'\n AND sensor_id IN ('sensor-00001', 'sensor-00042', 'sensor-01337')\n GROUP BY bucket, sensor_id\n ORDER BY bucket DESC\n\"\"\").to_pandas()\n```\n\nThe big deal with IOx is ` approx_percentile_cont `. Computing percentiles on time series data is something you need all the time for SLO monitoring, but it's expensive in traditional databases. IOx uses t-digest sketches under the hood, giving you approximate percentiles in constant memory."},{"heading":"The Benchmark: 1M Inserts/sec","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-benchmark-1m-insertssec","content":"A proper benchmark was set up on three identical [AWS](<https://aws.com/>) r6g.2xlarge instances (8 vCPU, 64GB RAM, gp3 SSD at 16,000 IOPS). Each database got the same workload: 50K simulated sensors, 4 float64 metrics each, 1-second reporting interval. Target: 200K rows/sec sustained, burst to 1M.\n\n```\nSustained ingestion (200K rows/sec, 24h test):\n┌──────────────┬─────────────┬───────────┬────────────┬─────────────┐\n│ Database │ Avg Latency │ p99 Lat │ CPU Usage │ Disk Write │\n├──────────────┼─────────────┼───────────┼────────────┼─────────────┤\n│ TimescaleDB │ 4.2ms/batch │ 18ms │ 45% │ 380 MB/hr │\n│ QuestDB │ 1.1ms/batch │ 4.8ms │ 28% │ 210 MB/hr │\n│ InfluxDB IOx │ 3.8ms/batch │ 22ms │ 38% │ 290 MB/hr │\n└──────────────┴─────────────┴───────────┴────────────┴─────────────┘\n\nBurst ingestion (1M rows/sec, 10-minute burst):\n┌──────────────┬──────────────┬─────────────┬──────────────────────┐\n│ Database │ Sustained? │ Peak Actual │ Notes │\n├──────────────┼──────────────┼─────────────┼──────────────────────┤\n│ TimescaleDB │ Yes (w/ tuning) │ 1.05M │ Needed 16 space │\n│ │ │ │ partitions + 8 │\n│ │ │ │ parallel writers │\n│ QuestDB │ Yes │ 1.4M │ Single node, WAL │\n│ │ │ │ mode, no special │\n│ │ │ │ tuning │\n│ InfluxDB IOx │ Mostly │ 920K │ Hit compaction │\n│ │ │ │ pressure at ~950K │\n└──────────────┴──────────────┴─────────────┴──────────────────────┘\n```\n\nQuestDB won the raw ingestion benchmark by a wide margin. On a single node with no special configuration beyond enabling WAL mode, it sustained 1.4M inserts/sec. TimescaleDB needed careful tuning of space partitions and parallel writers but got there. InfluxDB IOx hit compaction backpressure around 950K rows/sec, though this may improve as the IOx engine matures."},{"heading":"Query Performance","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#query-performance","content":"Ingestion speed is only half the story. Here's query performance on the same dataset after 24 hours of sustained ingestion (about 17 billion rows):\n\n```\nQuery: \"Average temperature by sensor, last 1 hour\"\n┌──────────────┬──────────────┬───────────────────────────┐\n│ Database │ Response Time│ Notes │\n├──────────────┼──────────────┼───────────────────────────┤\n│ TimescaleDB │ 340ms │ Using continuous aggregate│\n│ │ 2.8s │ Raw query, no aggregate │\n│ QuestDB │ 180ms │ SAMPLE BY, native │\n│ InfluxDB IOx │ 220ms │ DataFusion, columnar scan │\n└──────────────┴──────────────┴───────────────────────────┘\n\nQuery: \"Latest value per sensor (50K sensors)\"\n┌──────────────┬──────────────┬───────────────────────────┐\n│ Database │ Response Time│ Notes │\n├──────────────┼──────────────┼───────────────────────────┤\n│ TimescaleDB │ 4.2s │ DISTINCT ON, painful │\n│ QuestDB │ 8ms │ LATEST ON, instant │\n│ InfluxDB IOx │ 1.8s │ GROUP BY + last() │\n└──────────────┴──────────────┴───────────────────────────┘\n```\n\nThat ` LATEST ON ` result from QuestDB is real. 8ms to get the current state of 50,000 sensors from 17 billion rows. It maintains an internal index of the latest row per partition key, so the query doesn't scan data at all."},{"heading":"The Production Pipeline","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#the-production-pipeline","content":"Here is the final architecture for a representative IoT project:\n\n```\nSensors (50K)\n → MQTT Broker (EMQX cluster)\n → Kafka (3-node, 12 partitions)\n → QuestDB (ingestion, hot data, last 7 days)\n → TimescaleDB (analytics, warm data, last 90 days, continuous aggregates)\n → S3 Parquet export (cold data, historical archive)\n```\n\n```python\n# kafka_to_questdb.py - consumer that bridges Kafka to QuestDB\nfrom confluent_kafka import Consumer\nfrom questdb.ingress import Sender, TimestampNanos\nimport json\n\nconsumer = Consumer({\n 'bootstrap.servers': 'kafka:9092',\n 'group.id': 'questdb-ingestor',\n 'auto.offset.reset': 'latest',\n 'fetch.min.bytes': 65536,\n 'fetch.wait.max.ms': 100,\n})\nconsumer.subscribe(['sensor-data'])\n\nconf = 'tcp::addr=questdb:9009;init_buf_size=65536;'\nwith Sender.from_conf(conf) as sender:\n while True:\n messages = consumer.consume(num_messages=5000, timeout=0.1)\n for msg in messages:\n if msg.error():\n continue\n data = json.loads(msg.value())\n sender.row(\n 'sensor_data',\n symbols={'sensor_id': data['sensor_id']},\n columns={\n 'temperature': data['temperature'],\n 'humidity': data['humidity'],\n 'pressure': data['pressure'],\n 'vibration': data['vibration'],\n },\n at=TimestampNanos(data['timestamp_ns'])\n )\n sender.flush()\n consumer.commit()\n```\n\nQuestDB handles hot data (last 7 days) because of its ingestion speed and ` LATEST ON ` performance for real-time dashboards. TimescaleDB handled the analytical layer with continuous aggregates feeding Grafana dashboards. Old data got exported to Parquet on S3 for cost efficiency."},{"heading":"What Would Be Different Today","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#what-would-be-different-today","content":"Starting this project today, it would be worth seriously considering running only QuestDB with its newer enterprise features for the hot path, and using InfluxDB IOx for the analytical layer since the DataFusion engine handles complex analytics better than QuestDB's SQL subset. TimescaleDB remains the recommendation when joins are needed with relational data, it's PostgreSQL, after all, but for pure time series workloads, the purpose-built engines have a clear performance advantage.\n\nIn one deployment, the pipeline ran for fourteen months without a major incident. The DBA sleeps through the night. That is the best benchmark there is."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#questions","content":""},{"heading":"Can I use PostgreSQL instead of a dedicated time series database?","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-1","content":"For low-volume use cases (under 10K inserts/sec), PostgreSQL with proper partitioning and indexes works fine. TimescaleDB is literally PostgreSQL with extensions, so you're not abandoning the ecosystem. But once you hit high ingestion rates, the columnar storage engines in QuestDB and InfluxDB IOx genuinely outperform row-oriented stores."},{"heading":"Which time series database is best for IoT?","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-2","content":"QuestDB for raw ingestion speed on a single node. TimescaleDB if you need SQL compatibility and the PostgreSQL ecosystem. InfluxDB if your team already uses the TICK stack or you need built-in downsampling and retention policies."},{"heading":"How do I handle backfilling historical data?","level":3,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#question-3","content":"TimescaleDB handles this natively since hypertable chunks are just partitions. QuestDB requires data to be roughly time-ordered for optimal performance, use the 'o3' (out-of-order) commit lag setting. InfluxDB IOx handles out-of-order writes well but at the cost of more compaction work."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#article-tips","content":"1. **Tip:** TimescaleDB's ` chunk_time_interval ` should match your most common query range. If you query last 24h mostly, set chunks to 12h, not 1 week.\n2. **Tip:** QuestDB's ` WAL ` mode is mandatory for concurrent writes. Without it, you're limited to a single writer thread. Set ` cairo.wal.enabled.default=true ` in server.conf.\n3. **Tip:** Never use InfluxDB tags for high-cardinality fields. If your tag has more than 100K unique values, it should be a field. This single mistake has killed more InfluxDB deployments than anything else.\n4. **Tip:** For all three databases, batch your inserts. Going from row-at-a-time to batches of 5000 typically gives you a ` 50-100x ` throughput improvement.\n\n- time-series\n- timescaledb\n- questdb\n- influxdb\n- iot\n- data-pipeline\n- performance\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/time-series-databases-timescaledb-questdb-influx"},{"text":"TimescaleDB","url":"https://www.timescaledb.com/"},{"text":"QuestDB","url":"https://questdb.io/"},{"text":"AWS","url":"https://aws.com/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx"},"headline":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","description":"Building a real IoT data pipeline that handles 1M inserts/sec, comparing TimescaleDB hypertables, QuestDB's columnar engine, and InfluxDB IOx with actual benchmarks and production gotchas.","datePublished":"2026-01-10","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Databases","keywords":"time-series, timescaledb, questdb, influxdb, iot, data-pipeline, performance","wordCount":1713,"timeRequired":"PT9M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/time-series-databases-timescaledb-questdb-influx"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Time Series Databases: TimescaleDB Hypertables, QuestDB Columnar Ingestion, and InfluxDB IOx","item":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Can I use PostgreSQL instead of a dedicated time series database?","acceptedAnswer":{"@type":"Answer","text":"For low-volume use cases (under 10K inserts/sec), PostgreSQL with proper partitioning and indexes works fine. TimescaleDB is literally PostgreSQL with extensions, so you're not abandoning the ecosystem. But once you hit high ingestion rates, the columnar storage engines in QuestDB and InfluxDB IOx genuinely outperform row-oriented stores."}},{"@type":"Question","name":"Which time series database is best for IoT?","acceptedAnswer":{"@type":"Answer","text":"QuestDB for raw ingestion speed on a single node. TimescaleDB if you need SQL compatibility and the PostgreSQL ecosystem. InfluxDB if your team already uses the TICK stack or you need built-in downsampling and retention policies."}},{"@type":"Question","name":"How do I handle backfilling historical data?","acceptedAnswer":{"@type":"Answer","text":"TimescaleDB handles this natively since hypertable chunks are just partitions. QuestDB requires data to be roughly time-ordered for optimal performance, use the 'o3' (out-of-order) commit lag setting. InfluxDB IOx handles out-of-order writes well but at the cost of more compaction work."}}]}]}],"contentHash":"660b88926c34733b52e7de04d85e34c060e836cadcee67f5d2d90fd644494a49","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/time-series-databases-timescaledb-questdb-influx","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":75496,"contentBytes":17390}}