{"schemaVersion":1,"id":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication","title":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","description":"A practical deep dive into PostgreSQL 17's most impactful features: incremental backup with pg_basebackup, logical replication failover, new JSON functions, pg_stat_io insights, and real migration strategies from PG 15/16.","language":"en","datePublished":"2025-10-14","dateModified":null,"contentFormat":"text/markdown","content":"Databases\n\n# PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements\n\n2025-10-1410 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/postgresql-17-incremental-backup-logical-replication>)\n\n## Incremental Backup: Native Support in PG 17\n\nBefore PG 17, ` pg_basebackup ` always took a full backup. Every time. A 500GB database meant copying 500GB, whether you changed 10 rows or 10 million since the last backup. The alternatives were third-party tools like pgBackRest (excellent, but another thing to operate) or filesystem-level snapshots (which have their own issues).\n\nPG 17 adds native incremental backup support. Here's how it works:\n\n```bash\n# Step 1: Enable WAL summarization (required, do this first!)\n# postgresql.conf\nsummarize_wal = on\n\n# Restart PostgreSQL for the change to take effect\nsudo systemctl restart postgresql@17-main\n\n# Step 2: Take the first full backup (this is the baseline)\npg_basebackup -D /backups/full_20260114 \\\n --checkpoint=fast \\\n --wal-method=stream \\\n --manifest-checksums=SHA256 \\\n -U replication_user\n\n# Step 3: Take incremental backups (only changed blocks)\npg_basebackup -D /backups/incr_20260115 \\\n --incremental=/backups/full_20260114/backup_manifest \\\n --checkpoint=fast \\\n --wal-method=stream \\\n -U replication_user\n```\n\nThe ` --incremental ` flag points to the manifest file from the previous backup. PostgreSQL compares WAL summaries to determine which 8KB blocks have changed since that manifest was created, and only transfers those blocks.\n\n**Real-world numbers from a representative production database (480GB):**\n\n| Backup type | Size | Duration | Notes |\n| --- | --- | --- | --- |\n| Full backup | 480GB | 47 min | Baseline |\n| Day 1 incremental | 8.2GB | 4.2 min | ~1.7% daily change rate |\n| Day 2 incremental | 9.1GB | 4.8 min | Slightly busier day |\n| Day 7 incremental | 7.6GB | 3.9 min | Weekend, less activity |\n| Weekly synthetic full | 480GB | 12 min | Reconstructed from chain |\n\nThat last row is important. You can reconstruct a full backup from an incremental chain using ` pg_combinebackup `:\n\n```bash\n# Combine full + incrementals into a synthetic full backup\npg_combinebackup \\\n /backups/full_20260114 \\\n /backups/incr_20260115 \\\n /backups/incr_20260116 \\\n /backups/incr_20260117 \\\n -o /backups/synthetic_full_20260117\n\n# The synthetic full can be used as a new baseline\n# for future incremental chains\n```\n\nThe reconstruction takes ~12 minutes for a 480GB database because it's mostly sequential reads and writes, no network transfer needed if all backups are on the same storage.\n\n**The WAL summarization gotcha:** The ` summarize_wal ` setting creates summary files in ` pg_wal/summaries/ `. These are small (a few KB each), but they're only created for WAL that's generated *after* the setting is enabled. If you enable it and immediately try an incremental backup, it'll fail because there's no summary covering the period between your previous full backup and now. Enable it first, take a fresh full backup, then start your incremental chain.\n\n## Logical Replication Failover: Finally\n\nPG 17 adds ` failover = true ` on subscriptions and ` sync ` parameter on replication slots. This means logical replication slots can survive a failover to a standby. Before this, if your primary went down and you promoted a standby, all logical replication subscribers had to be re-initialized from scratch. On a 500GB database, that's hours of downtime.\n\nHere's the setup:\n\n```sql\n-- On the primary: create a publication\nCREATE PUBLICATION my_pub FOR ALL TABLES;\n\n-- On the subscriber: create subscription with failover support\nCREATE SUBSCRIPTION my_sub\n CONNECTION 'host=primary-host port=5432 dbname=mydb'\n PUBLICATION my_pub\n WITH (failover = true);\n```\n\nOn the primary, you'll see the replication slot has ` failover = true `:\n\n```sql\nSELECT slot_name, plugin, failover, synced\nFROM pg_replication_slots\nWHERE slot_name = 'my_sub';\n\n-- slot_name | plugin | failover | synced\n-- -----------+---------+----------+--------\n-- my_sub | pgoutput| t | f\n```\n\nThe ` synced ` column becomes ` true ` once the standby has synchronized the slot position. When failover happens, the new primary has the exact LSN position of the subscriber, so replication resumes without data loss or re-sync.\n\nThe configuration on the standby that enables this:\n\n```\n# postgresql.conf on standby\nsync_replication_slots = on\nhot_standby_feedback = on\n```\n\nTesting with a simulated failover (promoting the standby via ` pg_ctl promote `) shows the subscriber reconnects and resumes replication within 3 seconds. Before PG 17, the same scenario required dropping and recreating the subscription, which meant a full table sync, 2 hours for a comparable dataset.\n\n## JSON\\_TABLE: SQL/JSON Done Right\n\nWriting ` jsonb_array_elements ` with lateral joins has long felt like a workaround. PG 17 implements ` JSON_TABLE() ` from the SQL/JSON standard, and it's substantially cleaner:\n\n```sql\n-- The old way (PG 16 and below)\nSELECT\n e->>'id' AS id,\n e->>'name' AS name,\n e->>'email' AS email\nFROM orders o,\n LATERAL jsonb_array_elements(o.metadata->'contacts') AS e\nWHERE o.created_at > '2026-01-01';\n\n-- The new way (PG 17)\nSELECT t.*\nFROM orders o,\n JSON_TABLE(\n o.metadata,\n '$.contacts[*]'\n COLUMNS (\n id INTEGER PATH '$.id',\n name TEXT PATH '$.name',\n email TEXT PATH '$.email'\n )\n ) AS t\nWHERE o.created_at > '2026-01-01';\n```\n\nBut the real power is in nested paths and error handling:\n\n```sql\n-- Nested JSON_TABLE with error handling\nSELECT t.*\nFROM api_responses r,\n JSON_TABLE(\n r.body,\n '$.data.users[*]'\n COLUMNS (\n user_id INTEGER PATH '$.id',\n username TEXT PATH '$.profile.name'\n DEFAULT 'unknown' ON EMPTY\n DEFAULT 'error' ON ERROR,\n NESTED PATH '$.orders[*]' COLUMNS (\n order_id INTEGER PATH '$.id',\n total NUMERIC(10,2) PATH '$.total',\n items_count INTEGER PATH '$.items.size()'\n )\n )\n ) AS t;\n```\n\nThe ` ON EMPTY ` and ` ON ERROR ` clauses are game-changers for dealing with inconsistent JSON structures. No more ` COALESCE(jsonb_extract_path_text(...), 'default') ` chains.\n\nPerformance-wise, ` JSON_TABLE ` isn't magic, it still does the same work under the hood. But the query planner can reason about it better because it sees a single table-returning function instead of a lateral join chain. On a query hitting 50,000 rows with an average of 8 JSON array elements each, benchmarks show a 22% improvement in execution time versus the equivalent ` jsonb_array_elements ` approach.\n\n## pg\\_stat\\_io: Understanding Your I/O Patterns\n\nThis view was added in PG 16, but PG 17 expanded it significantly with better backend type categorization and extend statistics. It has become an essential diagnostic tool.\n\n```sql\n-- Top I/O consumers by backend type\nSELECT\n backend_type,\n io_object,\n io_context,\n reads,\n read_bytes / (1024*1024*1024.0) AS read_gb,\n writes,\n write_bytes / (1024*1024*1024.0) AS write_gb,\n extends,\n fsyncs\nFROM pg_stat_io\nWHERE reads > 0 OR writes > 0\nORDER BY reads + writes DESC\nLIMIT 20;\n```\n\nWhat this reveals on a representative production server:\n\n```\n backend_type | io_object | io_context | reads | read_gb | writes | write_gb\n-------------------+-----------+------------+----------+---------+----------+---------\n autovacuum worker | relation | normal | 14822041 | 112.8 | 2841023 | 21.6\n client backend | relation | normal | 4201882 | 32.0 | 891204 | 6.8\n client backend | relation | bulkread | 3102441 | 23.6 | 0 | 0.0\n background writer | relation | normal | 0 | 0.0 | 1204812 | 9.2\n checkpointer | relation | normal | 0 | 0.0 | 982104 | 7.5\n```\n\nAutovacuum was doing 3.5x more reads than all client queries combined. That is not necessarily wrong, autovacuum is doing its job, but it indicates the ` autovacuum_vacuum_cost_delay ` was too aggressive and vacuum was scanning tables that did not need it.\n\nAfter tuning:\n\n```sql\n-- Per-table autovacuum settings for large, mostly-static tables\nALTER TABLE historical_events SET (\n autovacuum_vacuum_threshold = 10000,\n autovacuum_vacuum_scale_factor = 0.01,\n autovacuum_analyze_threshold = 5000\n);\n```\n\nAutovacuum reads dropped by 60%, and client query I/O performance improved measurably because there was less contention on the buffer pool.\n\n## VACUUM Improvements: 8x Fewer Page Reads\n\nPG 17 fundamentally changed how VACUUM handles pages. The previous behavior was to read every page in a table to check for dead tuples, even if the page was already all-frozen (every tuple visible to all transactions). PG 17 tracks frozen-page status in the visibility map and skips those pages entirely.\n\nFor a large table with a high frozen-to-total page ratio (common for append-heavy tables like event logs), this is transformative:\n\n```sql\n-- Check your table's frozen page ratio\nSELECT\n relname,\n pg_size_pretty(pg_relation_size(oid)) AS table_size,\n n_tup_ins AS inserts_since_vacuum,\n n_tup_upd AS updates_since_vacuum,\n n_dead_tup AS dead_tuples,\n last_autovacuum\nFROM pg_stat_user_tables\nWHERE relname = 'events'\nORDER BY pg_relation_size(oid) DESC;\n```\n\nOn a representative ` events ` table (89GB, 95% frozen pages), VACUUM time dropped from 18 minutes to 2.2 minutes. The I/O reduction was even more dramatic, from 89GB of reads to under 5GB.\n\n## The Migration Path: PG 15/16 → PG 17\n\nThis migration has been performed multiple times across different environments. Here is the playbook that works:\n\n### Option A: pg\\_upgrade (simplest, brief downtime)\n\n```bash\n# 1. Install PG 17 alongside existing version\nsudo apt install postgresql-17\n\n# 2. Stop both clusters\nsudo systemctl stop postgresql@16-main\nsudo systemctl stop postgresql@17-main\n\n# 3. Run pg_upgrade in check mode first\nsudo -u postgres pg_upgrade \\\n --old-datadir=/var/lib/postgresql/16/main \\\n --new-datadir=/var/lib/postgresql/17/main \\\n --old-bindir=/usr/lib/postgresql/16/bin \\\n --new-bindir=/usr/lib/postgresql/17/bin \\\n --check\n\n# 4. If check passes, run the actual upgrade\nsudo -u postgres pg_upgrade \\\n --old-datadir=/var/lib/postgresql/16/main \\\n --new-datadir=/var/lib/postgresql/17/main \\\n --old-bindir=/usr/lib/postgresql/16/bin \\\n --new-bindir=/usr/lib/postgresql/17/bin \\\n --link # Use hardlinks, 10x faster\n\n# 5. Start PG 17 and run the post-upgrade script\nsudo systemctl start postgresql@17-main\nsudo -u postgres /var/lib/postgresql/17/main/analyze_new_cluster.sh\n```\n\nThe ` --link ` option is critical, it creates hardlinks instead of copying data files, reducing upgrade time from hours to minutes for large databases. The old cluster's data files become invalid after this (they share pages with the new cluster), so keep a proper backup.\n\nFor a 480GB database, typical upgrade times:\n\n- With ` --link `: 4 minutes (mostly catalog operations)\n- Without ` --link `: 3 hours 20 minutes (copying 480GB)\n\n### Option B: Logical replication (zero downtime)\n\n```bash\n# 1. Set up PG 17 as a fresh instance\n# 2. Create the schema on PG 17 (pg_dump --schema-only)\npg_dump -h pg16-host -d mydb --schema-only | psql -h pg17-host -d mydb\n\n# 3. Create publication on PG 16\npsql -h pg16-host -d mydb -c \"CREATE PUBLICATION upgrade_pub FOR ALL TABLES;\"\n\n# 4. Create subscription on PG 17\npsql -h pg17-host -d mydb -c \"\n CREATE SUBSCRIPTION upgrade_sub\n CONNECTION 'host=pg16-host port=5432 dbname=mydb'\n PUBLICATION upgrade_pub;\"\n\n# 5. Wait for initial sync (check pg_stat_subscription)\n# 6. When caught up, point application to PG 17\n# 7. Drop subscription and publication\n```\n\nThis approach has zero downtime but requires more operational care, you need to handle sequences, large objects, and any DDL changes during the migration window.\n\n### The PG 17-specific gotcha\n\nIf you're using logical replication on PG 16 and upgrading to PG 17, be aware that PG 17 changed how replication slot LSN tracking works with the new failover feature. If your replication slots were created without ` failover = true ` (the default), they'll work fine after upgrade. But if you try to add ` failover = true ` to existing slots post-upgrade, you need to ensure ` sync_replication_slots = on ` on all standbys first, or the slot will appear healthy but not actually be synchronized.\n\n## Limitations and Caveats\n\nNot everything in PG 17 is a win for every workload:\n\n- **The MERGE improvements** are nice but still lag behind MySQL's INSERT ON DUPLICATE KEY UPDATE in raw performance for high-concurrency upsert patterns.\n- **SQL/JSON path language** is powerful but the error messages when you get a path expression wrong are cryptic. Debugging ` JSON_TABLE ` queries takes patience.\n- **The new COPY option DEFAULT** is useful but has a surprising interaction with GENERATED ALWAYS columns, you need to explicitly specify column lists to avoid errors.\n\nThese are minor complaints in the context of a genuinely solid release. PG 17 is the kind of upgrade that doesn't require you to change anything but makes everything work a little better. And the incremental backup feature alone justifies the upgrade for anyone managing databases over 100GB.\n\nThe recommendation: if on PG 15, start planning the migration now. If you're on PG 16, the upgrade is straightforward and the operational improvements are worth it. If you're still on PG 14 or below, you really need to move, PG 14 hits end-of-life in November 2026.\n\n## Frequently asked questions\n\n### How significant are the performance improvements in PostgreSQL 17?\n\nFor most workloads, expect 5-15% improvement in query throughput, mainly from VACUUM optimization (reads 8x fewer pages for mostly-frozen tables), improved sort algorithms for queries with LIMIT, and better hash join handling. The biggest wins are operational, incremental backup reduces backup storage by 80-90% and backup time proportionally.\n\n### Is it safe to upgrade from PostgreSQL 15 or 16 to 17?\n\nYes, with caveats. pg\\_upgrade works reliably for 16→17 based on production experience. For 15→17, test thoroughly, some changes to logical replication slot behavior can break existing replication setups. The pg\\_createsubscriber approach for logical replication-based upgrades is the safest path for zero-downtime migrations.\n\n### What's the biggest operational improvement in PG 17?\n\nIncremental backup, by a wide margin. For a 500GB database that changes 2% daily, the daily backup went from 500GB (full) to ~12GB (incremental). Combined with WAL archiving, this makes point-in-time recovery practical even for multi-terabyte databases on cloud storage.\n\n## Tips\n\n1. **Incremental backup requires WAL summarization enabled:** Set ` summarize_wal = on ` in postgresql.conf BEFORE taking your first full backup. Without it, ` pg_basebackup --incremental ` fails silently and takes a full backup instead. Check ` pg_wal_summary_stats ` view to verify summaries are being generated.\n2. **Use ` pg_createsubscriber ` for zero-downtime PG16→17 upgrades:** Instead of pg\\_dump/restore, convert a physical standby to a logical subscriber with ` pg_createsubscriber --pgdata /var/lib/postgresql/17/main -d 'host=pg16-primary' `. This gives you a live replica you can promote when ready.\n3. **` pg_stat_io ` reveals hidden I/O patterns:** Query ` SELECT backend_type, io_object, io_context, reads, writes, extends FROM pg_stat_io WHERE reads > 0 ORDER BY reads DESC ` to find which backend processes generate the most I/O. This can reveal that autovacuum is doing 3x more reads than client queries, a tuning goldmine.\n4. **The new ` JSON_TABLE() ` eliminates most JSON-to-rows gymnastics:** Instead of ` jsonb_array_elements ` with lateral joins, use ` SELECT * FROM JSON_TABLE(data, '$.items[*]' COLUMNS (id INT PATH '$.id', name TEXT PATH '$.name')) `. It's SQL-standard, and the query planner handles it better than the equivalent jsonb functions.\n\n- postgresql\n- databases\n- backup\n- replication\n- migration\n- pg17\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#title"},{"level":2,"text":"Incremental Backup: Native Support in PG 17","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#incremental-backup-native-support-in-pg-17"},{"level":2,"text":"Logical Replication Failover: Finally","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#logical-replication-failover-finally"},{"level":2,"text":"JSON_TABLE: SQL/JSON Done Right","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#jsontable-sqljson-done-right"},{"level":2,"text":"pg_stat_io: Understanding Your I/O Patterns","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#pgstatio-understanding-your-io-patterns"},{"level":2,"text":"VACUUM Improvements: 8x Fewer Page Reads","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#vacuum-improvements-8x-fewer-page-reads"},{"level":2,"text":"The Migration Path: PG 15/16 → PG 17","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#the-migration-path-pg-1516-pg-17"},{"level":3,"text":"Option A: pg_upgrade (simplest, brief downtime)","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#option-a-pgupgrade-simplest-brief-downtime"},{"level":3,"text":"Option B: Logical replication (zero downtime)","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#option-b-logical-replication-zero-downtime"},{"level":3,"text":"The PG 17-specific gotcha","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#the-pg-17-specific-gotcha"},{"level":2,"text":"Limitations and Caveats","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#limitations-and-caveats"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#questions"},{"level":3,"text":"How significant are the performance improvements in PostgreSQL 17?","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-1"},{"level":3,"text":"Is it safe to upgrade from PostgreSQL 15 or 16 to 17?","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-2"},{"level":3,"text":"What's the biggest operational improvement in PG 17?","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#article-tips"}],"sections":[{"heading":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","level":1,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication","content":"Databases"},{"heading":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","level":1,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#title","content":"2025-10-1410 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/postgresql-17-incremental-backup-logical-replication>)"},{"heading":"Incremental Backup: Native Support in PG 17","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#incremental-backup-native-support-in-pg-17","content":"Before PG 17, ` pg_basebackup ` always took a full backup. Every time. A 500GB database meant copying 500GB, whether you changed 10 rows or 10 million since the last backup. The alternatives were third-party tools like pgBackRest (excellent, but another thing to operate) or filesystem-level snapshots (which have their own issues).\n\nPG 17 adds native incremental backup support. Here's how it works:\n\n```bash\n# Step 1: Enable WAL summarization (required, do this first!)\n# postgresql.conf\nsummarize_wal = on\n\n# Restart PostgreSQL for the change to take effect\nsudo systemctl restart postgresql@17-main\n\n# Step 2: Take the first full backup (this is the baseline)\npg_basebackup -D /backups/full_20260114 \\\n --checkpoint=fast \\\n --wal-method=stream \\\n --manifest-checksums=SHA256 \\\n -U replication_user\n\n# Step 3: Take incremental backups (only changed blocks)\npg_basebackup -D /backups/incr_20260115 \\\n --incremental=/backups/full_20260114/backup_manifest \\\n --checkpoint=fast \\\n --wal-method=stream \\\n -U replication_user\n```\n\nThe ` --incremental ` flag points to the manifest file from the previous backup. PostgreSQL compares WAL summaries to determine which 8KB blocks have changed since that manifest was created, and only transfers those blocks.\n\n**Real-world numbers from a representative production database (480GB):**\n\n| Backup type | Size | Duration | Notes |\n| --- | --- | --- | --- |\n| Full backup | 480GB | 47 min | Baseline |\n| Day 1 incremental | 8.2GB | 4.2 min | ~1.7% daily change rate |\n| Day 2 incremental | 9.1GB | 4.8 min | Slightly busier day |\n| Day 7 incremental | 7.6GB | 3.9 min | Weekend, less activity |\n| Weekly synthetic full | 480GB | 12 min | Reconstructed from chain |\n\nThat last row is important. You can reconstruct a full backup from an incremental chain using ` pg_combinebackup `:\n\n```bash\n# Combine full + incrementals into a synthetic full backup\npg_combinebackup \\\n /backups/full_20260114 \\\n /backups/incr_20260115 \\\n /backups/incr_20260116 \\\n /backups/incr_20260117 \\\n -o /backups/synthetic_full_20260117\n\n# The synthetic full can be used as a new baseline\n# for future incremental chains\n```\n\nThe reconstruction takes ~12 minutes for a 480GB database because it's mostly sequential reads and writes, no network transfer needed if all backups are on the same storage.\n\n**The WAL summarization gotcha:** The ` summarize_wal ` setting creates summary files in ` pg_wal/summaries/ `. These are small (a few KB each), but they're only created for WAL that's generated *after* the setting is enabled. If you enable it and immediately try an incremental backup, it'll fail because there's no summary covering the period between your previous full backup and now. Enable it first, take a fresh full backup, then start your incremental chain."},{"heading":"Logical Replication Failover: Finally","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#logical-replication-failover-finally","content":"PG 17 adds ` failover = true ` on subscriptions and ` sync ` parameter on replication slots. This means logical replication slots can survive a failover to a standby. Before this, if your primary went down and you promoted a standby, all logical replication subscribers had to be re-initialized from scratch. On a 500GB database, that's hours of downtime.\n\nHere's the setup:\n\n```sql\n-- On the primary: create a publication\nCREATE PUBLICATION my_pub FOR ALL TABLES;\n\n-- On the subscriber: create subscription with failover support\nCREATE SUBSCRIPTION my_sub\n CONNECTION 'host=primary-host port=5432 dbname=mydb'\n PUBLICATION my_pub\n WITH (failover = true);\n```\n\nOn the primary, you'll see the replication slot has ` failover = true `:\n\n```sql\nSELECT slot_name, plugin, failover, synced\nFROM pg_replication_slots\nWHERE slot_name = 'my_sub';\n\n-- slot_name | plugin | failover | synced\n-- -----------+---------+----------+--------\n-- my_sub | pgoutput| t | f\n```\n\nThe ` synced ` column becomes ` true ` once the standby has synchronized the slot position. When failover happens, the new primary has the exact LSN position of the subscriber, so replication resumes without data loss or re-sync.\n\nThe configuration on the standby that enables this:\n\n```\n# postgresql.conf on standby\nsync_replication_slots = on\nhot_standby_feedback = on\n```\n\nTesting with a simulated failover (promoting the standby via ` pg_ctl promote `) shows the subscriber reconnects and resumes replication within 3 seconds. Before PG 17, the same scenario required dropping and recreating the subscription, which meant a full table sync, 2 hours for a comparable dataset."},{"heading":"JSON_TABLE: SQL/JSON Done Right","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#jsontable-sqljson-done-right","content":"Writing ` jsonb_array_elements ` with lateral joins has long felt like a workaround. PG 17 implements ` JSON_TABLE() ` from the SQL/JSON standard, and it's substantially cleaner:\n\n```sql\n-- The old way (PG 16 and below)\nSELECT\n e->>'id' AS id,\n e->>'name' AS name,\n e->>'email' AS email\nFROM orders o,\n LATERAL jsonb_array_elements(o.metadata->'contacts') AS e\nWHERE o.created_at > '2026-01-01';\n\n-- The new way (PG 17)\nSELECT t.*\nFROM orders o,\n JSON_TABLE(\n o.metadata,\n '$.contacts[*]'\n COLUMNS (\n id INTEGER PATH '$.id',\n name TEXT PATH '$.name',\n email TEXT PATH '$.email'\n )\n ) AS t\nWHERE o.created_at > '2026-01-01';\n```\n\nBut the real power is in nested paths and error handling:\n\n```sql\n-- Nested JSON_TABLE with error handling\nSELECT t.*\nFROM api_responses r,\n JSON_TABLE(\n r.body,\n '$.data.users[*]'\n COLUMNS (\n user_id INTEGER PATH '$.id',\n username TEXT PATH '$.profile.name'\n DEFAULT 'unknown' ON EMPTY\n DEFAULT 'error' ON ERROR,\n NESTED PATH '$.orders[*]' COLUMNS (\n order_id INTEGER PATH '$.id',\n total NUMERIC(10,2) PATH '$.total',\n items_count INTEGER PATH '$.items.size()'\n )\n )\n ) AS t;\n```\n\nThe ` ON EMPTY ` and ` ON ERROR ` clauses are game-changers for dealing with inconsistent JSON structures. No more ` COALESCE(jsonb_extract_path_text(...), 'default') ` chains.\n\nPerformance-wise, ` JSON_TABLE ` isn't magic, it still does the same work under the hood. But the query planner can reason about it better because it sees a single table-returning function instead of a lateral join chain. On a query hitting 50,000 rows with an average of 8 JSON array elements each, benchmarks show a 22% improvement in execution time versus the equivalent ` jsonb_array_elements ` approach."},{"heading":"pg_stat_io: Understanding Your I/O Patterns","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#pgstatio-understanding-your-io-patterns","content":"This view was added in PG 16, but PG 17 expanded it significantly with better backend type categorization and extend statistics. It has become an essential diagnostic tool.\n\n```sql\n-- Top I/O consumers by backend type\nSELECT\n backend_type,\n io_object,\n io_context,\n reads,\n read_bytes / (1024*1024*1024.0) AS read_gb,\n writes,\n write_bytes / (1024*1024*1024.0) AS write_gb,\n extends,\n fsyncs\nFROM pg_stat_io\nWHERE reads > 0 OR writes > 0\nORDER BY reads + writes DESC\nLIMIT 20;\n```\n\nWhat this reveals on a representative production server:\n\n```\n backend_type | io_object | io_context | reads | read_gb | writes | write_gb\n-------------------+-----------+------------+----------+---------+----------+---------\n autovacuum worker | relation | normal | 14822041 | 112.8 | 2841023 | 21.6\n client backend | relation | normal | 4201882 | 32.0 | 891204 | 6.8\n client backend | relation | bulkread | 3102441 | 23.6 | 0 | 0.0\n background writer | relation | normal | 0 | 0.0 | 1204812 | 9.2\n checkpointer | relation | normal | 0 | 0.0 | 982104 | 7.5\n```\n\nAutovacuum was doing 3.5x more reads than all client queries combined. That is not necessarily wrong, autovacuum is doing its job, but it indicates the ` autovacuum_vacuum_cost_delay ` was too aggressive and vacuum was scanning tables that did not need it.\n\nAfter tuning:\n\n```sql\n-- Per-table autovacuum settings for large, mostly-static tables\nALTER TABLE historical_events SET (\n autovacuum_vacuum_threshold = 10000,\n autovacuum_vacuum_scale_factor = 0.01,\n autovacuum_analyze_threshold = 5000\n);\n```\n\nAutovacuum reads dropped by 60%, and client query I/O performance improved measurably because there was less contention on the buffer pool."},{"heading":"VACUUM Improvements: 8x Fewer Page Reads","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#vacuum-improvements-8x-fewer-page-reads","content":"PG 17 fundamentally changed how VACUUM handles pages. The previous behavior was to read every page in a table to check for dead tuples, even if the page was already all-frozen (every tuple visible to all transactions). PG 17 tracks frozen-page status in the visibility map and skips those pages entirely.\n\nFor a large table with a high frozen-to-total page ratio (common for append-heavy tables like event logs), this is transformative:\n\n```sql\n-- Check your table's frozen page ratio\nSELECT\n relname,\n pg_size_pretty(pg_relation_size(oid)) AS table_size,\n n_tup_ins AS inserts_since_vacuum,\n n_tup_upd AS updates_since_vacuum,\n n_dead_tup AS dead_tuples,\n last_autovacuum\nFROM pg_stat_user_tables\nWHERE relname = 'events'\nORDER BY pg_relation_size(oid) DESC;\n```\n\nOn a representative ` events ` table (89GB, 95% frozen pages), VACUUM time dropped from 18 minutes to 2.2 minutes. The I/O reduction was even more dramatic, from 89GB of reads to under 5GB."},{"heading":"The Migration Path: PG 15/16 → PG 17","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#the-migration-path-pg-1516-pg-17","content":"This migration has been performed multiple times across different environments. Here is the playbook that works:"},{"heading":"Option A: pg_upgrade (simplest, brief downtime)","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#option-a-pgupgrade-simplest-brief-downtime","content":"```bash\n# 1. Install PG 17 alongside existing version\nsudo apt install postgresql-17\n\n# 2. Stop both clusters\nsudo systemctl stop postgresql@16-main\nsudo systemctl stop postgresql@17-main\n\n# 3. Run pg_upgrade in check mode first\nsudo -u postgres pg_upgrade \\\n --old-datadir=/var/lib/postgresql/16/main \\\n --new-datadir=/var/lib/postgresql/17/main \\\n --old-bindir=/usr/lib/postgresql/16/bin \\\n --new-bindir=/usr/lib/postgresql/17/bin \\\n --check\n\n# 4. If check passes, run the actual upgrade\nsudo -u postgres pg_upgrade \\\n --old-datadir=/var/lib/postgresql/16/main \\\n --new-datadir=/var/lib/postgresql/17/main \\\n --old-bindir=/usr/lib/postgresql/16/bin \\\n --new-bindir=/usr/lib/postgresql/17/bin \\\n --link # Use hardlinks, 10x faster\n\n# 5. Start PG 17 and run the post-upgrade script\nsudo systemctl start postgresql@17-main\nsudo -u postgres /var/lib/postgresql/17/main/analyze_new_cluster.sh\n```\n\nThe ` --link ` option is critical, it creates hardlinks instead of copying data files, reducing upgrade time from hours to minutes for large databases. The old cluster's data files become invalid after this (they share pages with the new cluster), so keep a proper backup.\n\nFor a 480GB database, typical upgrade times:\n\n- With ` --link `: 4 minutes (mostly catalog operations)\n- Without ` --link `: 3 hours 20 minutes (copying 480GB)"},{"heading":"Option B: Logical replication (zero downtime)","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#option-b-logical-replication-zero-downtime","content":"```bash\n# 1. Set up PG 17 as a fresh instance\n# 2. Create the schema on PG 17 (pg_dump --schema-only)\npg_dump -h pg16-host -d mydb --schema-only | psql -h pg17-host -d mydb\n\n# 3. Create publication on PG 16\npsql -h pg16-host -d mydb -c \"CREATE PUBLICATION upgrade_pub FOR ALL TABLES;\"\n\n# 4. Create subscription on PG 17\npsql -h pg17-host -d mydb -c \"\n CREATE SUBSCRIPTION upgrade_sub\n CONNECTION 'host=pg16-host port=5432 dbname=mydb'\n PUBLICATION upgrade_pub;\"\n\n# 5. Wait for initial sync (check pg_stat_subscription)\n# 6. When caught up, point application to PG 17\n# 7. Drop subscription and publication\n```\n\nThis approach has zero downtime but requires more operational care, you need to handle sequences, large objects, and any DDL changes during the migration window."},{"heading":"The PG 17-specific gotcha","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#the-pg-17-specific-gotcha","content":"If you're using logical replication on PG 16 and upgrading to PG 17, be aware that PG 17 changed how replication slot LSN tracking works with the new failover feature. If your replication slots were created without ` failover = true ` (the default), they'll work fine after upgrade. But if you try to add ` failover = true ` to existing slots post-upgrade, you need to ensure ` sync_replication_slots = on ` on all standbys first, or the slot will appear healthy but not actually be synchronized."},{"heading":"Limitations and Caveats","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#limitations-and-caveats","content":"Not everything in PG 17 is a win for every workload:\n\n- **The MERGE improvements** are nice but still lag behind MySQL's INSERT ON DUPLICATE KEY UPDATE in raw performance for high-concurrency upsert patterns.\n- **SQL/JSON path language** is powerful but the error messages when you get a path expression wrong are cryptic. Debugging ` JSON_TABLE ` queries takes patience.\n- **The new COPY option DEFAULT** is useful but has a surprising interaction with GENERATED ALWAYS columns, you need to explicitly specify column lists to avoid errors.\n\nThese are minor complaints in the context of a genuinely solid release. PG 17 is the kind of upgrade that doesn't require you to change anything but makes everything work a little better. And the incremental backup feature alone justifies the upgrade for anyone managing databases over 100GB.\n\nThe recommendation: if on PG 15, start planning the migration now. If you're on PG 16, the upgrade is straightforward and the operational improvements are worth it. If you're still on PG 14 or below, you really need to move, PG 14 hits end-of-life in November 2026."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#questions","content":""},{"heading":"How significant are the performance improvements in PostgreSQL 17?","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-1","content":"For most workloads, expect 5-15% improvement in query throughput, mainly from VACUUM optimization (reads 8x fewer pages for mostly-frozen tables), improved sort algorithms for queries with LIMIT, and better hash join handling. The biggest wins are operational, incremental backup reduces backup storage by 80-90% and backup time proportionally."},{"heading":"Is it safe to upgrade from PostgreSQL 15 or 16 to 17?","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-2","content":"Yes, with caveats. pg\\_upgrade works reliably for 16→17 based on production experience. For 15→17, test thoroughly, some changes to logical replication slot behavior can break existing replication setups. The pg\\_createsubscriber approach for logical replication-based upgrades is the safest path for zero-downtime migrations."},{"heading":"What's the biggest operational improvement in PG 17?","level":3,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#question-3","content":"Incremental backup, by a wide margin. For a 500GB database that changes 2% daily, the daily backup went from 500GB (full) to ~12GB (incremental). Combined with WAL archiving, this makes point-in-time recovery practical even for multi-terabyte databases on cloud storage."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#article-tips","content":"1. **Incremental backup requires WAL summarization enabled:** Set ` summarize_wal = on ` in postgresql.conf BEFORE taking your first full backup. Without it, ` pg_basebackup --incremental ` fails silently and takes a full backup instead. Check ` pg_wal_summary_stats ` view to verify summaries are being generated.\n2. **Use ` pg_createsubscriber ` for zero-downtime PG16→17 upgrades:** Instead of pg\\_dump/restore, convert a physical standby to a logical subscriber with ` pg_createsubscriber --pgdata /var/lib/postgresql/17/main -d 'host=pg16-primary' `. This gives you a live replica you can promote when ready.\n3. **` pg_stat_io ` reveals hidden I/O patterns:** Query ` SELECT backend_type, io_object, io_context, reads, writes, extends FROM pg_stat_io WHERE reads > 0 ORDER BY reads DESC ` to find which backend processes generate the most I/O. This can reveal that autovacuum is doing 3x more reads than client queries, a tuning goldmine.\n4. **The new ` JSON_TABLE() ` eliminates most JSON-to-rows gymnastics:** Instead of ` jsonb_array_elements ` with lateral joins, use ` SELECT * FROM JSON_TABLE(data, '$.items[*]' COLUMNS (id INT PATH '$.id', name TEXT PATH '$.name')) `. It's SQL-standard, and the query planner handles it better than the equivalent jsonb functions.\n\n- postgresql\n- databases\n- backup\n- replication\n- migration\n- pg17\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/postgresql-17-incremental-backup-logical-replication"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication"},"headline":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","description":"A practical deep dive into PostgreSQL 17's most impactful features: incremental backup with pg_basebackup, logical replication failover, new JSON functions, pg_stat_io insights, and real migration strategies from PG 15/16.","datePublished":"2025-10-14","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Databases","keywords":"postgresql, databases, backup, replication, migration, pg17","wordCount":1838,"timeRequired":"PT10M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/postgresql-17-incremental-backup-logical-replication"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"PostgreSQL 17: Incremental Backup, Logical Replication Failover, and Operational Improvements","item":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How significant are the performance improvements in PostgreSQL 17?","acceptedAnswer":{"@type":"Answer","text":"For most workloads, expect 5-15% improvement in query throughput, mainly from VACUUM optimization (reads 8x fewer pages for mostly-frozen tables), improved sort algorithms for queries with LIMIT, and better hash join handling. The biggest wins are operational, incremental backup reduces backup storage by 80-90% and backup time proportionally."}},{"@type":"Question","name":"Is it safe to upgrade from PostgreSQL 15 or 16 to 17?","acceptedAnswer":{"@type":"Answer","text":"Yes, with caveats. pg_upgrade works reliably for 16→17 based on production experience. For 15→17, test thoroughly, some changes to logical replication slot behavior can break existing replication setups. The pg_createsubscriber approach for logical replication-based upgrades is the safest path for zero-downtime migrations."}},{"@type":"Question","name":"What's the biggest operational improvement in PG 17?","acceptedAnswer":{"@type":"Answer","text":"Incremental backup, by a wide margin. For a 500GB database that changes 2% daily, the daily backup went from 500GB (full) to ~12GB (incremental). Combined with WAL archiving, this makes point-in-time recovery practical even for multi-terabyte databases on cloud storage."}}]}]}],"contentHash":"cddad302da4ceedf302e53caba7b764ad2595e334b1d5e57cbbea5d23259be81","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/postgresql-17-incremental-backup-logical-replication","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":72563,"contentBytes":15892}}