3/25/2019 PostgreSQL Backups the Modern Way

PostgreSQL Backups
the Modern Way
Nordic PGDay 2019
Copenhagen, Denmark

Magnus Hagander
[email protected]

localhost:9999/?print-pdf/#/ 1/50
3/25/2019 PostgreSQL Backups the Modern Way

Magnus Hagander
Redpill Linpro
Infrastructure services
Principal database consultant
PostgreSQL
Core Team member
Committer
PostgreSQL Europe

localhost:9999/?print-pdf/#/ 2/50
3/25/2019 PostgreSQL Backups the Modern Way

So, backups...
Do you make them?

localhost:9999/?print-pdf/#/ 3/50
3/25/2019 PostgreSQL Backups the Modern Way

Backups
Are not superseded by replication
Or cloud
Or containers
..

localhost:9999/?print-pdf/#/ 4/50
3/25/2019 PostgreSQL Backups the Modern Way

Backups
Are boring
But I'm glad you have them

localhost:9999/?print-pdf/#/ 5/50
3/25/2019 PostgreSQL Backups the Modern Way

Backups
When did you last restore?

localhost:9999/?print-pdf/#/ 6/50
3/25/2019 PostgreSQL Backups the Modern Way

PostgreSQL backups
Ok, enough generic
What about backups in PostgreSQL?

localhost:9999/?print-pdf/#/ 7/50
3/25/2019 PostgreSQL Backups the Modern Way

Seen this before?

pg_dump options:

-Fc = custom format

-Z = compression
-j = parallel
-a = data only, -s = schema only
-n = schema, -t = table
...

localhost:9999/?print-pdf/#/ 8/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_dump
Don't use for backups
Has other good usecases
Too slow to restore
Too much overhead
No PITR
Exceptions, of course

localhost:9999/?print-pdf/#/ 9/50
3/25/2019 PostgreSQL Backups the Modern Way

Physical backups
Base backups
With or without log archive
Fast restore
Full cluster only
Platform specific

localhost:9999/?print-pdf/#/ 10/50
3/25/2019 PostgreSQL Backups the Modern Way

Base backups
#!/bin/bash
set -e

psql -U postgres -q "SELECT pg_start_backup('foo')"

tar cfz /backup/$(date +%Y%m%d).tar.gz /var/lib/pgsql/data

psql -U postgres -q "SELECT pg_stop_backup()"

localhost:9999/?print-pdf/#/ 11/50
3/25/2019 PostgreSQL Backups the Modern Way

Base backups
So many ways to get that wrong
Spot one?

localhost:9999/?print-pdf/#/ 12/50
3/25/2019 PostgreSQL Backups the Modern Way

Base backups
This used to be the only way
Many scripts around that does it
Most of those are broken...

localhost:9999/?print-pdf/#/ 13/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_basebackup
Base backup over replication protocol
Safe
Error handling and recovery
For most cases
(we'll cover other options later)

localhost:9999/?print-pdf/#/ 14/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_basebackup
#!/bin/bash
set -e

pg_basebackup -D /backup/$(date +%Y%m%d) -Ft

localhost:9999/?print-pdf/#/ 15/50
3/25/2019 PostgreSQL Backups the Modern Way

Needs replication
Enabled by default in 10!
Older versions:

wal_level = hot_standby
max_wal_senders = 10

local replication postgres peer

localhost:9999/?print-pdf/#/ 16/50
3/25/2019 PostgreSQL Backups the Modern Way

Backup formats
plain
Safe copy of data directory
Not good with multiple tablespaces
tar
Destination still a directory
Each tablespace gets one file
base.tar

localhost:9999/?print-pdf/#/ 17/50
3/25/2019 PostgreSQL Backups the Modern Way

Transaction log
WAL required to restore backup
From beginning of backup to end
In the log archive, right?

localhost:9999/?print-pdf/#/ 18/50
3/25/2019 PostgreSQL Backups the Modern Way

Including WAL
Always use -x or -X to include WAL
Default in 10
Makes backup independently consistent
With or without log archive
May back up WAL twice
Use even with log archive!

localhost:9999/?print-pdf/#/ 19/50
3/25/2019 PostgreSQL Backups the Modern Way

Including WAL
-X fetch
Fetches WAL at end of backup
Can fail if WAL rotated
-X stream
Replicates WAL over secondary connection
Fewer failure scenarios
Does not work with tar prior to version 10
-X none
Turn off (10+ only)

localhost:9999/?print-pdf/#/ 20/50
3/25/2019 PostgreSQL Backups the Modern Way

Replication slots
pg_basebackup can fall behind on < 10
Use replication slot
Don't forget to drop!
PostgreSQL 10 uses ephemeral slot

localhost:9999/?print-pdf/#/ 21/50
3/25/2019 PostgreSQL Backups the Modern Way

Backup compression
pg_basebackup -Z
Compression happens in pg_basebackup
Tar format only
CPU usage
Remote server?

localhost:9999/?print-pdf/#/ 22/50
3/25/2019 PostgreSQL Backups the Modern Way

Transfer compression
SSL compression
Much harder these days
ssh tunneling
Does not work with WAL

ssh mydbserver -c "pg_basebackup -Ft -D- -Z9 -Xnone" > backup.tgz

localhost:9999/?print-pdf/#/ 23/50
3/25/2019 PostgreSQL Backups the Modern Way

That's it!
With that, you have backups
That work
And are (reasonably) safe

localhost:9999/?print-pdf/#/ 24/50
3/25/2019 PostgreSQL Backups the Modern Way

PITR
Point in time recovery
You all want it
A bit more setting up

localhost:9999/?print-pdf/#/ 25/50
3/25/2019 PostgreSQL Backups the Modern Way

archive_command
To use PITR, we use log archiving
like this?

archive_command =
'test ! -f /mnt/archivedir/%f && cp %p /mnt/archivedir/%f'

localhost:9999/?print-pdf/#/ 26/50
3/25/2019 PostgreSQL Backups the Modern Way

Don't do that!

localhost:9999/?print-pdf/#/ 27/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_receivewal
Runs on archive server
Uses streaming replication
Generates log archive

localhost:9999/?print-pdf/#/ 28/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_receivewal
More granular recovery
Safe against server restarts
Can follow timeline switches on master

localhost:9999/?print-pdf/#/ 29/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_receivewal
Always use with replication slot
As of 9.4
But we said modern..
Backups should block

localhost:9999/?print-pdf/#/ 30/50
3/25/2019 PostgreSQL Backups the Modern Way

pg_receivewal
pg_receivewal -D /log/archive -h master -S backup

Ensure it's restarted!

localhost:9999/?print-pdf/#/ 31/50
3/25/2019 PostgreSQL Backups the Modern Way

Backup retention
How long to keep around?
What granularity?
...

localhost:9999/?print-pdf/#/ 32/50
3/25/2019 PostgreSQL Backups the Modern Way

Backup retention
Recovery needs:
Base backup
All WAL from start to end
All WAL from end to pitr
(that's why we use -x!)

localhost:9999/?print-pdf/#/ 33/50
3/25/2019 PostgreSQL Backups the Modern Way

Backup retention
find is o en enough
Delete logs older than X, base older than Y
Safe if -x was used!

#!/bin/bash

find /var/backups/basebackup -type f -mtime +30 -print0 |

xargs -0 -r /bin/rm

find /var/backups/wal -type f -mtime +7 -print0 |

xargs -0 -r /bin/rm

localhost:9999/?print-pdf/#/ 34/50
3/25/2019 PostgreSQL Backups the Modern Way

Not enough?
Handles the simple cases
But has limitations
Particularly in management

localhost:9999/?print-pdf/#/ 35/50
3/25/2019 PostgreSQL Backups the Modern Way

Other tools
pgBackRest
Barman

localhost:9999/?print-pdf/#/ 36/50
3/25/2019 PostgreSQL Backups the Modern Way

pgBackRest
Backup scheduling
Log archiving
Retention management
Multi-server
Restore shortcuts
Obsessive validation

localhost:9999/?print-pdf/#/ 37/50
3/25/2019 PostgreSQL Backups the Modern Way

pgBackRest
Developed by CrunchyData
Perl
Moving to C
MIT license
ssh but not rsync

localhost:9999/?print-pdf/#/ 38/50
3/25/2019 PostgreSQL Backups the Modern Way

pgBackRest
Custom protocol
Parallel backups (compression)
Full/Differential/Incremental
Segment based
Delta restore

localhost:9999/?print-pdf/#/ 39/50
3/25/2019 PostgreSQL Backups the Modern Way

pgBackRest
Validates checksums
Checksums backups
Every time

localhost:9999/?print-pdf/#/ 40/50
3/25/2019 PostgreSQL Backups the Modern Way

pgBackRest
No pg_receivewal support
Yet
No Windows support
Yet

localhost:9999/?print-pdf/#/ 41/50
3/25/2019 PostgreSQL Backups the Modern Way

Barman
Backup scheduling
Log archiving
Retention management
Multi-server
Restore shortcuts

localhost:9999/?print-pdf/#/ 42/50
3/25/2019 PostgreSQL Backups the Modern Way

Barman
Developed by 2ndQuadrant
Python
GPLv3
Primarily ssh+rsync
1.6 learned about pg_receivewal
2.0 learned about pg_basebackup
Before that, no (safe) concurrent backup support

localhost:9999/?print-pdf/#/ 43/50
3/25/2019 PostgreSQL Backups the Modern Way

What about

Enterprise product X
?

localhost:9999/?print-pdf/#/ 44/50
3/25/2019 PostgreSQL Backups the Modern Way

Enterprise product X
pg_basebackup
Run as pre-backup command
Optionally clean up in post-backup
pgbackrest/barman
To disk
Let backup so ware take it form there

localhost:9999/?print-pdf/#/ 45/50
3/25/2019 PostgreSQL Backups the Modern Way

Summary

localhost:9999/?print-pdf/#/ 46/50
3/25/2019 PostgreSQL Backups the Modern Way

Don't roll your own!

localhost:9999/?print-pdf/#/ 47/50
3/25/2019 PostgreSQL Backups the Modern Way

Don't roll your own

Too many pitfalls
Both base backups and archiving
Backups are too important!

localhost:9999/?print-pdf/#/ 48/50
3/25/2019 PostgreSQL Backups the Modern Way

Don't roll your own

Primary choice
Built-in (pg_basebackup)
If it's enough
Secondary choice
pgBackRest
Barman
Tertiary choice
Restart from top of slide

localhost:9999/?print-pdf/#/ 49/50
3/25/2019 PostgreSQL Backups the Modern Way

Thank you!
Magnus Hagander
[email protected]
@magnushagander
https://www.hagander.net/talks/
This material is licensed

localhost:9999/?print-pdf/#/ 50/50