Built-in Physical and Logical

Replication in Postgresql
Speaker
Fırat Güleç
- Company
Hepsiexpress

- Now
Infrastructure & Database Manager
Member of Postgresql Europe
Contributing to Open Source Community

- Next Talks
Swiss PGDay 2019, June 28th, 2019
Austria PGDay 2019, September 6th, 2019
Agenda
• What is Replication?
• Why do we need Replication?
• How many replication layers do we have?
• Understanding milestones of built-in Database Physical Replication.
• What is the purpose of replication? and How to rescue system in case of failover?
• What is Streaming Replication and what is its advantages? Async vs Sync, Hot standby etc.
• How to configurate Master and Standby Servers? And What is the most important parameters?
Example of topoloji.
• What is Cascading Replication and how to configurate it? Live Demo on Terminal.
• Quorum Commit for Sync Replication etc.
• What is Logical Replication coming with PostgreSQL 10? And What is its advantages?
• What is the purpose of Logical replication?
• How to set up Logical Replication and What are its benefits?
• Limitations of Logical Replication
• Logical Replication vs Physical Replication in detail.
• 10 Questions quiz and giving some gifts to participants according to their success.
What is Replication?
Slave

Slave

Slave

Slave

Slave
Master Slave

Slave

Slave Slave
Replication Layers
Pgpool II
Londiste
Bucardo
Application Slony

Database logical Logical Replication

Database physical Streaming Replication

Operating System DRBD

SAN
Hardware

NAS
High Availability, Load Balancing, and
Replication Feature Matrix

https://www.postgresql.org/docs/11/different-replication-solutions.html
Pyhsical Replication
The purpose of Streaming Replication?
Client

App Server

High availability

Master Slave
The purpose of Streaming Replication?
Client

App Server

Load Balancing
%50 %50

Master Slave
Database Physical Replication Milestones

Streaming Quorum
File based Synchronous
Replication commit since
from 8.3 since 9.1
9.0 10.0
Architecture of Streaming Replication

WAL Record
WAL WAL
sender receiver

Master Slave

WAL
Archieve WAL 16 MB
directory

pg_wal
Database Physical

Streaming Quorum
File based Synchronous
Replication commit since
from 8.3 since 9.1
9.0 10.0
File based Replication from Postgresql 8.3

WAL Record
WAL WAL
sender receiver

Master Slave

WAL
Archieve WAL 16 MB
directory

pg_wal
Database Physical

Streaming Quorum
File based Synchronous
Replication commit since
from 8.3 since 9.1
9.0 10.0
Streaming Replication
The primary and standby servers so that they are as similar as possible
Record-based log shipping
1- Major PostgreSQL release levels is not possible
2- 32-bit to a 64-bit system will not work.

WAL Record
WAL WAL
sender receiver

Master Slave
WAL

WAL
Archieve WAL 16 MB
directory

pg_wal pg_wal
Database Physical

Streaming Quorum
File based Synchronous
Replication commit since
from 8.3 since 9.1
9.0 10.0
Streaming Replication 1- Async vs Sync
2- Hot Standby or Warm
Standby?
write read read

WAL Record
WAL WAL
sender receiver

Master Slave

WAL
Archieve WAL 16 MB
directory

pg_wal
Database Physical

Streaming Quorum
File based Synchronous
Replication commit since
from 8.3 since 9.1
9.0 10.0
Quorum Commit for Sync Replication
• 10.0: Quorum Commit

sync Slave 1

sync

Master Slave 2

10.0synchronous_standby_names FIRST 2 (s1, s2, s3) sync

10.0synchronous_standby_names ANY 2 (s1, s2, s3)

Slave 3
 6 Steps for Streaming Replication
3. Authentication
6. Create Recovery.conf

1. Replica user is created

5. Synchronization

Master Slave

2. Configuration for Master 4. Configuration for Slave

Hot Standy - postgresql.conf
• wal_level  determines how much information is written

wal_level=‘minimal’
wal_level=‘replica’
wal_level=‘logical’
max_wal_senders
• max_wal_senders= specifies the maximum
number of concurrent connections Slave

Master Slave

Slave
wal_keep_segments
wal_keep_segments

WAL Record
WAL WAL
sender receiver

Master Slave

WAL
Archieve WAL 16 MB WAL
directory

pg_wal
1-Replication User for Master
• sudo -u postgres psql
Next, create a new user and role with the following command:
• postgres=#CREATE USER replica REPLICATION LOGIN ENCRYPTED
PASSWORD ‘*******';

postgres=#\du
• You should see the following output:
2-Hot Standby Configuration for Master
in postgresql.conf

• wal_level=replica
• wal_keep_segment=20
• max_wal_sender=3
• archieve_mode=on
• archive_command = 'test ! -f /var/lib/postgresql/pg_log_archive/%f
&& cp %p /var/lib/postgresql/pg_log_archive/%f'
3-pg_hba.conf configuration for Master
For authentication:

host replication replica 10.90.82.61/32 md5

4-Hot standy configuration for slave
In Postgresql.conf

• hot standby=on

Below configuration in case of fail over

• archive_mode = on
• archive_command = 'test ! -f /var/lib/postgresql/pg_log_archive/%f && cp
%p /var/lib/postgresql/pg_log_archive/%f‘
• wal_keep_segment=20
• max_wal_sender=3
5-Syncronize Data from Master Server to
Slave Server
On the slave server, stop the postgresql service:

• sudo systemctl stop postgresql and move existing data folder.

• pg_basebackup -h 10.90.82.31 -D /var/lib/postgresql/11/main/ -P -U
replica --wal-method=fetch
10.90.82.31 10.90.82.61
Transfering……

Data file directory

Master Slave
/var/lib/postgresql/11/main
6-Recovery.conf file on standby
Datafile Directory/var/lib/postgresql/11/main
Test Replication Streaming Replication is DONE

Commandpsql -x -c "select * from pg_stat_replication;"

Cascading Postgresql Replication Downstream Slaves

Upstream Slaves
async 1.
Slave
2. hot_standby=on
3. Syncronization
Replication
Slave 4. recovery.conf
Cascading
Replicaiton
Slave

Master Slave

Slave
Slave
max_standby_archive_delay for standby
Logical Replication
What is Logical Replication?
Logical replication
DB A
Employee
DB A
DB B

Master Slave

Streaming replication

DB A DB A

DB B DB B

Master Slave
What is Logical Replication?
Logical replication
Select Create index
Create table
Drop table
Create function
Insert
Master Slave Delete
New Instance
Streaming replication

Hot Standby Select Create index

Create table
Drop table
Warm Standby Select Create function
Insert
Master Slave Delete
Recovery Mode
What is Logical Replication?
Logical replication
PostgreSQL 11 PostgreSQL 12 PostgreSQL 10
PostgreSQL 11

Master Slave

Streaming replication
PostgreSQL 9.5 PostgreSQL 9.5 PostgreSQL 9.6
PostgreSQL 10

Master Slave
Expected use cases of Logical replication
2. Analytical
1. Sharing a subnet
Purposeof database

Master A

DB C

Master B

Streaming replication Client

1. High availability 2. Load Balancing

Master Standby
Expected use cases of Logical replication
3. Online Upgrade

App

PostgreSQL 10 PostgreSQL 11
Logical Replication
Dump
Schema
Master A Master B

Streaming replication Client

1. High availability 2. Load Balancing

Master Standby
Architecture
Logical replication

Publication Subscription

WAL Apply
sender Worker
Master Decoded message Slave Logical
Decoding triggers Replication
WAL Launcher

Streaming replication

WAL Streaming
WAL WAL
sender receiver

Master Slave
WAL
 What is Publication&Subscription?
Logical replication
Users Name Async vs Sync Pub 3 Subs 3
Logical
Replication Slot 1
Subscription 1

Publication 1
Slave A Slave C

Publication 2
Users Name Surname Subscription 2
Master
Logical
Replication Slot 2
Slave B
Streaming replication

Streaming Slave Cascading Slave

Master
Replication Replication
Logical replication
Surname Address

Pub 1 Sub 1

Sub 2 Pub 2

DB A DB B

Streaming replication

Slave Slave
Master Streaming Cascading
Replication Replication
Logical replication
Surname Surname

Bi directional replication not allowed

Server A Server B

Streaming replication

Slave Slave
Master Streaming Cascading
Replication Replication
Logical Replication Limitations in 11.0
• does not replicate schema and DDL
• does not replicate sequences
• does not replicate Large Objects
• Replication is only possible from base tables to base tables
 Logical replication with PostgreSQL 11
repluser

Publication 1 Subscribe1
user user

WAL

Master Slave
10.90.82.30 10.90.82.31
Pg_hba.conf host all repluser 10.90.82.31/0 md5

1. wal_level = logical 1. Connected to db with postgres \c database2

2. Connected to db with postgres \c database1 2. CREATE TABLE user (user_name text PRIMARY KEY,
full_name text);
3. CREATE TABLE user (user_name text PRIMARY KEY, full_name text); 3. CREATE SUBSCRIPTION Subscribe1 CONNECTION
4. CREATE PUBLICATION Publication1 for table user; 'host=10.90.82.31 dbname=database1 user=repluser
5. CREATE ROLE repluser WITH LOGIN PASSWORD 'admin123' REPLICATION ; password=admin123' PUBLICATION Publication 1 ;

6. GRANT SELECT ON public.delivery TO repluser;

7. GRANT USAGE ON SCHEMA public TO repluser;
Logical Replication is DONE
 Topology(Logical & Pyhsical Replication)
Client

App Server App Server Reporting Server

write write read read

read read
CPU & Memory

Logical Replication Streaming Replication Cascading Replication

Slave Master Slave Slave
Analytical purposes
Monitoring on Terminal

Slave(Logical) Slave(Physical)
Master
1- pg_stat_replication 1- pg_subscription 1- pg_stat_wal_receiver
2- pg_replication_slots 2- pg_stat_subscription
3- pg_publication_tables 3- pg_subscription_rel
https://kahoot.it/
Thank you

Fırat Güleç