Powering GIS Application with PostgreSQL and Postgres Plus

Supercharge
Your GIS Platform
Postgres Plus with PostGIS
3/6/2014
CONFIDENTIAL
©2011EnterpriseDB.Allrightsreserved.

• Introduction
• PostgreSQL and buzz it has created
• Postgres Plus Advanced Server
• Architecture
• Oracle Compatibility
• Performance Feature
• Security Features
• High Availability Features
• DBA Tools
• How to start adopting
• User Stories
• PostgreSQL and PostGIS
Agenda
2
CONFIDENTIAL

Postgres’ IMPACT
3
CONFIDENTIAL

• Independent & Thriving Development Community for over 20 years
• Thousands of active deployments worldwide in public and private
sector organizations of all sizes
PostgreSQL and the Community
4
CONFIDENTIAL

Postgres with PostGIS
5
CONFIDENTIAL

PostgreSQL – Postgres Plus Users, Globally
6
CONFIDENTIAL

7
Malaysia
Philippines
Singapore
Vietnam Thailand
Indonesia
PostgreSQL – Postgres Plus Users, across ASEAN
CONFIDENTIAL

EnterpriseDB – The Company
8
• The Enterprise PostgreSQL company was founded in 2004,
first product GA in 2005
• 2,000+ customers across all market segments
• 70,000+ downloads/week of PostgreSQL and related products
• Enabling database consolidation using PostgreSQL and
advanced Oracle compatibility
• Saving customers millions through the power of open source
• Strong financial backing:
CONFIDENTIAL
Postgres Plus - Recognized
by Gartner’s Magic
Quadrant as a challenger
in the Database Market

The EnterpriseDB Advantage
9
• Products and Tools
• Advanced database server software
• Deep Oracle compatibility
• Bundled development and management tools
• Technical Support and Services
• Around the clock support
• Expert consulting
• Oracle migration services
• Remote management and monitoring
• Professional Training
• Learn PostgreSQL from the experts
• Web and on-site training
• Training for developers and DBAs
CONFIDENTIAL

• Fast development cycles
• Thousands of developers
• Better code
• Lower cost
• 24/7 support
• Services and training
• Certification
• Indemnification
• Product strategy
Bringing the Advantage to Enterprises
10
Open Source
Software
Commercial
SoftwareEnterpriseDB
CONFIDENTIAL

About PPAS
11
PostgreSQL
xDB
REPLICATION
Server
HIGH
AVAILABILITY
and
Scalability
PERFORMANCE
and
Manageability
SECURITY
and
Cloud
Database
MIGRATION
Toolkit
CONFIDENTIAL

Architectural Overview
11
CONFIDENTIAL

Process Architecture
Postmaster
BGWRITER STATS
COLLECTOR
ARCHIVERAUTO--VACUUM
BGWRITER STATS
COLLECTOR
Data
Files
WAL
Segments
Archived
WAL
Shared Memory
Shared Buffers Process ArrayWAL Buffers
WAL Writer LOG WRITER
Error Log Files
12
CONFIDENTIAL

Database Limitations
14
CONFIDENTIAL
• Limitations are generally
defined by
• Operating System Limits
• Compile-Time Parameters
– Data Type Usage
General Database
Limitations
Limit Value
Maximum Database Size Unlimited
Maximum Table Size 32 TB
Maximum Row Size 1.6 TB
Maximum Field Size 1 GB
Maximum Rows per Table Unlimited
Maximum Columns per Table 250-1600 (Depending on
Column types)
Maximum Indexes per Table Unlimited

Oracle Compatibility
15
• Run applications written for Oracle
virtually unchanged
• No need to re-train Oracle DBAs and
developers
• Support for PL/SQL language and OCI
interoperability
• Replication for easy sharing of data
• Dramatic Cost Savings
• No Vendor Lock-in
CONFIDENTIAL

Survey: Re-Use of Oracle DBA Skills
16
CONFIDENTIAL

Compatibility Means
17
• SQL Extension support
• Decode, NVL, Substr, NVL2,replace,translate, table()
• Date/time functions: add_months, extract, next_day,months_between,trunc, current_date support
• Support for cascade option in drop table
• PL/SQL support
• REF Cursors, Implicit and explicit cursors
• Looping, variable declarations, conditional statements
• Collections: Associative Arrays, Varrays, Nested tables
• Bulk binding
• Named parameters
• User Defined Exceptions
• Explicit Transaction Control - within a stored procedure
• Tools
• EDB*Plus – SQL*Plus look-a-like
• EDB*Loader – SQL*Loader equivalent
• EDB*Wrap – similar to the PL/SQL wrapper
CONFIDENTIAL

Compatibility Means (cont.)
18
• Packages
• Stored procedures
• Functions
• Triggers
• Optimizer Hints
• Database Links
• Hierarchical Queries
• Synonyms – Public and
Private
• Sequences
• Materialized Views
CONFIDENTIAL
• Rownum
• Object types
o Create type … as object
o Create type … as table
o Create type …as varray
o Constructor and collection
methods
• PL/SQL like SUBTYPE, which
inherits form base type
• Users/Roles
• Dynamic SQL
• Features

Compatibility Means (cont.)
19
• Data Types
• Integer, number, char, double precision, float, varchar2, blob, clob, xmltype, rowid, boolean
• Built-in Packages
• DBMS_:
o SQL, LOB, JOB, PIPE, ALERT, OUTPUT, UTILITY, PROFILER, RLS
• UTL_:
o FILE, MAIL, SMTP, ENCODE, TCP
• Oracle-like Data Dictionary
• ALL_, DBA_, USER_ views
• Most commonly accessed views
• Diagnostics - DRITA
• System and session waits
o Not exposed in PostgreSQL
o Part of Advanced Server
• Statspack-like reporting
CONFIDENTIAL

Flexible Partitioning Scheme in PostgreSQL
20
• Partitioning Example in PostgreSQL
o CREATE TABLE employees
o Create Child Tables
o create table emp_mgmt (check (job in (MANAGER'))) inherits (employees);
o create table emp_sales (check (job in ('SALESMAN'))) inherits (employees);
o Create the partitioning function
o CREATE OR REPLACE FUNCTION partition_emp();
o Create the trigger for partition
o CREATE TRIGGER employees_partition_trigger BEFORE INSERT ON employees
FOR EACH ROW EXECUTE PROCEDURE partition_emp();
• Advantages
• Flexible
• Customize your Partitioning Policy
• Disadvantages
• Complex
• Difficult to add/delete/split Partitions
CONFIDENTIAL

• Partitioning Example in PPAS
CREATE TABLE employees
(empno numeric(4,0),ename varchar(10),job varchar(9),hiredate timestamp)
PARTITION BY LIST (job)
(PARTITION emp_mgmt VALUES ('MANAGER', 'PRESIDENT') tablespace tbsp_1, PARTITION
emp_sales VALUES ('SALESMAN') tablespace tbsp_2, PARTITION emp_ops VALUES ('CLERK')
tablespace tbsp_3
);
• PPAS Offers
• LIST and RANGE Partitioning Syntax
• Easy to manage and maintain partitions
• You can still take benefit of PostgreSQL syntax for complex partitioning e.g. Dynamic Partitions
• Easy to Add/Delete/Split/Swap/Detach/Attach partitions
• Brings the best of both the world!
Compatible and Flexible Partitioning Scheme in PPAS
21
CONFIDENTIAL

Scaling with Partitioning in PPAS v9.3
22
CONFIDENTIAL
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
100
250
500
1000
Time in Seconds
NumberofTablePartitions
Partitioning Performance: PPAS 9.3 v. PostgreSQL 9.3 v. PPAS 9.2
• Postgres Plus Advanced
Server 9.3 is up to 460
times faster with row
insertions into
partitioned tables than
PostgreSQL 9.3 or PPAS
9.2
• Better performance for
bulk loading and disaster
recovery

• Online Migration Toolkit enables point and click migration from Oracle
• Automatically Migrates:
Database Migration Toolkit
23
CONFIDENTIAL
• Data
• Schemas
• Stored Procedures
• Triggers
• Functions
• Sequences
• Packages
• Views
• Database Links
• Synonyms

• Developed for High Performance Transaction
Environments (OLTP)
• DynaTune:
• Dynamic tuning of the database server to make the
optimal usage of the system resources available on
the host machine
• Index Advisor:
• Helps determine which columns you should index
to improve performance in a given workload.
• Query Optimization Hints, Hi-Speed Bulk Loader,
Multi-Threaded Replication
Performance enhancement in PPAS
24
CONFIDENTIAL

• Infinite Cache
• High performance
horizontal scaling
architecture for
cache memory
• Cache expands with
inexpensive
commodity
hardware
Scalability with Infinite Cache
25
CONFIDENTIAL

26
• Architecture: How it Works
• Disk access for data is slower than RAM access
• Infinite Cache puts disk data into the RAM cache
• Additional requests for that data are read from the faster cache
• The cache scales as needed with inexpensive hardware
CONFIDENTIAL
Application
High
speed
pg_cach
e buffers
in RAM
Slow speed
disk
High
Speed
Network
Update
d Data
Blocks
Update
Read
If not in buffer
Database Server
InfiniteCache
RAMblades
High
speed

27
• Single Machine Performance
CONFIDENTIAL
Advanced Server is 16X
faster on a single machine
with large amounts of
memory
(e.g. greater than 2 GB)
Infinite Cache can be used
on a single machine!

28
• Infinite Cache - Features
• Expands and Distributes buffer cache across multiple machines
• Designed for Read-Mostly applications (e.g. Content Management, Query Intensive, Business Intelligence, Data
Warehouse)
• Cache is transparent to client applications (no cache coding needed)
• Compression feature enables caching entire databases (e.g. put a 250 GB database into 32 GB RAM Cache)
• Cache can be pre-warmed for immediate results
• Cache scales using inexpensive commodity hardware
• Infinite Cache can be run to boost single machine performance!
• Created For
• DBAs and data managers overseeing large amounts of data requiring fast response times for queries and
reporting loads
• Developers who don’t want to write specialized caching code
CONFIDENTIAL

• Red Hat Reference Architecture Series:
• Comparing BenchmarkSQL Performance
on Red Hat Enterprise Linux 5 to Windows
Server Enterprise (2010)
Performance Benchmark to SQL Server
29
CONFIDENTIAL

Scalability MySQL Vs PostgreSQL
30
CONFIDENTIAL

Security
31
• Object level privileges assigned to roles and users
• Virtual Private Database
• Kerberos and LDAP authentication
• Host base authentication
• SSL communication
• Data Level Encryption (AES, 3DES, etc)
• Ability to utilize 3rd party Key Stores in a full PKI Infrastructure
• Foundation for full compliance with the strictest of security standards (PCI Data Security
Standard)
• Flexible field level encryption and row level security
CONFIDENTIAL
“By default, PostgreSQL is probably the most
security-aware database available ...”
Database Hacker's Handbook- David Litchfield

PL/Secure and EDB*Wrap
32
• Encrypt your Pl/pgsql stored programs with PL/Secure
• Encrypt your SPL procedures with EDB*Wrap
CONFIDENTIAL

SQL/Protect
33
• DBA Managed SQL Injection Protection
• Preventing attacks is normally the responsibility of the application developer, but with SQL/Protect, DBAs can
now provide another layer of protection to prevent corruption or co-opting of the database.
• Multiple Prevention Techniques
• Unauthorized Relations
• Utility Commands (e.g. DDL)
• SQL Tautology
• Unbounded DML
CONFIDENTIAL

• Multi-Master Replication – Active-Active Technology
• Near real-time Replication
• Multi-Version Concurrency Control (MVCC)
• Point-in-Time Recovery
• Log Shipping for Standby (~ Oracle® Data Guard)
High Availability
34
CONFIDENTIAL
Availability

Backup and Recovery Options
35
CONFIDENTIAL
• Physical and Logical Backup
• Logical Backup using pg_dump
• Instance level logical backup using pg_dumpall
• Table level, Schema level or DB level logical backup
• pg_restore
• Physical Backups
• pg_basebackup
• Compressed backup
• Recovery with WAL and archived WAL
• Point in Time Recover

• Using OS Clustering
• Using a shared disk for data
Active – Passive OS HA Clustering
36
CONFIDENTIAL

• Uses WAL (Write Ahead Log)
• WAL is continuously shipped through
an opened TCP channel to Hot
Standby Node
• Hot Standby node can be used for
read scalability
Hot Streaming Replication (Hot Standby Mode)
37
CONFIDENTIAL

• Uses WAL (Write Ahead Log)
• Archived WAL is shipped to Hot
Standby Node
• Hot Standby node can be used for
read scalability
Log Shipping Replication (Hot Standby Mode)
38
CONFIDENTIAL

EnterpriseDB Postgres Plus Failover Manager
39
CONFIDENTIAL
Agent Agent

• Using pg-pool
• Using redundant disk for data
• Hot Standby node can be used
for read scalability
HA with read scaling (with pg-pool)
40
CONFIDENTIAL

• Trigger based replication
• Publication – Subscription
• Snapshot and Continuous
• Cascaded Replication
• Read Scalability
• Master/Slave DB can be:
• Oracle
• MSSQL
• Postgres Plus Advanced Server
• Either Master or Slave should be
PPAS/PostgreSQL
xDB Single Master Replication (SMR)
41
CONFIDENTIAL

• High Availability uses
• Geographic distribution of load
• For creation of Testing/staging env using snapshot replication
• Segregate OLTP and reporting
xDB Replication Use Cases
42
CONFIDENTIAL
Replication

xDB Single-Master Replication (SMR)
43
CONFIDENTIAL
Table
D
Table
C
Table
B
Table
A
Table
C
Table
D
Procs
Objects
ReportsQueries
Continuous
or Scheduled
----------------
Filtered
or All Rows
Transaction
Replication
Improved
OLTP
Performance
Inexpensive
Query /
Reporting
Oracle Server Postgres Plus Advanced Server

xDB SMR Heterogeneous Replication Support
44
CONFIDENTIAL
Source/Target Oracle MS SQL
Server
2005/2008
PostgreSQL
Advanced
Server
(Oracle
mode)
Advanced
Server
(PostgreSQL
mode)
Oracle No No Yes Yes Yes
MS SQL
Server
2005/2008
No No Yes Yes Yes
PostgreSQL No Yes Yes Yes Yes
Advanced
Server (Oracle
mode)
Yes Yes No
Yes
No
Advanced
Server
(PostgreSQL
mode)
No Yes Yes Yes Yes

• Trigger based replication
• 2 or more Masters can be Sync
• Auto Conflict Detection & Resolution
• Read & Write Scalability
xDB Multi Master Replication (MMR)
45
CONFIDENTIAL

• Java-based Replication Server and Replication Console
• Delta changes recorded in shadow tables via post Insert/Update/Delete triggers
• Data Replication through JDBC channel
xDB MMR Architecture
46
CONFIDENTIAL

Enterprise Class DBA Tools
47
CONFIDENTIAL

Remotely Manage, Monitor And Tune
• Management En-Mass Design
• Enterprise Performance Monitoring
• Proactive Alert Management
• Graphical Administration
• SQL Workload Profiling
• Simplified Capacity Planning
• Full SQL IDE
Postgres Enterprise Manager
48
CONFIDENTIAL

PEM Distributed Architecture
49
CONFIDENTIAL

• Performance
Monitoring Dashboards
• Capacity Manager
• Postgres Expert
• Alert Management
• Browser based console
and dashboard
• Audit Manager
• Team Support
• Distributed Architecture
• Convenient Access
PEM For DBAs: Centralized Tool
50
CONFIDENTIAL

Helps you apply patches and
updates for all the
products under PPAS:
Stackbuilder Plus
51
CONFIDENTIAL

•A solution to aid in the creation of highly
available configurations of Postgres
•Monitors the health of a Postgres HA
configuration
•Automates the failover process in the event
of a failure
•Used in conjunction with Streaming
Replication
Postgres Plus Failover Manager
52
CONFIDENTIAL
Slave
Witness
Master

Failover Manager Architecture
53
CONFIDENTIAL
Agent Agent

• Automatic Failover from master to replica node
• Configurable fencing operation
• By default uses VIP
• Parameter to specify alternative operation
• Ex: reconfigure a load balancer
• Manual failover configuration possible
• Email notifications when cluster status changes
• Witness node provides protection against ‘split brain’
scenarios
• User configurable wait times
• Built on PPCD/JGroups technology
Failover Manager Features
54
CONFIDENTIAL

Replicate below objects from MS SQL Server to
Postgres Plus Advanced Server:
• Schemas
• Tables
• Constraints
• Indexes
• Table data
• Views
Replicating & Migrating Microsoft SQL Server
55
CONFIDENTIAL

• Split input files into multiple files
• Load single tables in parallel
• Magnitudes better load time performance
• Parallel load capabilities also in Migration Toolkit and Migration Studio
Parallel Data Loader
56
CONFIDENTIAL

Use Cases:
- Sistema de Información Territorial
Estatal en Línea, México
- National Oceanic and Atmospheric
Administration
- TriMet
- http://www.ridethecity.com/
- FCC: Making the National Broadband
Map with OpenGeo Suite
- North Dakota State Water Commission
Comparative Studies:
- http://www.gise.cse.iitb.ac.in/
wiki/images/c/c4/Finalreport.p
df
- bit.ly/1e4meWo
Some Use Case for Postgres and PostGIS
57
CONFIDENTIAL

• OpenGeo
• QuantumGIS
• GeoServer
• ArcGIS
Popular GIS Applications are Compatible with
PostgreSQL
58
CONFIDENTIAL

You have a ‘Real’ alternative to Oracle or other conventional
proprietary Databases
Conclusion
59
CONFIDENTIAL
Economical
Technically
Sound
Easy to
Adopt

How to start adoption
60
CONFIDENTIAL
REPLICATION SERVER
BACKUP / DR SERVER
NEW APPLICATION
MIGRATION

61
CONFIDENTIAL

Powering GIS Application with PostgreSQL and Postgres Plus

More Related Content

What's hot (20)

Similar to Powering GIS Application with PostgreSQL and Postgres Plus (20)

More from Ashnikbiz (20)

Recently uploaded (20)

Powering GIS Application with PostgreSQL and Postgres Plus