Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL

Converting 85% of Dutch Primary Schools  
from Oracle to PostgreSQL
Martijn Dashorst

topicus.nl

Martĳn Dashorst
@dashorst
martijndashorst.com
github.com/dashorst

“I’d rather not work with
databases”
Martijn Dashorst (1997, ﬁrst job interview)

Big thank you to
Klaasjan Brand

Please Give Feedback at 
https://2019.pgconf.eu/f

What is ParnasSys?
Why migrate?
Planning
Stages
Conclusions & Questions

Dutch Education System
AGES
4-12
13-16/18
16-23
Primary Education
Secondary Education
Vocational/University Education
pre-vocational (4 yrs), pre-applied science (5 yrs), pre-research university (6 yrs)
vocational (4 yrs), applied science (4 yrs, bachelor), pre-research university (bachelor/master)
EDUCATION TYPE

Primary Education (4-12)
1.5M
students
141k
employees
6973
schoolsages 4-12, 2019-2020 2018 2018
Source: https://www.onderwijsincijfers.nl

Financing Primary Schools
€ government
school
teachers
buildings
materials
€
taxes lumpsum
number of 
students 
at 1 October

Financing Primary Schools
€ government
school
teachers
buildings
materials
€
taxes lumpsum
number of 
students 
at 1 October
administration

ParnasSys ~ noun.  
1. A SaaS to run primary schools by keeping student
records for school ﬁnancing, student counseling and
guidance, attendance keeping and communicating with
parents

ParnasSys ~ noun.  
1. A SaaS to run primary schools by keeping student
records for school ﬁnancing, student counseling and
guidance, attendance keeping and communicating with
parents
2. The only pinkmagenta administration  
system in the world

Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL

Java Web App
Hibernate (ORM)
Business Objects
(SAP)
MAXDB/SAPDB
Single database

Single schema

Multi-tenant
Development started

October 2003
Production in  
April 2004

First Database Migration in 2005
SAPDB/MAXDB ORACLE
User base growing fast

Unpredictable query performance in MaxDB
"Nobody got ever ﬁred
for buying Oracle"

85% of dutch
primary schools
use ParnasSys
~5M student dossiers
in our database
1,270,000 students in 2019-2020
2019

800+ pages 360 tables data & indices
~ 2TB
user ﬁles
~ 5TB

Mismatch Between Future of
Oracle and our Product

Cost Structure of Running
Oracle Not Compatible with our
Proﬁtability

The Oracle Migration Assistant
Paid Us a Visit
TM

The stranglehold 
of the software giants
[...] Especially Oracle would
force unfair contracts upon
clients, and claim many millions
per year of late payments. [...]
http://fd.nl/weekend/1318745/de-wurggreep-van-de-softwarereuzen

Developer Happiness
Or How Do I Run My Database
on My Local Machine

DevOps Happiness on Oracle
is Incompatible with our
Proﬁtability

Inventory
✅ Database contents

✅ Java Application Code

✅ Business Objects Universe

✅ Business Objects Reports

✅ Backup & Restore

✅ Monitoring

✅ Training
🚫 No stored procedures

🚫 No custom functions

🚫 No triggers

2 Moments in a Year to Migrate:
X-Mas & Summer Vacation

X-Mass 2 weeks
23 December 2019–5 January 2020

Summer vacation(s)
North
15 juli 2019–23 august 2019

Middle
22 juli 2019–30 august 2019

South
8 juli 2019–16 august 2019

Ideal migration moment: 
start of summer vacation

Stage 1: make it run 
incorrectly, correctly, fast

Stage 2: migrate production 
in accordance to customer expectations

1 Year Migration
3 Year Migration

Free

Works™

Great to ﬁnd issues in your
application

Perl 
“The only language that looks the same before
and after RSA encryption”

Direct datatype mapping 
E.G. number(1,0) does not map to boolean

Slow 
One import takes whole day

Stable or fast (pick one) 
Single-threaded: stable but slow 
Multi-threaded: fast but unstable

One-way, big bang only

Differences between Oracle and PostgreSQL
“The good thing about standards is that there are
so many to choose from.”
― Andrew S. Tanenbaum

Mapping datatypes
Oracle PostgreSQL
number(1,0) number(1,0)
boolean
Java
int
boolean
enum Sex { 
Female,
Male 
}
Business
study year (1-8)
present (yes/no)
sex (female/male)
number(1,0)
number(1,0) number(1,0)
varchar2(6) varchar2(6)

ORM is 100% portable
@Basic
@Temporal(TemporalType.DATE)
private Date birthdate;
@Formula("add_months(birthdate, (to_char(sysdate, 'yyyy') to_char(birth
@Temporal(TemporalType.DATE)
private Date birthday;
CREATE OR REPLACE FUNCTION birthday(birthdate IN DATE) RETURN D
IS
BEGIN
RETURN add_months(birthdate-1,  
(to_char(sysdate, 'yyyy')
- to_char(birthdate, 'yyyy')) * 12)+1;
END;

Oracle speaks in tongues
select 1 from dual
select nvl(null, 'n/a') from dual
select * from (select x,y from ...)  
where rownum between 100 and 110
sysdate
trunc

Performance
ID ORGANIZATION NAME BIRTHDATE ... 
14342 1231 JOHN SMITH 2010-09-23
14345 1231 MARTIJN DASHORST 2011-03-12
...
ID ORGANIZATION NAME 
151231 1231 Group 1A
151232 1231 Group 1B
...
tenant identifier

Oracle is smarter...
UPDATE
testresult tr
SET
tr.endresult = TRUE
WHERE
tr.organisation = ?
AND tr.testpart IN (
SELECT part1.id
FROM
testpart part1
WHERE
part1.preference = (
SELECT MIN(part2.preference)
FROM
testresult tr3
INNER JOIN testpart part2 ON
tr3.testpart = part2.id
WHERE
part2.preference > 0
AND tr3.score IS NOT NULL
AND tr3.test = tr.test
AND ((part1.usemodulenorm IS NULL
AND part2.usemodulenorm IS NULL)
OR part1.usemodulenorm = part2.usemodulenorm)))

Oracle is smarter...UPDATE
testresult tr
SET
tr.endresult = TRUE
WHERE
tr.organisation = ?
SELECT part1.id
FROM
testpart part1
WHERE
FROM
testresult tr3
WHERE
UPDATE
testresult tr
SET
tr.endresult = TRUE
WHERE
tr.organisation = ?
SELECT part1.id
FROM
testpart part1
WHERE
FROM
testresult tr3
WHERE
AND tr3.organisation = tr.organisation))
100x improvement

Java Web App
Business Objects
(SAP)
PDF
Universe
Report deﬁnition
Parameters
PDF

Teachers Generate a lot of PDFs 
but that's OK, because  
Business Objects generates a lot of SQL

Business Objects  
can be a bit nostalgic

Quizz!
SELECT *
FROM A, B
WHERE B.column(+) = A.column
SELECT *
FROM A
RIGHT OUTER JOIN B
ON B.column = A.column
SELECT *
FROM B
LEFT OUTER JOIN A
SELECT *
FROM A
LEFT OUTER JOIN B
SELECT *
FROM B
RIGHT OUTER JOIN A
1 2
3 4

Quizz!
SELECT *
FROM A, B
WHERE B.column(+) = A.column
SELECT *
FROM A
RIGHT OUTER JOIN B
SELECT *
FROM B
LEFT OUTER JOIN A
1 2

Application works

Business Objects works
GO!

How long takes migration?
Behavior of application under actual load? 
- Heavy use of application side caching in Java process 
- Modiﬁcation of data 
- Synthetic tests lie
Can we fall back to Oracle?

Need help... 
1-800-KILLORACLE

Why don't you use HVR?
–Splendid Data

Cross-database replication
using redo logs

Remote UNIX 
Machine
Remote XYZ 
Machine
Hub Machine
PC
HVR
HVR
Scheduler HVR
HVR
HVR
GUI
Database
Location
Hub DB
Database
Location
IntegrateCapture
Inetd
Router
Transaction
Files
Inetd
HVR
Listener
Service

Why HVR?
• Speed

• Real-time sync with low latency

• Options to test prior to actual migration

• Safe

• Fallback

• Expensive tool, but Special Price for Special Friend

Noteworthy issues
• Use primary key on each table

• Fix datatype incompatibilities because of diﬀerences between databases 
E.G. time in date ﬁeld ✅ Oracle 🚫 PostgreSQL

• Disable constraint checking in target database

• Don't perform DDL changes

• Don't update too many rows in a table too often

Oracle based deployment
DB1
PROD
failsafe
DB0
PROD TEST
APP
PROD
Business
Objects
APP
TEST
ORACLEORACLE ORACLE
APP
PROD
APP
PROD
Fail safe server
Oﬄine

Archive Log shipping

DB0 DB1 DB2
PROD PROD TEST
TEST
failsafe
TEST
PROD
failsafe
PROD 
standby
APP
PROD
Business
Objects
APP
TEST
ORACLEORACLE
ORACLE
ORACLE
PSQL PSQL
PSQL
HVR migration deployment

DB0 DB1 DB2
PROD PROD TEST
TEST
failsafe
TEST
PROD
failsafe
PROD 
standby
APP
PROD
Business
Objects
APP
TEST
ORACLEORACLE
ORACLE
ORACLE
PSQL PSQL
PSQL
Prod Testing Business Objects

PostgreSQL based deployment
DB1 DB2
PROD
failsafe
DB1
PROD TEST
APP
PROD
Business
Objects
APP
TEST
PSQLPSQL PSQL
APP
PROD
APP
PROD
DB0
PROD
ORACLE

D-Dayde-oracle day
22 Juli 2019

Make a *really* good inventory of things that have to be migrated

Pick a strategy that works for you

Plan around your business to minimise risk and maximise opportunity

A great way to ﬁnd bugs in your system

Don't be afraid to pay for tools and help

“Nobody got ﬁred from
migrating away from Oracle”
“Yet”

Migrate our Secondary Education
Application from Oracle to PostgreSQL

Feedback 
https://2019.pgconf.eu/f

THANK YOU!
@dashorst
martijndashorst.com
github.com/dashorst

Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL

Recommended

More Related Content

Similar to Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL (20)

More from Martijn Dashorst (20)

Recently uploaded (20)

Converting 85% of Dutch Primary Schools from Oracle to PostgreSQL