This post was originally published in December 2023 and was updated in March 2025.
These days, there’s significant industry interest in moving database systems to PostgreSQL. Often, these are greenfield projects focusing on design and architecture. However, decisions are increasingly being made to move from existing platforms, like MySQL, to PostgreSQL for compelling business and economic reasons. Sunsetting a stable, successful stack often relates more to changing realities and opportunities than performance alone.
So, if tasked to migrate a well-performing, albeit potentially aging, MySQL DBMS to a modern PostgreSQL one, how would you approach it? While both systems are robust, they have distinct differences in architecture, features, and even simple things like data type handling that can present challenges. This is where specialized tools become invaluable. This post details a practical example of migrating from MySQL to PostgreSQL using pgloader.
About pgLoader: A powerful migration tool
Enter pgloader, a command-line interface (CLI) specifically designed to facilitate the conversion and migration from various DBMS, including MySQL, into PostgreSQL. pgloader can extract data from MySQL, MSSQL, Oracle, and even other PostgreSQL instances. It can perform ETL (Extract, Transform, Load) operations during migration and, in specific scenarios, might even serve basic replication needs.
Database migration POC: MySQL to PostgreSQL with pgloader
Objective
Perform a database migration from MySQL to PostgreSQL using pgloader.
Attention: For the sake of brevity, I’ve NOT included the database schema in this blog.
Migration environment setup
The environment for this Proof of Concept (POC) involves a production MySQL 5.x database with hundreds of tables, including large ones with many columns, indexes (some composite), and object names exceeding PostgreSQL’s 63-character limit.
1 2 3 4 5 6 7 8 | MYSQL: OS: Ubuntu 16.04 MySQL version 5.7 host: mysql-001 port: 3306 DB: production (20GB, 68 million rows) USER: root PASSWORD: ##### |
1 2 3 4 5 6 7 8 | POSTGRES: OS: Ubuntu 22.04 PostgreSQL version 16.0 host: postgres-001 port: 5432 DB: dev user: postgres password: ##### |
Migration method: Preparing the source
While pgloader offers ETL capabilities, for this migration, the chosen approach was to minimize pgloader transformations by making necessary adjustments directly to the source MySQL database’s schema (column definitions, object names) beforehand.
TIP: Ensure any source database changes are compatible with the existing application stack before proceeding with the migration.
The development process involved:
Performing trial migrations using pgloader to identify necessary changes.
Observing and noting required updates on the MySQL database.
Documenting the process to replicate for the production migration.
Executing the final data migration using the pgloader CLI.
Step-by-step migration process using pgloader
1. PostgreSQL host preparation
- Install
- postgres server
- mysql client
- pgloader CLI
- Create target database
- Perform the database migration: execute pgloader with the appropriate configuration parameters
- Post database migration
- move all tables, indexes, and sequences to schema PUBLIC
- ANALYZE the database
MySQL host
- Upload database
- alter/update the database as required: refer to section PRE-Migration for more details
pgLoader configuration file
A number of iterative passes are made using different pgloader configuration parameters in order to determine what changes are needed on the MySQL database before a final database migration is possible.
The first step is to migrate the database schema without any data in order to identify failures related to data types, column and table names, and foreign constraints.
Here’s my initial pgloader configuration file; the default behavior is to execute all migration steps until the process is completed. In the case of an ERROR, it is logged, and the migration continues. Note that it is possible to halt the migration as soon as it encounters an ERROR with the parameter on error stop; refer to the reference documentation for more information.
1 2 3 4 5 6 | LOAD DATABASE FROM mysql://root:#####@192.168.150.224:3306/dev INTO postgresql://postgres:#####@localhost:5432/dev WITH TRUNCATE, DATA ONLY ; |
pgLoader invocation
Here’s a very simple execution of the migration process. Note that all messages are logged in the file DEV_migration.log:
1 | pgloader ./prod.load > DEV_migration.log |
Pre-migration: MySQL database updates
The following issues are documented in the order of their discovery in this POC. Remember that your own experiences will, of course, vary:
Issue #1: Incompatible values/data types, MySQL (datetime) -> Postgres (timestamp)
The values in many columns were changed from “0000-00-00 00:00:00” to “1970-01-01 00:00:00“. The values were explicitly updated in order to allow Postgres to accept the value. For the MySQL DBA, this is a known issue in older MySQL DBMS. Newer versions don’t allow this behavior.
Issue #2: Incompatible value/datatype, MySQL (time) -> Postgres (timestamp)
The table.column “pro_game_reports.game_time_tomorrow” was switched from datatype “time” to “integer“.
Issue #3: MySQL table names are too long
Since MySQL can have longer names than is legally acceptable for Postgres, pgLoader must rename them to shorter ones. There is a caveat, however, when one can encounter errors when the first 63+ characters of the source relation are the same, i.e., an attempt is made to generate duplicate names.
Issue #4: MySQL index names are too long
Similar to issue #3, pgloader automatically renames indexes as they are reconstituted into PostgreSQL, i.e., duplicate named indexes which are illegal in PostgreSQL. Assuming the name length is legal in Postgres, this issue can be resolved using the option preserve index names, preserving the original index names.
Issue #5: MySQL index names duplicated
Many indexes with the same name were duplicated because multiple columns were combined, forming a PRIMARY KEY constraint. Duplicate-named indexes in PostgreSQL, unlike MySQL, are not permitted. Consequently, they are manually renamed to unique ones:
1 2 3 4 5 6 7 8 9 | -- example of renaming the indexes in the MySQL DBMS BEGIN; alter table college_pitcher_seasons rename index amateur_player_id to idx_01; alter table draft_prospect_link_invitations rename index amateur_player_id to idx_02; alter table college_hitter_logs rename index amateur_team_id to idx_03; alter table college_hitter_logs rename index game_date to idx_03; alter table college_pitcher_logs rename index game_date to idx_04; COMMIT; |
Issue #6: Missing data detected in tables resulting in Foreign Key Constraint failures
Two values in table “pro_scouting_reports” were seen to be missing, thus preventing the creation of a number of Foreign Key constraints:
1 2 3 | # example values detected during the testing phase 921 4714 |
These two INSERT statements permit the creation of the FK constraints:
1 2 3 4 5 6 7 | BEGIN; insert into pro_scouting_reports(id,created_at,updated_at,report_state,ebis_id,scout_id) values (921,now(),now(),'published',772013,960206); insert into pro_scouting_reports(id,created_at,updated_at,report_state,ebis_id,scout_id) values (4714,now(),now(),'published',772013,960206); COMMIT; |
ATTENTION: Prior to adding column and table constraints, it is suggested that upon a successful data migration, all inconsistent records be deleted first.
Final pgloader configuration for data migration
After all the passes of identifying and updating the MySQL database have been performed, the final version of the loader configuration file results in a pretty simple invocation; see below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | LOAD DATABASE FROM mysql://root:#####@192.168.1.1:3306/dev INTO postgresql://postgres:#####@localhost:5432/dev WITH create indexes, preserve index names, foreign keys, reset sequences, multiple readers per thread, workers = 8, concurrency = 2, rows per range = 50000 SET PostgreSQL PARAMETERS maintenance_work_mem to '1GB' SET MySQL PARAMETERS net_read_timeout = '31536000', net_write_timeout = '31536000', lock_wait_timeout = '31536000' ; |
Conclusion
Migrating from MySQL to PostgreSQL using pgloader is a viable and powerful approach. As demonstrated in this POC, while pgloader handles much of the heavy lifting, successful migration often requires careful preparation, iterative testing to identify incompatibilities (like data types, object naming conflicts, constraint issues), and potentially making adjustments to the source MySQL database beforehand. Understanding common pitfalls and leveraging pgloader‘s configuration options allows for a smoother transition between these two popular database systems.
See why running open source PostgreSQL in-house demands more time, expertise, and resources than most teams expect — and what it means for IT and the business.
PostgreSQL in the Enterprise: The Real Cost of Going DIY
References
About pgloader
- https://pgloader.io/
- https://github.com/dimitri/pgloader
- https://pgloader.readthedocs.io/en/latest/pgloader.html
- https://manpages.ubuntu.com/manpages/xenial/man1/pgloader.1.html
About Percona Monitoring and Management
- https://www.percona.com/software/pmm/quickstart
- https://percona-pmm.readme.io/reference/introduction
- https://percona-pmm.readme.io/reference/changesecuritychecks
- https://docs.percona.com/percona-monitoring-and-management/index.html
- https://docs.percona.com/percona-monitoring-and-management/setting-up/server/docker.html
About MySQL
- https://docs.percona.com/percona-server/5.7/installation/apt_repo.html
- https://pgloader.readthedocs.io/en/latest/ref/mysql.html
FAQ: Migrating from MySQL to PostgreSQL using pgloader
Q1: What is pgloader?
A: pgloader is an open-source command-line tool designed to automate the migration of schema and data from various source databases (including MySQL, MS SQL, SQLite) into PostgreSQL. It handles data type casting, schema creation, data loading, index building, and constraint application.
Q2: Can pgloader migrate both schema and data from MySQL to PostgreSQL?
A: Yes, pgloader can perform a full migration, including creating tables, casting data types, loading data, creating indexes, adding foreign keys, and resetting sequences. You can also configure it to perform specific steps only (e.g., SCHEMA ONLY, DATA ONLY).
Q3: What are common problems encountered when using pgloader for MySQL to PostgreSQL migration?
A: Common issues include: incompatible data type values (like MySQL “zero dates”), object names (tables, indexes) exceeding PostgreSQL’s 63-character limit leading to naming conflicts, duplicate index names (allowed implicitly in MySQL, not in PostgreSQL), and foreign key constraint violations due to missing parent data or inconsistent child data. Careful pre-migration checks and potential source data cleanup are often needed.
Q4: Is pgloader free to use?
A: Yes, pgloader is open source software, typically released under the PostgreSQL License, making it free to download, use, and modify.
Q5: Does pgloader require downtime for migration?
A: pgloader performs a bulk load, which typically requires either significant downtime for the source application during the migration or a strategy involving initial load followed by change data capture (CDC) replication to minimize downtime, which pgloader itself doesn’t handle directly. The duration depends heavily on database size and network speed.
About the Author
Robert BernierRobert's first working computer was the very user-friendly IBM 360 with an awesome 4MB RAM.After a round of much needed therapy overcoming the trauma of programming with punch cards he discovered the IBM-XT and the miracle of DOS 2.0. Years later, Robert became enamored with Linux and the opensource world and after meeting one of the members of CORE his primary focus had become all things PostgreSQL.Robert has since then worked in mom and pop companies, fortune 50 corporations and a number of very cool environments including the famed Los Alamos National Laboratory in New Mexico, birthplace of the atomic age.Although reluctant to leave the enjoyable experience of California's Silicon Valley commuter life, he returned to the Pacific Northwest and once again experienced real weather.These days, he serves as the PostgreSQL Consultant here at Percona.
Hi Robert, we’ve used pgloader for this kind of migration for a few years with Percona 5.x but after upgrading to Percona 8.x pgloader does not work for us anymore:
>MySQL’s utf8mb3 (formerly utf8) not supportedhttps://github.com/dimitri/pgloader/issues/1592
You may want to re-test your flow.