2005 | Migration to Open Source Databases | Open Source Database Conference
0 likes•86 views
The document discusses the migration process to open source databases, outlining the objective to maintain functionality while minimizing risk and downtime. It covers factors such as where to migrate, the advantages and challenges of open source databases, and detailed workflows for executing the migration. Various migration strategies and tools are also explored, highlighting the need for careful planning and automation during the migration process.
2005 | Migration to Open Source Databases | Open Source Database Conference
- 1. Migration to Open Source Databases Jutta Horstmann www.osdbmigration.org
- 2. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 2 whoami ● Unix/Linux sysadmin ● DBA, developer (Informix) ● DB developer (Oracle) ● Web stuff (MySQL, PostgreSQL) ● Claim to Fame: OpenUsability.org ● Comp Sci Diploma Thesis: Migration to Open Source Databases
- 3. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 3 Agenda ● What's this? ● Why migrate? ● Where to migrate? ● How to migrate? ● What to migrate? ➔ About ➔ Pros and Cons ➔ Open Source Databases ➔ Workflow: Activities ➔ Workflow: Assets
- 4. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 4 Migration? "The process of changing from the use of one platform, environment, IT system, etc., to another, esp. in such a way as to avoid interruptions in service." (Oxford English Dictionary)
- 5. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 5 Migration: Objectives The target system: ● same functionality ● extendable ● incorporate all data ● modern hardware, software, architecture The migration workflow: ● Minimize risk ● Stay on budget ● Deliver in due time ● Minimize downtime
- 6. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 6 OSDB Migration:Pros and Cons YesNo ● Cost? Time? Effort? ● Lack of roadmap ● Licensing ● ISV Support ● Maintenance ● Accountability ● Features ● Features? ● TCO ● Open Source – Code – Community ● Standards ● Security ● Independence
- 7. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 7 Where to migrate? Firebird MaxDB Ingres MySQLPostgreSQL
- 8. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 8
- 9. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 9 Enterprise-Level? Requirements: ● Advanced data integrity mechanisms ● Advanced database objects ● Advanced SQL features ● Advanced features, tools, support for – Performance and scalability – (High) Availability – Security
- 10. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 10 OSDB Features Firebird 1.5 Ingres r3 MySQL 4.1 MaxDB 7.5 PostgreSQL 8.0 Advanced Indexing GIS support MVCC Two phase commit UD Functions UD data types Updatable Views MyISAM MyISAM InnoDB v.5.0v.7.6 v.5.0 v.8.1v.5.0
- 11. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 11 Performance / Scalability Firebird 1.5 Ingres r3 MySQL 4.1 MaxDB 7.5 PostgreSQL 8.0 Tablespaces Table Partitioning Parallelization Built-In Clustering Built-In Load Balancing v.8.1
- 12. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 12 How to migrate? Database Migration Workflow ● Activity-Centered? ● Asset-Centered? ● Proposal: Mixed-Model Workflow ● Walk-Through
- 13. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 13 Activity-Centered Workflow Analysis Data SQL Schema Infra- structure DBMS Data SQL Schema DBMS Data SQL Schema DBMS Data SQL Schema DBMS Design Implementation Test Source System Source vs. Target System Target System Target System Infra- structure Infra- structure Infra- structure
- 14. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 14 Asset-Centered Workflow Analysis Desig n DBMS Analysis Design Implementation Test Analysis Desig n Analysis Design Implementation Test Analysis Desig n Analysis Design Implementation Test Analysis Desig n Analysis Design Implementation Test Analysis Desig n Analysis Design Implementation Test Schema Data SQL Infra- structure Source System Source vs. Target System Target System Target System
- 15. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 15 Mixed-Model Workflow 1. Analyze the whole source system (all assets). 2. Migrate the DBMS software (design, implement, test). 3. Migrate the schema(s) (design, implement). 4. Migrate test data (design, implement, test). 5. Test the migrated schema(s) with the test data. 6. Migrate the client SQL (design, implement, test). 7. Migrate the database system's infrastructure (d, i, t). 8. Cut-over: Full data migration. 9. Final testing and evaluation.
- 16. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 16 Analyze the whole system Analysis Data SQL Schema Infra- structure DBMS ● How is the software configured? ● Complexity, quality of database design? ● How much data? Which condition? ● SQL isolated? Standard compliant? ● Tools, Policies, Tuning ...?
- 17. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 17 Where to look? Who to ask? ? Application Data Processes Documentation Data Dictionary History Data Flows Data Dictionary Developers, Admins, Users
- 18. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 18 Migrate the DBMS software 1. Choose Open Source DBMS 2. Devise mappings for configurations 3. Install & configure new software 4. Validate the working of the DBMS by using test databases and tools Design Implemen- tation Test DBMS
- 19. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 19 Migrate the schema(s) Design Implemen- tation Schema Test 1. Provide abstraction (logical, conceptual?) 2. Devise mappings to target system 3. Implement by loading converted schema into target system 4.Validate the schema's correctness based on loading and querying test data.
- 20. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 20 Migrate Test Data 1. Correction and “cleansing” of “dirty data” 2. Devise conversions and mappings 3. Transfer some test data: a) Extraction b)Transformation c) Load 4. Test: Errors at load? Query data! Design Implement Test Data Test Data
- 21. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 21 Migrate Client SQL Database connectivity ● Change ODBC/JDBC driver ● Switch client ● Client source available: Migrate! 1. Provide abstraction (Standard SQL) OR 2. Convert directly to target SQL syntax 3. Test statements against schema and test data Design Implemen- tation Test SQL
- 22. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 22 Migrate Infrastructure Design Infra- structure Test ● Tools: Administration, Development, Design ● Change ODBC/JDBC driver ● Switch client ● Administrative Tasks – Gather policies and jobs – Implement them in the target system way Implementation
- 23. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 23 Full Data Migration (Cut-Over) 1. Strategy: a) Cold Turkey? b) Chicken Little? c) Butterfly? 2. Data Transfer: Extraction, Transformation, Load 3. Test: Errors at load? Query data! Design Test Data Implementation
- 24. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 24 “Cold Turkey” ● Take source system offline ● Extract data ● Transform data ● Load data into target system ● Take target system online Problems: ● How long offline? How much data? ● Fallback?
- 25. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 25 “Chicken Little” ● Source and target system operate in parallel ● “Gateway” coordinates and maps queries ● Incremental data migration Problem: ● Complex implementation ● Update consistency? Gateway
- 26. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 26 “Butterfly” ● Only source system online ● Data gets frozen read-only ● Store manipulation results in “Temp Store 0” ● Transfer frozen data ● Freeze TS0, store manipulation results in TS1 ● Transfer frozen data ... and so on ● “Data Access Allocator coordinates and maps queries ● Problem: Complex implementation!
- 27. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 27 “Butterfly” Data Access Allocator Temp Store 1 TS 2 TS n Read-only TS n+1 Data Transformer
- 28. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 28 Summary: Problems ● Different SQL implementations ● User-defined data types/functions ● Stored Procedures ● Proprietary interfaces ● Flawed schema design ● Differences in Data Types ● “Dirty Data” ● Reserved Words
- 29. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 29 Help! - Migration Tools ● Automatization ● Integrity violation detection ● Re-Use ● Knowledge on source and target system ● Documentation ● Code Generation ● Script Scheduling ● Validation ● Speed!
- 30. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 30 Migration Tools, Examples ● Closed Source: – SQLWays (Ispirer). Only Windows. ● Source: any, Target: MySQL, PostgreSQL – ProgressionDB (Versora). Linux, Windows. ● MSSQL -> MySQL, PostgreSQL, Ingres ● Open Source: – Ingres Mio. $ Challenge (CA): shift2ingres – MySQL Migration Toolkit (MySQL AB)
- 31. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 31 Summary Migration to Open Source Databases ? ● Several Open Source options ● Migration is an ambitious project ● Workflow and results depending on state of source system ● Automatize the process, where possible!
- 32. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 32 Questions?