How to Effectively Migrate Data From Legacy Apps
1 like167 views
The document outlines an effective strategy for migrating data from legacy systems to modern platforms, emphasizing the importance of data validation, cleansing, and mapping throughout the process. It highlights challenges such as knowledge gaps, data quality issues, and scope changes, and underscores the necessity for close cooperation with business stakeholders. Additionally, it introduces CloverDX as a versatile data management platform that facilitates automation and integration in handling complex data migration tasks.
How to Effectively Migrate Data From Legacy Apps
- 1. How to effectively migrate data from legacy systems www.cloverdx.com
- 2. Migration from legacy system Legacy system New system Has not been modernized in a long time, runs on outdated platform In-house or heavily customized Holds business critical data – CRM, ERP, billing, … Required for correct functioning of the organization Modern, more capable system Often cloud-based Provides business benefit like better reporting, data quality, visibility, … Migrate
- 3. Telco Migration, The Need In-house system CLF nCRM Validate, cleanse, map Validate, cleanse, map Data flow 100k customers 6m customers
- 4. Planning and scope changes Gaps in knowledge (in source and in target) Data quality Execution and reporting Challenges
- 6. Optimists planning a migration Analysis and discovery Extraction Validation Cleansing Loading Reconciliation Done
- 7. In reality… …many activities happen in parallel
- 8. Documentation of what needs to be done and of deadlines. Can take long time to complete Work with business and review as early as possible to prevent issues Figuringout the scope Business specification
- 9. In-depth look at each system involved to determine implementation plan. Requires cooperation from each party Knowledge gaps can prolong the process quite a bit. Figuringout the scope Technical analysis
- 10. Hardest part, especially for complex migrations with many business rules. What is a cost of a business rule implementation? Big bang migration or trickle migration? Figuringout the scope Effort estimates and planning Analysis Implementation & testing Run Analysis Implementation & testing Run Run Run Run Big bang Trickle
- 11. Not all data has the same complexity Which data to process first? Simple Complex Data complexity Number of records
- 12. Not all data has the same complexity More complex data brings more problems Which data to process first? Simple Complex Data complexity Number of records Problems
- 13. Not all data has the same complexity More complex data brings more problems Migrate complex records first, maybe manually Which data to process first? Simple Complex Data complexity Number of records Problems Do this first
- 14. Oh no! The scope changed! Factor “known unknowns” into your estimates – possibility to change after technical analysis and technology risks. What if business changed their mind? This cannot be avoided. Full and close cooperation with business helps when this happens.
- 15. Telco Migration, The Plan 24 m Start 12 m In-house to CLF CLF to nCRM Timeline In-house system CLF nCRM Validate, cleanse, map Validate, cleanse, map Data flow 100k customers 6m customers
- 16. Data Quality
- 17. Single biggest issue in most legacy migration projects Data frequently comes directly from users o Typed in an application (or even Excel document) o People come and go o Processes and ideas about data change Garbage in, garbage out o Migration is your best chance to fix your data, business already expects complications and issues with the data Data Quality
- 18. Make this part of your technical analysis Measure on every step of the process o On input, but also after transformations o Can help you catch logic errors Provide frequent reports to business o KPIs can help you make decisions: data is too bad → redesign solution Measuring data quality
- 19. Fix as early as possible, ideally in the source system o Needs cooperation with the source data owner Fail on error or keep on going? o Both approaches valid for different usecases Fixing data quality issues
- 20. Data Mapping
- 21. How to map entities in old system to the ones in the one? o Same data domain, different ideas about how to represent everything Knowledge gaps o Old system might not be known very well anymore o New system is not yet known very well The mapping problem ?
- 22. How to make it accessible to business users? How to keep track of different mapping versions? Use Data Modeling tools or Data catalog Mapping specification
- 23. Split mappings into multiple “pieces” to make them easier to build and verify Document your code Mapping development
- 24. Split mappings into multiple “pieces” to make them easier to build and verify Document your code Validate your data and report to business Mapping development
- 25. Test data can be difficult to obtain o Manually built test data is often insufficient – does not cover corner cases Use data as close to production data as possible o Anonymize to prevent data leaks and comply with regulations Test with business Test as early (and as often) as possible Testing the Mapping Production Data sample Anonymize Use data in QA
- 26. Everything is tested, smooth sailing from here… Communication is the key: frequent updates to your business users. Detailed reports help ensure that production data is ok. Be ready to support the solution – hypercare. Execution
- 27. Telco Migration, The Plan 24 m Start 12 m In-house to CLF CLF to nCRM Timeline In-house system CLF nCRM Validate, cleanse, map Validate, cleanse, map Data flow 100k customers 6m customers
- 28. Telco Migration, The Result 36 m Start 24 m In-house to CLF CLF to nCRM Timeline In-house system CLF nCRM Validate, cleanse, map Validate, cleanse, map Data flow 100k customers 6m customers Project restart Additional target systems
- 29. Bespoke systems o Vendor no longer available or not cooperating Complex scenarios o Many-to-one or many-to-many scenarios o Lots of customizations over long periods of time Messy data o Needing cleaning up, repurposing, reconstructing or enrichment Tight deadlines o When traditional approach can’t deliver on time Data Management Platform
- 30. There is no “I can’t” with CloverDX. Combining capabilities of coding with the readability of a visual design, CloverDX lets developers solve hard problems in code while empowering less technical colleagues. While CloverDX is built so that everything can sit on a single platform, it is flexible to fill in gaps in an existing technology stack. Thanks to its open architecture, almost every part of CloverDX can be customized, versioned, parametrized. If 90% of the work is easy and 10% are the exceptions, error and outliers, then CloverDX is built to help you deal with those 10 percent. Solve 100% of a problem, not just the easy 90%
- 31. With CloverDX, you ultimately think of running everything on autopilot. Automate Everything Automate the entirety of your data workloads, including the exceptions and intricacies of real-life processes. Schedule jobs for automatic delivery to data warehouses, data lakes or operational databases in cloud or on premise. Orchestrate external systems and tools by CloverDX sitting at the center of your architecture – events, API calls or message queues. www.cloverdx.com