AWS 마이그레이션 서비스 - 김일호 :: 2015 리인벤트 리캡 게이밍

Editor's Notes

• #3: Introduce self and Sergei (Senior Product Manager) Migration from on-premises and traditionally hosted databases to AWS managed database services… Not only how to migrate between same engines, but also between different, like… But before you can migrate data anywhere, you need a schema; tables and objects into which to load the data. We’ll talk about how you can convert database objects, in order to support moving between engines.
• #4: These days, we’re hearing a lot of customers tell us they want to move their on-premises applications into the cloud. But moving applications is simpler than moving the databases they depend on. Applications are usually stateless, and can be moved fairly easily using a lift and shift approach. (CLICK) But databases are stateful, and they require more care. To move databases to AWS, requires a data migration strategy. (CLICK) And when it comes to designing those strategies, customers want to be able to do it with the least possible inconvenience and visibility to their users. (CLICK) And once an application is migrated to AWS, it’s not the end of the story. Often customers have several applications, some in the cloud and some on premises or in hosted environments.Customers need to be able to synchronize their data between on-premises and cloud-based applications. (CLICK) And the same goes for applications within AWS. Those applications often share data, and customers want to be able to synchronize and replicate data between the various databases they maintain within AWS. (CLICK) And one other thing: customers moving applications to the cloud, often see it as an opportunity to break free from commercial databases, which tend to have a heavy licensing burden. We often hear customers asking us for a way to convert their commercial databases into AWS solutions, such as RDS MySQL, Postgres, Aurora and Redshift.
• #6: Announcing preview of AWS DMS Explain what it basically is: A managed hosted data replication service engineered to support graceful migration from legacy database systems into the next generation managed databases at AWS.
• #7: There are multiple reasons why you’d want to migrate your data. Read-only replication: reporting, read scaling Read/write replication: multi-master
• #8: There is a multitude of approaches for migrating your data to AWS. Choose the method based on Data set size, Network Connectivity (access, latency, and bandwidth), ability to sustain downtime for source DB, Need for continuous data synchronization. If you can take or day or two of downtime, and you don’t have to do the migration process several times, you want to do a Bulk Load. If you need to minimize downtime, or when your dataset is large and you can’t shut down access to the source while you are migrating your data, you want to consider Ongoing Replication.
• #9: In the past we recommended the following high-performance technique for moving large databases. Use DataPump and export files in parallel. Use a box that has multiple disks, to parallelize IO Compression Makes 500 GB to 175 GB Time to export 500GB is ~2.5 hours Transport compressed files to EC2 instance using UDP and the Tsunami server. Install Tsunami on both the source data host and the target EC2 host Using UDP you can achieve higher rates for transferring files that using TCP Start Upload when first files from DataPump become available. Upload in Parallel. No need to wait till all 18 files are done to start upload Time to upload 175GB is ~2.5 hours Step 3. Transfer Files to Amazon RDS DB instance Amazon RDS DB instance has an externally accessible Oracle Directory Object DATA_PUMP_DIR Use UTL_FILE. to move data files to Amazon RDS DATA_PUMP_DIR BEGIN perl_global.fh := utl_file.fopen(:dirname, :fname, 'wb', :chunk); END; BEGIN utl_file.put_raw(perl_global.fh, :data, true); END; BEGIN utl_file.fclose(perl_global.fh); END; Transfer Files as They Are Received, No need to wait till all 18 files are received in the EC2 instance. Start transfer to RDS instance as soon as the first file is received. Total time to Transfer Files to RDS ~3.5Hours Step 4. Import data into the Amazon RDS instance • Import from within Amazon RDS instance using DBMS_DATAPUMP package • Submit a job using PL/SQL script Total Time to Import Data into Amazon RDS: ~4hours But because we do everything staged and have 18 distinct files, the total duration is ~7hours. Background ------------------- Open port 46224 for Tsunami communication Tsunami UDP Protocol: A fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks (≥ 1 Gbps and even 10 GE), designed to provide more throughput than possible with TCP over the same networks. Tsunami Servers: http://sourceforge.net/projects/tsunami-udp/files/latest/download?_test=goal Optimize the Data Pump Export • Reduce the data set to optimal size, avoid indexes • Use compression and parallel processing • Use multiple disks with independent I/O Optimize Data Upload • Use Tsunami for UDP-based file transfer • Use large Amazon EC2 instance with SSD or PIOPS volume • Use multiple disks with independent I/O • You could use multiple Amazon EC2 instances for parallel upload Optimize Data File Upload to RDS • Use the largest Amazon RDS DB instance possible during the import process • Avoid using Amazon RDS DB instance for any other load during this time • Provision enough storage in the Amazon RDS DB instance for the uploaded files and imported data
• #10: * Like all AWS services, it is easy and straightforward to get started. You can get started with your first migration task in 10 min or less. You simply connect it to your source and target databases, and it copies the data over, and begins replicating changes from source to target. *That means that you can keep your apps running during the migration, then switch over at a time that is convenient for your business. * In addition to one-time database migration, you can also use DMS for ongoing data replication. Replicate within, to or from AWS EC2 or RDS databases For rinstance, After migrating your database, use the AWS Database Migration Service to replicate data into your Redshift data warehouses, cross-region to other RDS instances, or back to on-premises *Again- it is heterogeneous ~. With DMS, you can move data between engines. Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift * If you would like to sign up for the preview of DMS, go to…
• #11: Let’s take a look at how to use the database migr. Service… From the landing page, just click “get started”. That will take you to page that describes how DMS works to migrate your data; how you connect it to a source database and target database, then define replication tasks to move the data.
• #12: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #13: But DMSis for much more than just migration. (CLICK) DMSenables customers to adopt a hybrid approach to the cloud, maintaining some applications on premises, and others within AWS. There are dozens of compelling use cases for a hybrid cloud approach using DMS. (CLICK) for customers just getting their feet wet, AWS is a great place to keep up-to-date read-only copies of on-premises data for reporting purposes.AWS services like Aurora, Redshift and RDS are great platforms for this. (CLICK) With DMS, you can maintain copies of critical business data from third-party or ERP applications, like employee data from Peoplesoft, or financial data from Oracle E-Business Suite, in the databases used by the other applications in your enterprise. In this way, it enables application integration in the enterprise. (CLICK) Another nice thing about the hybrid cloud approach is that it lets customers become familiar with AWS technology and services gradually.DMS enables that. Moving to the cloud is much simpler if you have a way to link the data and applications that have moved to AWS with those that haven’t.
• #14: With the AWS Database Migration Service you pay for the migration instance that moves your data from your source database to your target database.(CLICK) (Actually talk to points) Each database migration instance includes storage sufficient to support the needs of the replication engine, such as swap space, logs, and cache. (CLICK) (actually talk to points) Inbound data transfer is free. (CLICK) Additional charges only apply (CLICK) if you decide to allocate additional storage for data migration logs or when you replicate your data to a database in another region or on-premises. AWS Database Migration Service currently supports the T2 and C4 instance classes. T2 instances are low-cost standard instances designed to provide a baseline level of CPU performance with the ability to burst above the baseline. They are suitable for developing, configuring and testing your database migration process, and for periodic data migration tasks that can benefit from the CPU burst capability. C4 instances are designed to deliver the highest level of processor performance and achieve significantly higher packet per second (PPS) performance, lower network jitter, and lower network latency. You should use C4 instances if you are migrating large databases and are looking to minimize the migration time.
• #15: Elaborate on heterogeneous use cases Database engine migration – cost savings; Move to full managed and scalable cloud-native – Ent class like Aurora Low-cost reporting, analytics and BI for systems on commercial OLTP (MySQL Postgres Aurora) Data integration – customer accounts, data like that, can be presented no only on the master platform, but also in applications that are based on non-commercial But you can’t just pick up an Oracle table and put it down in MySQL. You can’t run an Oracle PL/SQL package on Postgres. To migrate or replicate data between engines, you need a way to convert the schema, to build a set of tables and objects on the destination that is native to that engine. We’ve been working on that problem. Introduce Sergei
• #17: The AWS Schema Conversion Tool is a development environment that you download to your desktop and use to save time when migrating from Oracle and SQL Server to next-generation cloud databases such as Amazon Aurora. You can convert database objects such as tables, indexes, views, stored procedures, and Data Manipulation Language (DML) statements like SELECT, INSERT, DELETE, UPDATE.