Training Slides: 351 - Tungsten Replicator for Data Warehouses
0 likes•56 views
This document outlines a short course on utilizing the Tungsten Replicator for data warehousing with a focus on Hadoop, Redshift, and Vertica. It covers replicator flow, configurations, and prerequisites for successful replication, including necessary user accounts and integration settings. Future sessions will introduce using Tungsten Replicator with MongoDB.
![Extractor Config
[defaults]
user=tungsten
install-directory=/opt/continuent
mysql-allow-intensive-checks=true
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=tr-ext-2
replication-user=tungsten
replication-password=secret
replication-port=3306
enable-heterogeneous-service=true](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-13-320.jpg)
![Applier Configs
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=verticahost
datasource-type=vertica
replication-user=dbadmin
replication-password=Secret123
replication-port=5433
vertica-dbname=demo
batch-enabled=true
batch-load-template=vertica6
batch-load-language=js
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=tr-app-1
datasource-type=redshift
replication-user=dbadmin
replication-password=Secret123
replication-port=5439
replication-host=redshift-endpoint
redshift-dbname=demo
batch-enabled=true
batch-load-template=redshift
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
VerticaRedshift](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-14-320.jpg)
![Applier Configs
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=hadoopapplier
datasource-type=file
property=replicator.datasource.global.csvType=hive
replication-user=tungsten
replication-password=secret
batch-enabled=true
batch-load-template=hadoop
batch-load-language=js
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
Hadoop](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-15-320.jpg)
Training Slides: 351 - Tungsten Replicator for Data Warehouses
- 1. The MySQL Availability Company Tungsten Replicator Master Class Advanced: Working with Data Warehouse Targets Chris Parker, Customer Success Director, EMEA & APAC
- 2. Topics In this short course, we will • Review replicator flow • Explore Hadoop, Redshift and Vertica specific pre-requisites • Review configurations • Demo
- 4. How Hadoop Replication Works THL Target Replicator: Applier CSV JS HDFS
- 5. How the Hadoop Materialisation Works Op Seqno ID Msg I 1 1 Hello World! I 2 2 Meet Continuent D 3 1 I 3 1 Goodbye World Op Seqno ID Msg I 2 2 Meet Continuent I 3 1 Goodbye World insert into t1 values (1,”Hello World!”); insert into t1 values (2,”Meet Continuent”); update t1 set msg=”Goodbye World” where ID = 1;
- 6. How Vertica Replication Works cpimport THL Target Replicator: Applier CSV JS JDBC merge
- 7. How Redshift Replication Works THL Target Replicator: Applier CSV JS JDBC merge S3 s3cmd copy
- 8. Prerequisites • Review online documentation • https://docs.continuent.com • Download the Prerequisite Checklist • Extractor/Applier Hosts • OS User • /etc/hosts • sudoers • Ruby • Java • Network • Review Port Requirements • MySQL • my.cnf settings • User accounts • Hadoop • HDFS writeable by replicator user • Vertica • User Accounts • JDBC Drivers • Redshift • User Accounts • S3 Bucket • S3 Tools for uploading • AWS JSON Config • All tables need Primary Keys
- 9. AWS JSON Config for Redshift • /opt/continuent/share/s3-config-<servicename>.json • awsS3Path — the location within your S3 storage where files should be loaded. • awsAccessKey — the S3 access key to access your S3 storage. Not required if awsIAMRole is used. • awsSecretKey — the S3 secret key associated with the Access Key. Not required if awsIAMRole is used. • awsIAMRole — the IAM role configured to allow Redshift to interact with S3. Not required if awsAccessKey and awsSecretKey are in use. • s3Binary — the binary to use for loading csv file up to S3. (Valid Values: s3cmd, s4cmd, aws) (Default: s3cmd) • cleanUpS3Files - a boolean value used to identify whether the CSV files loaded into S3 should be deleted after they have been imported and merged (Default: true)
- 10. Provisioning Options • Traditional CSV export and import • Dump and load through Blackhole engine • If target support standard SQL, extract data as INSERTS • For Hadoop, use Sqoop
- 11. Object Mapping • Hadoop • MySQL Database à HDFS Directory • Table à Hive Compatible CSV File • Row à Line in the file • Redshift & Vertica (PostgreSQL Interface and Syntax) • MySQL Instance à Database • MySQL Database à Schema
- 12. DDLScan • Reverse engineers MySQL objects • Creates target specific DDL • Can be used for single objects or entire databases • Must be configured prior to starting replicators • Must be run twice, once for base tables, once for staging tables ddlscan –service alpha -template ddl-mysql-redshift.vm -db test >ddl.sql ddlscan –service alpha -template ddl-mysql-redshift-staging.vm -db test >ddl-staging.sql
- 16. Demonstration
- 17. Summary What we have learnt today • Reviewed replicator flow • Explored Hadoop, Redshift and Vertica specific pre-requisites • Reviewed configurations
- 18. Next Steps In the next session we will • Learn how to use Tungsten Replicator with MongoDB
- 19. THANK YOU FOR LISTENING continuent.com The MySQL Availability Company Chris Parker, Customer Success Director, EMEA & APAC