Training Slides: 351 - Tungsten Replicator for Data Warehouses
0 likes54 views
Follow this 36min training session that looks at using Tungsten Replicator with data warehouse targets such as Hadoop, Redshift and Vertica in particular, including a demo showing how to set these configurations up. TOPICS COVERED - Review replicator flow - Explore Hadoop, Redshift and Vertica specific prerequisites - Review configurations - Demo
![Extractor Config
[defaults]
user=tungsten
install-directory=/opt/continuent
mysql-allow-intensive-checks=true
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=tr-ext-2
replication-user=tungsten
replication-password=secret
replication-port=3306
enable-heterogeneous-service=true](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-13-320.jpg)
![Applier Configs
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=verticahost
datasource-type=vertica
replication-user=dbadmin
replication-password=Secret123
replication-port=5433
vertica-dbname=demo
batch-enabled=true
batch-load-template=vertica6
batch-load-language=js
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=tr-app-1
datasource-type=redshift
replication-user=dbadmin
replication-password=Secret123
replication-port=5439
replication-host=redshift-endpoint
redshift-dbname=demo
batch-enabled=true
batch-load-template=redshift
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
VerticaRedshift](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-14-320.jpg)
![Applier Configs
[defaults]
user=tungsten
install-directory=/opt/continuent
profile-script=~/.bash_profile
disable-security-controls=true
[alpha]
master=tr-ext-2
members=hadoopapplier
datasource-type=file
property=replicator.datasource.global.csvType=hive
replication-user=tungsten
replication-password=secret
batch-enabled=true
batch-load-template=hadoop
batch-load-language=js
svc-applier-block-commit-interval=30s
svc-applier-block-commit-size=250000
Hadoop](https://image.slidesharecdn.com/351-tungsten-replicator-for-data-warehouses-200910193732/85/Training-Slides-351-Tungsten-Replicator-for-Data-Warehouses-15-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Training Slides: 351 - Tungsten Replicator for Data Warehouses (20)
Ad
More from Continuent (20)
Ad
Recently uploaded (19)
Training Slides: 351 - Tungsten Replicator for Data Warehouses
- 1. The MySQL Availability Company Tungsten Replicator Master Class Advanced: Working with Data Warehouse Targets Chris Parker, Customer Success Director, EMEA & APAC
- 2. Topics In this short course, we will • Review replicator flow • Explore Hadoop, Redshift and Vertica specific pre-requisites • Review configurations • Demo
- 4. How Hadoop Replication Works THL Target Replicator: Applier CSV JS HDFS
- 5. How the Hadoop Materialisation Works Op Seqno ID Msg I 1 1 Hello World! I 2 2 Meet Continuent D 3 1 I 3 1 Goodbye World Op Seqno ID Msg I 2 2 Meet Continuent I 3 1 Goodbye World insert into t1 values (1,”Hello World!”); insert into t1 values (2,”Meet Continuent”); update t1 set msg=”Goodbye World” where ID = 1;
- 6. How Vertica Replication Works cpimport THL Target Replicator: Applier CSV JS JDBC merge
- 7. How Redshift Replication Works THL Target Replicator: Applier CSV JS JDBC merge S3 s3cmd copy
- 8. Prerequisites • Review online documentation • https://docs.continuent.com • Download the Prerequisite Checklist • Extractor/Applier Hosts • OS User • /etc/hosts • sudoers • Ruby • Java • Network • Review Port Requirements • MySQL • my.cnf settings • User accounts • Hadoop • HDFS writeable by replicator user • Vertica • User Accounts • JDBC Drivers • Redshift • User Accounts • S3 Bucket • S3 Tools for uploading • AWS JSON Config • All tables need Primary Keys
- 9. AWS JSON Config for Redshift • /opt/continuent/share/s3-config-<servicename>.json • awsS3Path — the location within your S3 storage where files should be loaded. • awsAccessKey — the S3 access key to access your S3 storage. Not required if awsIAMRole is used. • awsSecretKey — the S3 secret key associated with the Access Key. Not required if awsIAMRole is used. • awsIAMRole — the IAM role configured to allow Redshift to interact with S3. Not required if awsAccessKey and awsSecretKey are in use. • s3Binary — the binary to use for loading csv file up to S3. (Valid Values: s3cmd, s4cmd, aws) (Default: s3cmd) • cleanUpS3Files - a boolean value used to identify whether the CSV files loaded into S3 should be deleted after they have been imported and merged (Default: true)
- 10. Provisioning Options • Traditional CSV export and import • Dump and load through Blackhole engine • If target support standard SQL, extract data as INSERTS • For Hadoop, use Sqoop
- 11. Object Mapping • Hadoop • MySQL Database à HDFS Directory • Table à Hive Compatible CSV File • Row à Line in the file • Redshift & Vertica (PostgreSQL Interface and Syntax) • MySQL Instance à Database • MySQL Database à Schema
- 12. DDLScan • Reverse engineers MySQL objects • Creates target specific DDL • Can be used for single objects or entire databases • Must be configured prior to starting replicators • Must be run twice, once for base tables, once for staging tables ddlscan –service alpha -template ddl-mysql-redshift.vm -db test >ddl.sql ddlscan –service alpha -template ddl-mysql-redshift-staging.vm -db test >ddl-staging.sql
- 16. Demonstration
- 17. Summary What we have learnt today • Reviewed replicator flow • Explored Hadoop, Redshift and Vertica specific pre-requisites • Reviewed configurations
- 18. Next Steps In the next session we will • Learn how to use Tungsten Replicator with MongoDB
- 19. THANK YOU FOR LISTENING continuent.com The MySQL Availability Company Chris Parker, Customer Success Director, EMEA & APAC