SlideShare a Scribd company logo
The MySQL Availability Company
Tungsten Replicator Master Class
Intermediate: Replicator Monitoring & Troubleshooting
Chris Parker, Customer Success Director, EMEA & APAC
Topics
In this short course, we will
• Discuss Monitoring & Troubleshooting
• Review Common Issues
• Triggers
• Finding and understanding Log Files
• Handling (and recovering from) Failures
• Skipping Transactions
• Resetting
• Scripts for Monitoring
Most Common Issues
Common Incidents:
• Network Failures
• Disk Capacity
• OOM
• MySQL Crashes
• MySQL Max Connections
• Human Error
• Deletion of THL
• Incorrect Shutdown Procedures
• Writing direct to a replica
Things to check:
• Set up Monitoring / Alerts
• Check status outputs
• Check Log Files
• Check OS
• Check MySQL state
Triggers
• Triggers are an issue in all MySQL<>MySQL replication topologies
• The combination of the binlog_format (ROW vs MIXED) and the TRIGGER type (DETERMINISTIC vs
NONDETERMINISTIC) will result in triggers behaving differently.
• Either:
• Avoid Triggers altogether if possible, or
• Code Triggers to intelligently fire ONLY on Primary hosts
• Read the docs for a full explanation:
• https://docs.continuent.com/tungsten-clustering-6.1/troubleshooting-known-issues-
triggers.html
Log Files
• /opt/continuent/service_logs/
• trepsvc.log for complete replicator log
• Use logrotate to manage
• `replicator dump`
• Dumps memory stack to log file
• Can be useful in support cases
• Automatically issued by `tpm diag`
Recovering from Failures
• Enable auto-recovery properties
• Review log files to understand errors
• Simply try restarting replicator – YES this sometimes works!
• Which node is experiencing an error?
• If statement clash, is skipping safe?
• If database crashed, have you lost binary logs?
• Have you lost THL Files?
• Worse case – Reset and Re-provision
Monitoring Scripts
• Nagios-compatible scripts are located in
• These scripts will only check the local host. They do not check other hosts.
• check_tungsten_online – report ok if all services are online
• check_tungsten_services –r – report ok if replicator services are healthy
• check_tungsten_latency – used for Nagios style warning and critical replicator latency
thresholds
• Write your own! - New API coming in v7
$CONTINUENT_ROOT/tungsten/cluster-home/bin/check_tungsten_*
Summary
What we have learnt today
• Reviewed Common Issues
• Discussed Triggers
• Looked at Log Files
• Discussed Failures
• Skipping Transactions
• Resetting
• Reviewed Scripts for Monitoring
Next Steps
In the next session we will
• Explore Filtering!
THANK YOU FOR LISTENING
continuent.com
The MySQL Availability Company
Chris Parker, Customer Success Director, EMEA & APAC

More Related Content

What's hot (20)

Open Source Load Testing: JMeter, Gatling and Taurus
Open Source Load Testing: JMeter, Gatling and TaurusOpen Source Load Testing: JMeter, Gatling and Taurus
Open Source Load Testing: JMeter, Gatling and Taurus
Guy Salton
 
Antifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failureAntifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failure
DiUS
 
Gatling - Bordeaux JUG
Gatling - Bordeaux JUGGatling - Bordeaux JUG
Gatling - Bordeaux JUG
slandelle
 
Bsides Knoxville - APT2
Bsides Knoxville - APT2Bsides Knoxville - APT2
Bsides Knoxville - APT2
Adam Compton
 
The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"
Knoldus Inc.
 
Load Testing with Taurus using Jenkins and AWS
Load Testing with Taurus using Jenkins and AWSLoad Testing with Taurus using Jenkins and AWS
Load Testing with Taurus using Jenkins and AWS
Guy Salton
 
A Practical Event Driven Model
A Practical Event Driven ModelA Practical Event Driven Model
A Practical Event Driven Model
Xi Wu
 
CPAN Gems From The Far East
CPAN Gems From The Far EastCPAN Gems From The Far East
CPAN Gems From The Far East
lestrrat
 
Speed geeking-lotusscript
Speed geeking-lotusscriptSpeed geeking-lotusscript
Speed geeking-lotusscript
Bill Buchan
 
WebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck ThreadsWebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck Threads
Maarten Smeets
 
CNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
CNIT 127 Ch 16: Fault Injection and 17: The Art of FuzzingCNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
CNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
Sam Bowne
 
EUC2015 - Load testing XMPP servers with Plain Old Erlang
EUC2015 - Load testing XMPP servers with Plain Old ErlangEUC2015 - Load testing XMPP servers with Plain Old Erlang
EUC2015 - Load testing XMPP servers with Plain Old Erlang
Paweł Pikuła
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
Blazing Cloud
 
Take a Look at Akka+Java (English version)
Take a Look at Akka+Java (English version)Take a Look at Akka+Java (English version)
Take a Look at Akka+Java (English version)
GlobalLogic Ukraine
 
Tuenti Release Workflow
Tuenti Release WorkflowTuenti Release Workflow
Tuenti Release Workflow
Tuenti
 
Summit 16: Stop Writing Legacy Code!
Summit 16: Stop Writing Legacy Code!Summit 16: Stop Writing Legacy Code!
Summit 16: Stop Writing Legacy Code!
OPNFV
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applications
Ortus Solutions, Corp
 
Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012
slandelle
 
Introduction to GOCD - Amulya Sharma
Introduction to GOCD - Amulya SharmaIntroduction to GOCD - Amulya Sharma
Introduction to GOCD - Amulya Sharma
Amulya Sharma
 
Project Reactor By Example
Project Reactor By ExampleProject Reactor By Example
Project Reactor By Example
Denny Abraham Cheriyan
 
Open Source Load Testing: JMeter, Gatling and Taurus
Open Source Load Testing: JMeter, Gatling and TaurusOpen Source Load Testing: JMeter, Gatling and Taurus
Open Source Load Testing: JMeter, Gatling and Taurus
Guy Salton
 
Antifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failureAntifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failure
DiUS
 
Gatling - Bordeaux JUG
Gatling - Bordeaux JUGGatling - Bordeaux JUG
Gatling - Bordeaux JUG
slandelle
 
Bsides Knoxville - APT2
Bsides Knoxville - APT2Bsides Knoxville - APT2
Bsides Knoxville - APT2
Adam Compton
 
The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"
Knoldus Inc.
 
Load Testing with Taurus using Jenkins and AWS
Load Testing with Taurus using Jenkins and AWSLoad Testing with Taurus using Jenkins and AWS
Load Testing with Taurus using Jenkins and AWS
Guy Salton
 
A Practical Event Driven Model
A Practical Event Driven ModelA Practical Event Driven Model
A Practical Event Driven Model
Xi Wu
 
CPAN Gems From The Far East
CPAN Gems From The Far EastCPAN Gems From The Far East
CPAN Gems From The Far East
lestrrat
 
Speed geeking-lotusscript
Speed geeking-lotusscriptSpeed geeking-lotusscript
Speed geeking-lotusscript
Bill Buchan
 
WebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck ThreadsWebLogic Stability; Detect and Analyse Stuck Threads
WebLogic Stability; Detect and Analyse Stuck Threads
Maarten Smeets
 
CNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
CNIT 127 Ch 16: Fault Injection and 17: The Art of FuzzingCNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
CNIT 127 Ch 16: Fault Injection and 17: The Art of Fuzzing
Sam Bowne
 
EUC2015 - Load testing XMPP servers with Plain Old Erlang
EUC2015 - Load testing XMPP servers with Plain Old ErlangEUC2015 - Load testing XMPP servers with Plain Old Erlang
EUC2015 - Load testing XMPP servers with Plain Old Erlang
Paweł Pikuła
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
Blazing Cloud
 
Take a Look at Akka+Java (English version)
Take a Look at Akka+Java (English version)Take a Look at Akka+Java (English version)
Take a Look at Akka+Java (English version)
GlobalLogic Ukraine
 
Tuenti Release Workflow
Tuenti Release WorkflowTuenti Release Workflow
Tuenti Release Workflow
Tuenti
 
Summit 16: Stop Writing Legacy Code!
Summit 16: Stop Writing Legacy Code!Summit 16: Stop Writing Legacy Code!
Summit 16: Stop Writing Legacy Code!
OPNFV
 
Into The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applicationsInto The Box 2018 | Assert control over your legacy applications
Into The Box 2018 | Assert control over your legacy applications
Ortus Solutions, Corp
 
Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012Hands On, Duchess 10/17/2012
Hands On, Duchess 10/17/2012
slandelle
 
Introduction to GOCD - Amulya Sharma
Introduction to GOCD - Amulya SharmaIntroduction to GOCD - Amulya Sharma
Introduction to GOCD - Amulya Sharma
Amulya Sharma
 

Similar to Training Slides: 252 - Monitoring & Troubleshooting (20)

Buytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemakerBuytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemaker
kuchinskaya
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014
Lari Hotari
 
Gearman: A Job Server made for Scale
Gearman: A Job Server made for ScaleGearman: A Job Server made for Scale
Gearman: A Job Server made for Scale
Mike Willbanks
 
Metasploit & Windows Kernel Exploitation
Metasploit & Windows Kernel ExploitationMetasploit & Windows Kernel Exploitation
Metasploit & Windows Kernel Exploitation
zeroSteiner
 
High Performance Mysql
High Performance MysqlHigh Performance Mysql
High Performance Mysql
liufabin 66688
 
Gearman - Northeast PHP 2012
Gearman - Northeast PHP 2012Gearman - Northeast PHP 2012
Gearman - Northeast PHP 2012
Mike Willbanks
 
Got Problems? Let's Do a Health Check
Got Problems? Let's Do a Health CheckGot Problems? Let's Do a Health Check
Got Problems? Let's Do a Health Check
Luis Guirigay
 
MariaDB Galera Cluster
MariaDB Galera ClusterMariaDB Galera Cluster
MariaDB Galera Cluster
Abdul Manaf
 
Maria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High AvailabilityMaria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High Availability
OSSCube
 
Ansible Best Practices - July 30
Ansible Best Practices - July 30Ansible Best Practices - July 30
Ansible Best Practices - July 30
tylerturk
 
Practical Windows Kernel Exploitation
Practical Windows Kernel ExploitationPractical Windows Kernel Exploitation
Practical Windows Kernel Exploitation
zeroSteiner
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applications
Lari Hotari
 
Techno-Fest-15nov16
Techno-Fest-15nov16Techno-Fest-15nov16
Techno-Fest-15nov16
Satish Navkar
 
Fastest Servlets in the West
Fastest Servlets in the WestFastest Servlets in the West
Fastest Servlets in the West
Stuart (Pid) Williams
 
Asynchronous programming using CompletableFutures in Java
Asynchronous programming using CompletableFutures in JavaAsynchronous programming using CompletableFutures in Java
Asynchronous programming using CompletableFutures in Java
Oresztész Margaritisz
 
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good ServerICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
Serdar Basegmez
 
LMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging LibraryLMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging Library
Sebastian Andrasoni
 
Scaling apps for the big time
Scaling apps for the big timeScaling apps for the big time
Scaling apps for the big time
proitconsult
 
MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014
Ryusuke Kajiyama
 
Hot to build continuously processing for 24/7 real-time data streaming platform?
Hot to build continuously processing for 24/7 real-time data streaming platform?Hot to build continuously processing for 24/7 real-time data streaming platform?
Hot to build continuously processing for 24/7 real-time data streaming platform?
GetInData
 
Buytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemakerBuytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemaker
kuchinskaya
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014
Lari Hotari
 
Gearman: A Job Server made for Scale
Gearman: A Job Server made for ScaleGearman: A Job Server made for Scale
Gearman: A Job Server made for Scale
Mike Willbanks
 
Metasploit & Windows Kernel Exploitation
Metasploit & Windows Kernel ExploitationMetasploit & Windows Kernel Exploitation
Metasploit & Windows Kernel Exploitation
zeroSteiner
 
High Performance Mysql
High Performance MysqlHigh Performance Mysql
High Performance Mysql
liufabin 66688
 
Gearman - Northeast PHP 2012
Gearman - Northeast PHP 2012Gearman - Northeast PHP 2012
Gearman - Northeast PHP 2012
Mike Willbanks
 
Got Problems? Let's Do a Health Check
Got Problems? Let's Do a Health CheckGot Problems? Let's Do a Health Check
Got Problems? Let's Do a Health Check
Luis Guirigay
 
MariaDB Galera Cluster
MariaDB Galera ClusterMariaDB Galera Cluster
MariaDB Galera Cluster
Abdul Manaf
 
Maria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High AvailabilityMaria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High Availability
OSSCube
 
Ansible Best Practices - July 30
Ansible Best Practices - July 30Ansible Best Practices - July 30
Ansible Best Practices - July 30
tylerturk
 
Practical Windows Kernel Exploitation
Practical Windows Kernel ExploitationPractical Windows Kernel Exploitation
Practical Windows Kernel Exploitation
zeroSteiner
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applications
Lari Hotari
 
Asynchronous programming using CompletableFutures in Java
Asynchronous programming using CompletableFutures in JavaAsynchronous programming using CompletableFutures in Java
Asynchronous programming using CompletableFutures in Java
Oresztész Margaritisz
 
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good ServerICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
Serdar Basegmez
 
LMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging LibraryLMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging Library
Sebastian Andrasoni
 
Scaling apps for the big time
Scaling apps for the big timeScaling apps for the big time
Scaling apps for the big time
proitconsult
 
MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014
Ryusuke Kajiyama
 
Hot to build continuously processing for 24/7 real-time data streaming platform?
Hot to build continuously processing for 24/7 real-time data streaming platform?Hot to build continuously processing for 24/7 real-time data streaming platform?
Hot to build continuously processing for 24/7 real-time data streaming platform?
GetInData
 

More from Continuent (20)

Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Continuent
 
Continuent Tungsten Value Proposition Webinar
Continuent Tungsten Value Proposition WebinarContinuent Tungsten Value Proposition Webinar
Continuent Tungsten Value Proposition Webinar
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControlWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB ClusterWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQLWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera ClusterWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS AuroraWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Continuent
 
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Continuent
 
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Continuent
 
Webinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Webinar Slides: Intelligent Database Proxies: Routing & Transparent FailoverWebinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Webinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Continuent
 
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Continuent
 
Training Slides: 205 - Installing and Configuring Tungsten Dashboard
Training Slides: 205 - Installing and Configuring Tungsten DashboardTraining Slides: 205 - Installing and Configuring Tungsten Dashboard
Training Slides: 205 - Installing and Configuring Tungsten Dashboard
Continuent
 
Training Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Training Slides: 352 - Tungsten Replicator for MongoDB & KafkaTraining Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Training Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Continuent
 
Training Slides: 351 - Tungsten Replicator for Data Warehouses
Training Slides: 351 - Tungsten Replicator for Data WarehousesTraining Slides: 351 - Tungsten Replicator for Data Warehouses
Training Slides: 351 - Tungsten Replicator for Data Warehouses
Continuent
 
Training Slides: 303 - Replicating out of a Cluster
Training Slides: 303 - Replicating out of a ClusterTraining Slides: 303 - Replicating out of a Cluster
Training Slides: 303 - Replicating out of a Cluster
Continuent
 
Training Slides: 206 - Using the Tungsten Cluster AMI
Training Slides: 206 - Using the Tungsten Cluster AMITraining Slides: 206 - Using the Tungsten Cluster AMI
Training Slides: 206 - Using the Tungsten Cluster AMI
Continuent
 
Training Slides: 254 - Using the Tungsten Replicator AMI
Training Slides: 254 - Using the Tungsten Replicator AMITraining Slides: 254 - Using the Tungsten Replicator AMI
Training Slides: 254 - Using the Tungsten Replicator AMI
Continuent
 
Training Slides: 253 - Filter like a Pro
Training Slides: 253 - Filter like a ProTraining Slides: 253 - Filter like a Pro
Training Slides: 253 - Filter like a Pro
Continuent
 
Training Slides: 302 - Securing Your Cluster With SSL
Training Slides: 302 - Securing Your Cluster With SSLTraining Slides: 302 - Securing Your Cluster With SSL
Training Slides: 302 - Securing Your Cluster With SSL
Continuent
 
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Continuent
 
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Continuent
 
Continuent Tungsten Value Proposition Webinar
Continuent Tungsten Value Proposition WebinarContinuent Tungsten Value Proposition Webinar
Continuent Tungsten Value Proposition Webinar
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControlWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #7: ClusterControl
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB ClusterWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #5: Oracle’s InnoDB Cluster
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQLWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #4: MS Azure Database MySQL
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera ClusterWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #2: Galera Cluster
Continuent
 
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS AuroraWebinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Webinar Slides: MySQL HA/DR/Geo-Scale - High Noon #1: AWS Aurora
Continuent
 
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Webinar Slides: AWS Aurora MySQL Replacement: Break Away From Geo-Limitations...
Continuent
 
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Webinar Slides: No Data Loss MySQL: Guaranteed Credit Card Transaction Availa...
Continuent
 
Webinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Webinar Slides: Intelligent Database Proxies: Routing & Transparent FailoverWebinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Webinar Slides: Intelligent Database Proxies: Routing & Transparent Failover
Continuent
 
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Continuent
 
Training Slides: 205 - Installing and Configuring Tungsten Dashboard
Training Slides: 205 - Installing and Configuring Tungsten DashboardTraining Slides: 205 - Installing and Configuring Tungsten Dashboard
Training Slides: 205 - Installing and Configuring Tungsten Dashboard
Continuent
 
Training Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Training Slides: 352 - Tungsten Replicator for MongoDB & KafkaTraining Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Training Slides: 352 - Tungsten Replicator for MongoDB & Kafka
Continuent
 
Training Slides: 351 - Tungsten Replicator for Data Warehouses
Training Slides: 351 - Tungsten Replicator for Data WarehousesTraining Slides: 351 - Tungsten Replicator for Data Warehouses
Training Slides: 351 - Tungsten Replicator for Data Warehouses
Continuent
 
Training Slides: 303 - Replicating out of a Cluster
Training Slides: 303 - Replicating out of a ClusterTraining Slides: 303 - Replicating out of a Cluster
Training Slides: 303 - Replicating out of a Cluster
Continuent
 
Training Slides: 206 - Using the Tungsten Cluster AMI
Training Slides: 206 - Using the Tungsten Cluster AMITraining Slides: 206 - Using the Tungsten Cluster AMI
Training Slides: 206 - Using the Tungsten Cluster AMI
Continuent
 
Training Slides: 254 - Using the Tungsten Replicator AMI
Training Slides: 254 - Using the Tungsten Replicator AMITraining Slides: 254 - Using the Tungsten Replicator AMI
Training Slides: 254 - Using the Tungsten Replicator AMI
Continuent
 
Training Slides: 253 - Filter like a Pro
Training Slides: 253 - Filter like a ProTraining Slides: 253 - Filter like a Pro
Training Slides: 253 - Filter like a Pro
Continuent
 
Training Slides: 302 - Securing Your Cluster With SSL
Training Slides: 302 - Securing Your Cluster With SSLTraining Slides: 302 - Securing Your Cluster With SSL
Training Slides: 302 - Securing Your Cluster With SSL
Continuent
 
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Webinar Slides: Global MySQL Availability: SaaS Cloud Contact Center Secures ...
Continuent
 

Recently uploaded (19)

DNS Resolvers and Nameservers (in New Zealand)
DNS Resolvers and Nameservers (in New Zealand)DNS Resolvers and Nameservers (in New Zealand)
DNS Resolvers and Nameservers (in New Zealand)
APNIC
 
Best web hosting Vancouver 2025 for you business
Best web hosting Vancouver 2025 for you businessBest web hosting Vancouver 2025 for you business
Best web hosting Vancouver 2025 for you business
steve198109
 
OSI TCP IP Protocol Layers description f
OSI TCP IP Protocol Layers description fOSI TCP IP Protocol Layers description f
OSI TCP IP Protocol Layers description f
cbr49917
 
Computers Networks Computers Networks Computers Networks
Computers Networks Computers Networks Computers NetworksComputers Networks Computers Networks Computers Networks
Computers Networks Computers Networks Computers Networks
Tito208863
 
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC
 
Understanding the Tor Network and Exploring the Deep Web
Understanding the Tor Network and Exploring the Deep WebUnderstanding the Tor Network and Exploring the Deep Web
Understanding the Tor Network and Exploring the Deep Web
nabilajabin35
 
project_based_laaaaaaaaaaearning,kelompok 10.pptx
project_based_laaaaaaaaaaearning,kelompok 10.pptxproject_based_laaaaaaaaaaearning,kelompok 10.pptx
project_based_laaaaaaaaaaearning,kelompok 10.pptx
redzuriel13
 
Perguntas dos animais - Slides ilustrados de múltipla escolha
Perguntas dos animais - Slides ilustrados de múltipla escolhaPerguntas dos animais - Slides ilustrados de múltipla escolha
Perguntas dos animais - Slides ilustrados de múltipla escolha
socaslev
 
5-Proses-proses Akuisisi Citra Digital.pptx
5-Proses-proses Akuisisi Citra Digital.pptx5-Proses-proses Akuisisi Citra Digital.pptx
5-Proses-proses Akuisisi Citra Digital.pptx
andani26
 
White and Red Clean Car Business Pitch Presentation.pptx
White and Red Clean Car Business Pitch Presentation.pptxWhite and Red Clean Car Business Pitch Presentation.pptx
White and Red Clean Car Business Pitch Presentation.pptx
canumatown
 
APNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC Update, presented at NZNOG 2025 by Terry SweetserAPNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC
 
highend-srxseries-services-gateways-customer-presentation.pptx
highend-srxseries-services-gateways-customer-presentation.pptxhighend-srxseries-services-gateways-customer-presentation.pptx
highend-srxseries-services-gateways-customer-presentation.pptx
elhadjcheikhdiop
 
Smart Mobile App Pitch Deck丨AI Travel App Presentation Template
Smart Mobile App Pitch Deck丨AI Travel App Presentation TemplateSmart Mobile App Pitch Deck丨AI Travel App Presentation Template
Smart Mobile App Pitch Deck丨AI Travel App Presentation Template
yojeari421237
 
Reliable Vancouver Web Hosting with Local Servers & 24/7 Support
Reliable Vancouver Web Hosting with Local Servers & 24/7 SupportReliable Vancouver Web Hosting with Local Servers & 24/7 Support
Reliable Vancouver Web Hosting with Local Servers & 24/7 Support
steve198109
 
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHostingTop Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
steve198109
 
Mobile database for your company telemarketing or sms marketing campaigns. Fr...
Mobile database for your company telemarketing or sms marketing campaigns. Fr...Mobile database for your company telemarketing or sms marketing campaigns. Fr...
Mobile database for your company telemarketing or sms marketing campaigns. Fr...
DataProvider1
 
Determining Glass is mechanical textile
Determining  Glass is mechanical textileDetermining  Glass is mechanical textile
Determining Glass is mechanical textile
Azizul Hakim
 
IT Services Workflow From Request to Resolution
IT Services Workflow From Request to ResolutionIT Services Workflow From Request to Resolution
IT Services Workflow From Request to Resolution
mzmziiskd
 
(Hosting PHising Sites) for Cryptography and network security
(Hosting PHising Sites) for Cryptography and network security(Hosting PHising Sites) for Cryptography and network security
(Hosting PHising Sites) for Cryptography and network security
aluacharya169
 
DNS Resolvers and Nameservers (in New Zealand)
DNS Resolvers and Nameservers (in New Zealand)DNS Resolvers and Nameservers (in New Zealand)
DNS Resolvers and Nameservers (in New Zealand)
APNIC
 
Best web hosting Vancouver 2025 for you business
Best web hosting Vancouver 2025 for you businessBest web hosting Vancouver 2025 for you business
Best web hosting Vancouver 2025 for you business
steve198109
 
OSI TCP IP Protocol Layers description f
OSI TCP IP Protocol Layers description fOSI TCP IP Protocol Layers description f
OSI TCP IP Protocol Layers description f
cbr49917
 
Computers Networks Computers Networks Computers Networks
Computers Networks Computers Networks Computers NetworksComputers Networks Computers Networks Computers Networks
Computers Networks Computers Networks Computers Networks
Tito208863
 
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC -Policy Development Process, presented at Local APIGA Taiwan 2025
APNIC
 
Understanding the Tor Network and Exploring the Deep Web
Understanding the Tor Network and Exploring the Deep WebUnderstanding the Tor Network and Exploring the Deep Web
Understanding the Tor Network and Exploring the Deep Web
nabilajabin35
 
project_based_laaaaaaaaaaearning,kelompok 10.pptx
project_based_laaaaaaaaaaearning,kelompok 10.pptxproject_based_laaaaaaaaaaearning,kelompok 10.pptx
project_based_laaaaaaaaaaearning,kelompok 10.pptx
redzuriel13
 
Perguntas dos animais - Slides ilustrados de múltipla escolha
Perguntas dos animais - Slides ilustrados de múltipla escolhaPerguntas dos animais - Slides ilustrados de múltipla escolha
Perguntas dos animais - Slides ilustrados de múltipla escolha
socaslev
 
5-Proses-proses Akuisisi Citra Digital.pptx
5-Proses-proses Akuisisi Citra Digital.pptx5-Proses-proses Akuisisi Citra Digital.pptx
5-Proses-proses Akuisisi Citra Digital.pptx
andani26
 
White and Red Clean Car Business Pitch Presentation.pptx
White and Red Clean Car Business Pitch Presentation.pptxWhite and Red Clean Car Business Pitch Presentation.pptx
White and Red Clean Car Business Pitch Presentation.pptx
canumatown
 
APNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC Update, presented at NZNOG 2025 by Terry SweetserAPNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC Update, presented at NZNOG 2025 by Terry Sweetser
APNIC
 
highend-srxseries-services-gateways-customer-presentation.pptx
highend-srxseries-services-gateways-customer-presentation.pptxhighend-srxseries-services-gateways-customer-presentation.pptx
highend-srxseries-services-gateways-customer-presentation.pptx
elhadjcheikhdiop
 
Smart Mobile App Pitch Deck丨AI Travel App Presentation Template
Smart Mobile App Pitch Deck丨AI Travel App Presentation TemplateSmart Mobile App Pitch Deck丨AI Travel App Presentation Template
Smart Mobile App Pitch Deck丨AI Travel App Presentation Template
yojeari421237
 
Reliable Vancouver Web Hosting with Local Servers & 24/7 Support
Reliable Vancouver Web Hosting with Local Servers & 24/7 SupportReliable Vancouver Web Hosting with Local Servers & 24/7 Support
Reliable Vancouver Web Hosting with Local Servers & 24/7 Support
steve198109
 
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHostingTop Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
Top Vancouver Green Business Ideas for 2025 Powered by 4GoodHosting
steve198109
 
Mobile database for your company telemarketing or sms marketing campaigns. Fr...
Mobile database for your company telemarketing or sms marketing campaigns. Fr...Mobile database for your company telemarketing or sms marketing campaigns. Fr...
Mobile database for your company telemarketing or sms marketing campaigns. Fr...
DataProvider1
 
Determining Glass is mechanical textile
Determining  Glass is mechanical textileDetermining  Glass is mechanical textile
Determining Glass is mechanical textile
Azizul Hakim
 
IT Services Workflow From Request to Resolution
IT Services Workflow From Request to ResolutionIT Services Workflow From Request to Resolution
IT Services Workflow From Request to Resolution
mzmziiskd
 
(Hosting PHising Sites) for Cryptography and network security
(Hosting PHising Sites) for Cryptography and network security(Hosting PHising Sites) for Cryptography and network security
(Hosting PHising Sites) for Cryptography and network security
aluacharya169
 

Training Slides: 252 - Monitoring & Troubleshooting

  • 1. The MySQL Availability Company Tungsten Replicator Master Class Intermediate: Replicator Monitoring & Troubleshooting Chris Parker, Customer Success Director, EMEA & APAC
  • 2. Topics In this short course, we will • Discuss Monitoring & Troubleshooting • Review Common Issues • Triggers • Finding and understanding Log Files • Handling (and recovering from) Failures • Skipping Transactions • Resetting • Scripts for Monitoring
  • 3. Most Common Issues Common Incidents: • Network Failures • Disk Capacity • OOM • MySQL Crashes • MySQL Max Connections • Human Error • Deletion of THL • Incorrect Shutdown Procedures • Writing direct to a replica Things to check: • Set up Monitoring / Alerts • Check status outputs • Check Log Files • Check OS • Check MySQL state
  • 4. Triggers • Triggers are an issue in all MySQL<>MySQL replication topologies • The combination of the binlog_format (ROW vs MIXED) and the TRIGGER type (DETERMINISTIC vs NONDETERMINISTIC) will result in triggers behaving differently. • Either: • Avoid Triggers altogether if possible, or • Code Triggers to intelligently fire ONLY on Primary hosts • Read the docs for a full explanation: • https://docs.continuent.com/tungsten-clustering-6.1/troubleshooting-known-issues- triggers.html
  • 5. Log Files • /opt/continuent/service_logs/ • trepsvc.log for complete replicator log • Use logrotate to manage • `replicator dump` • Dumps memory stack to log file • Can be useful in support cases • Automatically issued by `tpm diag`
  • 6. Recovering from Failures • Enable auto-recovery properties • Review log files to understand errors • Simply try restarting replicator – YES this sometimes works! • Which node is experiencing an error? • If statement clash, is skipping safe? • If database crashed, have you lost binary logs? • Have you lost THL Files? • Worse case – Reset and Re-provision
  • 7. Monitoring Scripts • Nagios-compatible scripts are located in • These scripts will only check the local host. They do not check other hosts. • check_tungsten_online – report ok if all services are online • check_tungsten_services –r – report ok if replicator services are healthy • check_tungsten_latency – used for Nagios style warning and critical replicator latency thresholds • Write your own! - New API coming in v7 $CONTINUENT_ROOT/tungsten/cluster-home/bin/check_tungsten_*
  • 8. Summary What we have learnt today • Reviewed Common Issues • Discussed Triggers • Looked at Log Files • Discussed Failures • Skipping Transactions • Resetting • Reviewed Scripts for Monitoring
  • 9. Next Steps In the next session we will • Explore Filtering!
  • 10. THANK YOU FOR LISTENING continuent.com The MySQL Availability Company Chris Parker, Customer Success Director, EMEA & APAC