Loadays managing my sql with percona toolkit

Managing MySQL with
Percona Toolkit

LOADAYS 2012 – April 1st

Who am I ?
Frédéric Descamps
@lefred
http://about.me/lefred
Percona Consultant
Managing mysql since 3.23 (as far as I
remember)
devops believer

www.percona.com

Agenda

● What is Percona
Toolkit ?
● What tools are
included and what
can they do ?
● Quick intro to several
tools
● Resources

www.percona.com

Percona Toolkit

● Origin: Maatkit and Aspersa (both projects
started by Baron Schwartz with Daniel Nichter's
contribution)
● Indispensable tools for MySQL administrators
● Opensource GNU GPL v2
● https://launchpad.net/percona-toolkit
● http://www.percona.com/software/percona-toolkit/
● Current version is 2.0.4

www.percona.com

Other Toolkits

www.percona.com

Percona Toolkit ;-)

www.percona.com

More seriously:

www.percona.com

Installation
●
Requirements:
➢
Perl, DBI, DBD::pgsql, Wine or Mono
➢
Better performance with Wine
●
Download zip:
➢
http://megaupload.com/PTK204
➢
It's in Percona's MSN repository
●
We are working on a Java version
●
Easy to run: wine pt-diskstats_win32.exe

www.percona.com

Installation
●
Requirements:
➢
Perl, DBI, DBD::mysql, Term::ReadKey
●
Download tarball, RPM or DEB packages:
➢
wget percona.com/get/percona-toolkit.rpm
➢
It's in Percona repository
●
Download individual tools:
➢
wget percona.com/get/pt-diskstats

www.percona.com

Know your environment:
Hardware & Operating System

www.percona.com 11

Know your environment
(Hardware & OS)
We need to find the answers to these questions:
● On what kind of machine runs my db ? Physical/Virtual ?
● How much memory is used/free ? Does it swap ? Can I add more ?
● How many CPUs and CPU cores ? Speed ?
● Is this a 64bit machine ?
● What kind of RAID controller ? Volumes ? Disks ?
● What about network interfaces ? Connections ? Processes ?
● What's the Operating System ? Kernel ? Release ?
● Which IO scheduler is in use ?
● On which filesystem is the data stored ?
● Do we have free space ?

www.percona.com 12

Summary !
Name: pt-summary
Function: summarize system information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-summary.html

www.percona.com 13

Taking inventory
$ wget percona.com/get/pt-summary
$ chmod +x pt-summary
$ ./pt-summary
# Percona Toolkit System Summary Report ######################
Date | 2012-02-02 12:25:18 UTC (local TZ: EET +0200)
Hostname | db01
Uptime | 42 days, 16:14, 1 user, load average: 1.15, 1.22, 1.08
System | HP; ProLiant DL580 G7; vNot Specified (Rack Mount Chassis)
Service Tag | FOSDEM2012
Platform | Linux
Release | CentOS release 6.1 (Final)
Kernel | 2.6.32-131.21.1.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
Compiler | GNU CC version 4.4.5 20110214 (Red Hat 4.4.5-6).
SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##################################################
Processors | physical = 4, cores = 40, virtual = 80, hyperthreading = yes
Speeds | 80x2397.393
Models | 80xIntel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz

[ Further output omitted ]

www.percona.com 14

Know your environment:
MySQL

www.percona.com 15

Know your environment (MySQL)
We need to find the answers to these questions:
● Version ? Build ?
● How many database ? Where is the data directory ?
● What about replication ? How many replicas ?
● Do you use some Key XtraDB features ?
● What are the Key InnoDB settings ?
● Which storage engines are used ?
● What types of data fields are used ? Index types ?
● Do we use foreign keys ? Views ? Triggers ?
● Do we use Full-text indexes ? Partitioning ?
● What about Privileges ?

www.percona.com 16

Summary... again !
Name: pt-mysql-summary
Function: Summarize MySQL information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-mysql-summary.html

www.percona.com 17

Taking inventory (cont.)
$ wget percona.com/get/pt-mysql-summary
$ chmod +x pt-mysql-summary
$ ./pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
System time | 2012-02-02 12:24:37 UTC (local TZ: EET +0200)
# Instances ##################################################
Port Data Directory Socket
===== ========================== ======
/mysql
# Report On Port 3306 ########################################
User | root@localhost
Time | 2012-02-02 14:24:38 (EET)
Hostname | db01
Version | 5.5.18-55-log Percona
Built On | Linux x86_64
Started | 2011-12-26 21:09 (up 37+17:15:35)
Databases | 6
Datadir | /mysql/
Processes | 10 connected, 4 running
Replication | Is a slave, has 1 slaves connected
Pidfile | /mysql/mysqld.pid (exists)

[ Further output omitted ]

www.percona.com 18

Master ? Slave ?... both ?
Name: pt-slave-find
Function: find and print replication hierarchy
tree of MySQL slaves.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-find.html

www.percona.com 19

Inventory of Replicas
$ ./pt-slave-find localhost
localhost
Version 5.5.18-55-log
Server ID 11
Uptime 37+17:45:50 (started 2011-12-26T21:09:03)
Replication Is a slave, has 1 slaves connected, is not read_only
Filters
Binary logging MIXED
Slave status 0 seconds behind, running, no errors
Slave mode STRICT
Auto-increment increment 2, offset 1
InnoDB version 1.1.8-rel23.0
+- 10.0.0.17
Version 5.5.18-55-log
Server ID 12
Uptime 27+21:02:35 (started 2012-01-05T17:52:18)
Replication Is a slave, has 1 slaves connected, is not read_only
Filters
Binary logging MIXED
Slave status 0 seconds behind, running, no errors
Slave mode STRICT
Auto-increment increment 2, offset 2
InnoDB version 1.1.8-rel23.0

www.percona.com 20

Where's My Disk I/O Going ?
● Use pt-diskstats as an improved iostat
➢ Interactively controllable, slice-and-dice
➢ Doesn't lump reads and writes together

➢ Meaningful, correct columns

● Use pt-ioprofile to measure & time your
I/O
➢ Which files is MySQL using ? How much times does it
spend reading, writing, syncing, etc? How many
reads/writes? How large are they ?
pt-ioprofile can be dangerous in production
www.percona.com 21

Disk I/O
Name: pt-diskstats
Function: Aggregate and summarize
/proc/diskstats
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-diskstats.html

www.percona.com 22

Disk I/O
# ./pt-diskstats

#ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime

{1} sdb 158.0 5.8 0.9 0% 0.8 4.9 392.1 24.3 9.3 51% 0.6 0.7 54% 0 550.1 0.9 0.6
{1} sdb1 158.0 5.8 0.9 0% 0.8 4.9 392.1 24.3 9.3 51% 0.6 0.7 54% 0 550.1 0.9 0.6

{2} sda 0.5 4.0 0.0 0% 0.0 8.0 3.0 59.3 0.2 93% 0.0 0.2 1% 0 3.5 0.0 0.3
{2} sda2 0.0 0.0 0.0 0% 0.0 0.0 1.0 16.0 0.0 75% 0.0 1.0 0% 0 1.0 0.0 1.0
{2} sda5 0.5 4.0 0.0 0% 0.0 8.0 2.0 81.0 0.2 95% 0.0 0.1 1% 0 2.5 0.0 0.2
{2} sdb 150.0 8.7 1.3 3% 1.1 6.9 448.5 40.6 17.8 84% 1.4 0.5 53% 0 598.6 0.6 0.2
{2} sdb1 143.0 8.9 1.2 3% 1.0 6.8 400.5 24.7 9.7 49% 0.7 0.9 51% 0 543.6 1.3 0.5
{2} sdb2 7.0 4.0 0.0 0% 0.1 10.0 48.0 173.0 8.1 98% 0.7 0.3 9% 0 55.0 0.3 0.0

wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt
392.1 24.3 9.3 51% 0.6 0.7
392.1 24.3 9.3 51% 0.6 0.7

3.0 59.3 0.2 93% 0.0 0.2
1.0 16.0 0.0 75% 0.0 1.0
2.0 81.0 0.2 95% 0.0 0.1
448.5 40.6 17.8 84% 1.4 0.5
400.5 24.7 9.7 49% 0.7 0.9
48.0 173.0 8.1 98% 0.7 0.3

www.percona.com 23

I/O Profile
Name: pt-ioprofile
Function: Watch process IO and print a table
of file and I/O activity.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-ioprofile.html

www.percona.com 24

I/O profile
# ./pt-ioprofile
Sat Feb 4 13:44:08 CET 2012
Tracing process ID 24179
total read write open close lseek filename
0.007547 0.000583 0.004245 0.002181 0.000268 0.000270 /tmp/#sql_5e73_1.MYI
0.000686 0.000000 0.000000 0.000420 0.000266 0.000000 /tmp/#sql_5e73_1.MYD
0.000065 0.000000 0.000024 0.000029 0.000012 0.000000 /var/lib/mysql/fred/db.opt
0.000055 0.000000 0.000000 0.000055 0.000000 0.000000 /var/lib/mysql/mysql/db.opt
0.000046 0.000000 0.000000 0.000046 0.000000 0.000000 /var/lib/mysql/test/db.opt

www.percona.com 25

Be intimate with your database

www.percona.com 26

Let's try to find the answer to these
indiscreet questions:

● How are the indexes used ?
● Are there duplicate keys ?
● Which queries are eating most of the
resources ?

www.percona.com 27

Analyze Schema & Queries
● Duplicate/redundant indexes or foreign keys?
➢ pt-duplicate-key-checker can tell you that
● Which indexes are unused?
➢ pt-index-usage can tell you that
➢ It can also show which queries have unstable
plans, which indexes have alternatives that are
never chosen, and lots more
● Bad SQL?
➢ pt-query-advisor to the rescue

www.percona.com 28

Analyze & Profile Your Workload
● Use pt-query-digest to analyze the “slow
query log” and show a profile of the workload
● It does the Right Thing by default
● Mostly used with slow query logs & tcpdumps

www.percona.com 29

The Report
● Each class of query is ranked by execution time
● There is a report for each class of query
● It's easy to use
● It's a great way to start optimizing queries

www.percona.com 30

Overview Report
# 13903.5s user time, 39.9s system time, 1.78G rss, 1.92G vsz
# Current date: Mon Jan 30 20:36:26 2012
# Hostname: hp-db01
# Files: /var/lib/mysql/db01-slow.log
# Overall: 32.98M total, 1.32k unique, 1.73k QPS, 2.06x concurrency ______
# Time range: 2012-01-30 11:08:22 to 16:25:40
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 39309s 1us 161s 1ms 515us 109ms 30us
# Lock time 4471s 0 161s 135us 89us 90ms 0
# Rows sent 118.72M 0 1010.29k 3.77 1.96 963.89 0
# Rows examine 15.54G 0 3.51M 505.92 1.96 11.61k 0
# Rows affecte 1.64M 0 2.85k 0.05 0 1.82 0
# Rows read 118.72M 0 1010.29k 3.77 1.96 963.89 0
# Bytes sent 52.40G 0 45.18M 1.67k 5.20k 20.30k 537.02
# Merge passes 128 0 1 0.00 0 0.00 0
# Tmp tables 957.76k 0 4 0.03 0 0.17 0
# Tmp disk tbl 297.92k 0 2 0.01 0 0.10 0
# Tmp tbl size 5.21T 0 2.22G 169.59k 0 1.78M 0
# Query size 8.52G 6 130.40k 277.38 833.10 709.99 174.84
# InnoDB:
# IO r bytes 672.00k 0 32.00k 0.06 0 35.90 0
# IO r ops 42 0 2 0.00 0 0.00 0
# IO r wait 19ms 0 2ms 0 0 1us 0
# pages distin 329.98M 0 32.08k 28.18 42.48 307.62 4.96
# queue wait 0 0 0 0 0 0 0
# rec lock wai 3402s 0 161s 277us 0 147ms 0
# Boolean:
# Filesort 2% yes, 97% no
# Filesort on 0% yes, 99% no
# Full join 0% yes, 99% no
# Full scan 2% yes, 97% no
# QC Hit 55% yes, 44% no
# Tmp table 2% yes, 97% no
# Tmp table on 0% yes, 99% no

www.percona.com 31

Profile Report

# Profile
# Rank Query ID Response time Calls R/Call Apdx V/M Item
# ==== ================== =============== ======== ======= ==== ===== ====
# 1 0x844F3DA35A4A426D 8695.6994 22.1% 41470 0.2097 1.00 0.12 SELECT catalog
# 2 0xD566217EB8EB1A27 4098.9154 10.4% 48740 0.0841 1.00 0.11 SELECT geoip
# 3 0xB0A48B410B3CC151 2684.9671 6.8% 55 48.8176 0.00 34.18 UPDATE SELECT sales_
# 4 0x60CDD6CAE6B487BC 2541.5271 6.5% 352 7.2202 0.20 4.01 SELECT catalog catalog_category_product
# 5 0xD9DED53EF74C877F 1285.6118 3.3% 5530829 0.0002 1.00 0.00 SELECT op_image
# 6 0x1F3D58843008ACEC 1249.1639 3.2% 182 6.8635 0.12 1.65 SELECT catalog_product
# 7 0x50175ED6D1B1C8B1 1188.1541 3.0% 153964 0.0077 1.00 0.00 SELECT enterprise
# 8 0x70E2BE84868444A4 1035.3475 2.6% 1786 0.5797 0.98 0.16 SELECT sales_order
# 9 0x125DFE84FFA50675 891.3327 2.3% 382 2.3333 0.97 11... INSERT UPDATE product
# 10 0xBB968DD9BF27E8E5 797.3765 2.0% 153964 0.0052 1.00 0.00 SELECT enterprise

www.percona.com 32

Per Query Report
# Query 1: 2.25 QPS, 0.47x concurrency, ID 0x844F3DA35A4A426D at byte 4779840203
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 0 41470
# Exec time 22 8696s 97ms 3s 210ms 455ms 159ms 155ms
# Lock time 0 7s 107us 50ms 177us 247us 272us 159us
# Rows sent 1 1.60M 26 50 40.45 49.17 10.74 46.83
# Rows examine 16 2.63G 66.43k 66.60k 66.49k 65.68k 0 65.68k
# Rows affecte 0 0 0 0 0 0 0 0
# Rows read 1 1.60M 26 50 40.45 49.17 10.74 46.83
# Bytes sent 0 200.43M 3.39k 6.14k 4.95k 6.01k 1.08k 5.45k
# Merge passes 0 0 0 0 0 0 0 0
# Tmp tables 4 40.50k 1 1 1 1 0 1
# Tmp disk tbl 0 0 0 0 0 0 0 0
# Tmp tbl size 12 647.83G 16.00M 16.00M 16.00M 16.00M 0 16.00M
# Query size 0 71.78M 1.77k 1.77k 1.77k 1.77k 0 1.77k
# InnoDB:
# IO r bytes 0 0 0 0 0 0 0 0
# IO r ops 0 0 0 0 0 0 0 0
# IO r wait 0 0 0 0 0 0 0 0
# pages distin 47 158.13M 3.90k 3.92k 3.90k 3.88k 0.00 3.88k
# queue wait 0 0 0 0 0 0 0 0
# rec lock wai 0 0 0 0 0 0 0 0
# Boolean:
# Filesort 100% yes, 0% no
# Full scan 100% yes, 0% no
# Tmp table 100% yes, 0% no
# String:
# Databases prod
# Hosts
# InnoDB trxID 3CB60AED (1/0%), 3CB60B3E (1/0%)... 41468 more
# Last errno 0
# Users prod
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms
# 100ms ################################################################
# 1s #
# 10s+

www.percona.com 33

Maintain your environment

www.percona.com 34

Log & Analyze Errors ?
● Use pt-deadlock-logger to capture
deadlocks
● Use pt-fk-error-logger for foreign key
errors
● The tools give you the possibility to log the
results to a database table for analysis

www.percona.com 35

Monitor dead locks
Name: pt-deadlock-logger
Function: extract and log MySQL deadlock
information
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-deadlock-logger.html
● SHOW ENGINE INNODB STATUS reports only the most
recent deadlock
● Information in status output lacks details about user, client
host, other locks in the deadlock cycle.
● pt-deadlock-logger formats the needed information
nicely and saves it
www.percona.com 36

Monitor Foreign Key Errors
Name: pt-fk-error-logger
Function: extract and log MySQL foreign key
errors.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-fk-error-logger.html
● Like pt-deadlock-logger, this tool collects more useful
information, formats it nicely and saves it.

www.percona.com 37

Non-Blocking ALTER TABLE
● Use pt-online-schema-change to alter
tables
➢ It makes a “shadow copy” and swaps them

➢ Extremely useful for large, long-running

ALTER
➢ Facebook uses the same technique

www.percona.com 38

Validate Upgrades
● Upgrades are a leading cause of downtime
➢ Are any queries using different indexes ? Is QEP
different ?
➢ Are there new errors or warnings ?

➢ Do queries return the same results ?

➢ Are there significant response time differences ?

● Validate the results before you upgrade!
➢ Capture a snapshot of the data
➢ Capture a log of queries

➢ Compare the queries on two test servers running the

old and the new version
www.percona.com 39

Validate upgrades
Name: pt-upgrade
Function: Execute queries on multiple
servers and check for differences.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-upgrade.html

www.percona.com 40

pt-upgrade
# Query 2: ID 0xE2C8DC0286B7F44E at byte 0 _______________________________
# Found 23281 differences in 58207 samples:
# checksums 0
# column counts 0
# column types 0
# query times 23281
# row counts 0
# warning counts 0
# warning levels 0
# warnings 0
# AF002020:3306 AF001885:3306
# Errors 0 0
# Warnings 0 0
# Query_time
# sum 34s 70s
# min 395us 342us
# max 218ms 308ms
# avg 587us 1ms
# pct_95 725us 3ms
# stddev 2ms 4ms
# median 541us 690us
# row_count
# sum 1.75k 1.75k
# min 0 0
# max 1 1
# avg 0.03 0.03
# pct_95 0 0
# stddev 0.17 0.17
# median 0 0

www.percona.com 41

Tools for Replication
● Verify replication integrity (pt-table-checksum)
● Repair out-of-sync replicas (pt-table-sync)
● Measure delay accurately (pt-heartbeat)
● Deliberately delay replication (pt-slave-delay)
● Watch & restart replication (pt-slave-restart)

www.percona.com 42

Verify replication Integrity
Name: pt-table-checksum
Function: perform an online replication
consistency check, or checksum MySQL
tables efficiently on one or many servers.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-table-checksum.html
● Use it routinely (mandatory for 95% of MySQL users)
● It used to be complicated; no more. Just run it!
● Put it in a weekly crontab
● Repair differences with pt-table-sync
www.percona.com 43

I feel bad...

www.percona.com 44

Find problems
● What do you advise me doctor ?
● Get the symptoms when it hurts !

www.percona.com 45

Get the symptoms when it hurts !
To perform this action, do a check-up when
you feel bad:

● pt-stalk
● pt-sift

www.percona.com 46

Does it hurt ?
Name: pt-stalk
Function: wait for a condition to occur then
begin collecting data.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-stalk.html

www.percona.com 47

We need your health parameters
Name: pt-collect
Function: collect information from a server
for some period of time.
OBSOLETE
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

www.percona.com 48

Let me check...
Name: pt-sift
Function: browses files created by pt-stalk.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html

www.percona.com 49

Very useful
Name: pt-mext
Function: Look at many samples of MySQL
SHOW GLOBAL STATUS side-by-side
Url: http://www.percona.com/doc/percona-toolkit/pt-mext.html

● Default STATUS shows counter since the MySQL
instances started.
● It is more helpful to see a delta of recent activity, to check if
a change you applied made a difference.

www.percona.com 50

pt-mext
pt-mext -r -- mysqladmin ext -i10 -c5
All values are interesting but you should pay even more attention to:
● Created_tmp_*
● Handler_read_*
● Innodb_buffer_pool_read*
● Innodb_os_log_written
● Key_read_requests
● Key_reads
● Qchache*
● Select_full_*join
● Select_range
● Select_scan
● Table_locks_waited
● Threads_running

www.percona.com 51

Stop Bad Queries
● Use pt-kill to watch for long-running queries
and optionally kill them
● Prevent long-running idle transactions too.

www.percona.com 52

Percona Toolkit's Future
● Rapid development & improvement of tools
● Fully supported via Percona Support contract
● Upcoming:
➢ pt-*summary redesigned

➢ pt-stalk “magical fault detection algorithm”

➢ Redesign of pt-online-schema-change (see

http://www.percona.com/docs/wiki/ptdev:blueprints:redesign-pt-
online-schema-change)
➢ more goodies
•https://blueprints.launchpad.net/percona-toolkit

www.percona.com 53

Report bugs & patches

www.percona.com 54

Report bugs & patches

Belgian Beers ....... what else !?
www.percona.com 55

Thank you for your participation to
LOADAYS

Due to time I was not able to present
two new Percona projects related to
High Availability.

If you want to discuss PRM and
XtraDB Cluster... I am available

PRM
● Percona Replication Manager
● Uses Corosync / Pacemaker
● Distributed architecture, no SPOF
● One master multiple slaves
● Slaves maintain a master score attribute
based on replication state
● Full stonith/fencing stack
● Flexible logic, score based

www.percona.com 58

Percona XtraDB Cluster
● Percona Server + Galera
● Synchronous replication
● Multi-master replication
● Multi-threaded slaves
● Data consistency across all peers
● Automatic nodes provisioning

www.percona.com 59

lefred@percona.com
@lefred

We're Hiring! www.percona.com/about-us/careers/

Loadays managing my sql with percona toolkit

Recommended

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Loadays managing my sql with percona toolkit (20)

More from Frederic Descamps (20)

Recently uploaded (20)

Loadays managing my sql with percona toolkit