Troubleshooting MySQL Performance
1 like875 views
This presentation discusses troubleshooting MySQL performance issues. It begins by explaining how to determine if MySQL is the source of slow performance by measuring query response times. The presentation then covers various aspects to check like queries, server options, hardware resources, and replication setup. Specific diagnostic tools are also introduced like EXPLAIN, slow query log, and various system monitoring commands. Finally, it recommends following a structured process of tuning queries, options, and hardware to isolate and resolve any performance problems.
![MySQL Sandbox
• Designed by Giuseppe Maxia to create MySQL
Sandbox of distribution you like with single command
• Available at http://mysqlsandbox.net/
• Collection of Perl scripts
• On UNIX, Linux, Mac OS X
–
–
–
–
perl Makefile.PL
make
[make test] - optionally
sudo make install
• Use Cygwin on Windows
• Need mysql-*.tar.gz or mysql-*.zip package](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-7-320.jpg)
![General Query Log
●
●
●
●
●
●
●
●
●
●
mysql> select * from mysql.general_log where
thread_id = 1312 order by event_time G
******************* 1. row *******************
event_time: 20091001 15:54:11
user_host: root[root] @ localhost []
thread_id: 1312
server_id: 51
command_type: Query
argument: begin
●
●
●
●
●
●
●
●
●
●
******************* 2. row *******************
event_time: 20091001 15:54:13
user_host: root[root] @ localhost []
thread_id: 1312
server_id: 51
command_type: Query
argument: insert into t1 values(2,'19941230',
'19941203')
2 rows in set (0.12 sec)](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-13-320.jpg)
![MySQL Server Variables or Options:
How to control
§ SHOW [GLOBAL] STATUS
§ GLOBAL
– Since server start
§ SESSION
– For operations in current session
– Can be reset
–
FLUSH STATUS](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-16-320.jpg)
![MySQL Server Variables or Options:
When affecting option is not known
§ Record currently used variables
– SHOW [GLOBAL] VARIABLES
§ Start mysqld with option –no-defaults
– This option must be first one!
§ Check if problem is solved
§ Change variable values one-by-one until you find one
which leads to the problem](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-17-320.jpg)
![Memory: diagnostic
• Free
–
–
–
–
–
[sveta@delly meb-trunk]$ free
total
used
free shared buffers cached
Mem:
7937924 7665236 272688
0 708056 5079376
-/+ buffers/cache: 1877804 6060120
Swap:
8546572
6200 8540372
• Top
–
–
–
–
Tasks: 246 total,
2 running, 244 sleeping,
0 stopped,
0 zombie
Cpu(s): 3.2%us, 1.2%sy, 0.0%ni, 95.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:
7937924k total, 7662172k used,
275752k free,
708148k buffers
Swap: 8546572k total,
6200k used, 8540372k free, 5075036k cached
–
–
PID USER
PR
NI
VIRT
RES
–
1914 mysql
20
0
670m
95m 1296 S
• vmstat
SHR S %CPU %MEM
0.7
1.2
TIME+
COMMAND
2:42.14 mysqld](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-19-320.jpg)
![Disk: diagnostic
• df
–
–
–
–
–
–
[sveta@delly mysql-5.1]$ df
Filesystem
1K-blocks
Used Available Use% Mounted on
/dev/sda3
126388036 90639344 29328520 76% /
tmpfs
3968960
672
3968288
1% /dev/shm
/dev/sda2
341869564 114894920 226974644 34% /mnt/windows
/dev/mmcblk0p1
31154688
309376 30845312
1% /media/MYDATA
• iostat
–
–
–
–
–
–
–
–
[sveta@delly mysql-5.1]$ iostat -x
Linux 2.6.32-131.0.15.el6.x86_64 (delly)
05/03/2012
_x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
5.29 0.00
1.47
2.45
0.00 90.79
Device:
rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
Scd0
2.91
0.00
0.09 0.00 11.99
0.00 129.97 0.00
46.46 24.55 0.23
sda
1.43
56.87
2.10 2.39 201.15 473.77 150.25 0.49
109.38 4.83 2.17
mmcblk0
0.03
0.00
0.00 0.00 0.03
0.00 31.08 0.00
3.14 2.69 0.00](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-20-320.jpg)
![CPU: diagnostic
• top
–
–
–
–
Tasks: 246 total,
2 running, 244 sleeping,
0 stopped,
0 zombie
Cpu(s): 3.2%us, 1.2%sy, 0.0%ni, 95.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:
7937924k total, 7662172k used,
275752k free,
708148k buffers
Swap: 8546572k total,
6200k used, 8540372k free, 5075036k cached
–
PID USER
PR
NI
VIRT
RES
SHR S %CPU %MEM
–
1914 mysql
20
0
670m
95m 1296 S
0.7
1.2
TIME+
COMMAND
2:42.14 mysqld
• iostat
• ps
– [sveta@delly src]$ ps -eo pid,user,comm,pcpu,pmem,vsz | grep mysqld
mysqld_safe
0.0 0.0 108192
– 1817 root
mysqld
0.0 1.1 686328
– 1914 mysql
mysqld
0.0 0.5 735744
– 14231 sveta](https://image.slidesharecdn.com/troubleshootingmysqlperformancefosdem-131215045908-phpapp01/85/Troubleshooting-MySQL-Performance-21-320.jpg)
Troubleshooting MySQL Performance
- 1. <Insert Picture Here> Troubleshooting MySQL Performance Sveta Smirnova Principal Technical MySQL Support Engineer
- 2. What will we discuss • Workflow –Missed details • http://www.sql-error.js-client.com/devconf2012 • http://www.microbecal.com/ftp_load/Troubleshooting_MySQL_Performance_addons_en.pdf • http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
- 3. What are we going to troubleshoot? • What does mean slowly? • It depends • Think about your application
- 4. How to find if MySQL is a weak link? • Pseudo-code – – – – Measure current time at moment1 Call MySQL function Measure current time at moment2 Difference between moment2 and moment1 is how much time MySQL spent working on your request. • Usually built-in into software you use • Only rough imagination needed
- 5. What to check • Query • Options • Hardware
- 6. Sandbox • Test! • But testing in production is dangerous! • Copy data – CREATE TABLE test_copy LIKE test; – INSERT INTO test_copy SELECT * FROM test; – Any backup-restore technique which can copy-restore single table • OR – CREATE DATABASE test_copy; – Copy all involved objects (tables, views and so on) into new database – Use your favorite backup-restore technique
- 7. MySQL Sandbox • Designed by Giuseppe Maxia to create MySQL Sandbox of distribution you like with single command • Available at http://mysqlsandbox.net/ • Collection of Perl scripts • On UNIX, Linux, Mac OS X – – – – perl Makefile.PL make [make test] - optionally sudo make install • Use Cygwin on Windows • Need mysql-*.tar.gz or mysql-*.zip package
- 8. How to find out if query runs slow? • At development stage • On production only –EXPLAIN –Slow query log
- 9. What to do next? • EXPLAIN • EXPLAIN EXTENDED • EXPLAIN PARTITIONS • SHOW STATUS LIKE 'Handler_%'
- 10. Suddenly slow query • Worked fast initially • Diagnostic utilities showed good result • After some time, usually long, started to work slow – Variance: works fast on development server and slow on production • Stable slowdown: it never works fast anymore • How to find: – Slow Query Log – SHOW PROCESSLIST • How to fix: – Same methods like for simple slow query – Data and index file maintaining
- 11. SHOW PROCESSLIST Your first friend when debugging Shows all queries, executing at the moment Accessible in any version You can find same data in the INFORMATION_SCHEMA.PROCESSLIST table, introduced in version 5.1 • Only tool for MyISAM • MDL locks are also visible in • • • • – performance_schema.event_waits
- 12. Parallel execution: InnoDB • SHOW ENGINE INNODB STATUS • INNODB_* tables in the INFORMATION SCHEMA –INNODB_TRX –INNODB_LOCKS –INNODB_LOCK_WAITS
- 14. MySQL Server Variables or Options: Scope § Global – Control parameters, necessary for all server processes Location of server files: datadir etc. – Shared buffers – More § Session – Control connection-specific parameters –
- 15. MySQL Server Variables or Options: When allocated § Those which control behavior of whole server – Once at server startup Can start with low values, then grow to specified § Connection options – For every connection when connection opens § Operation-specific – For every operation when needed –
- 16. MySQL Server Variables or Options: How to control § SHOW [GLOBAL] STATUS § GLOBAL – Since server start § SESSION – For operations in current session – Can be reset – FLUSH STATUS
- 17. MySQL Server Variables or Options: When affecting option is not known § Record currently used variables – SHOW [GLOBAL] VARIABLES § Start mysqld with option –no-defaults – This option must be first one! § Check if problem is solved § Change variable values one-by-one until you find one which leads to the problem
- 18. System resources • Memory • CPU • Disk • Network
- 19. Memory: diagnostic • Free – – – – – [sveta@delly meb-trunk]$ free total used free shared buffers cached Mem: 7937924 7665236 272688 0 708056 5079376 -/+ buffers/cache: 1877804 6060120 Swap: 8546572 6200 8540372 • Top – – – – Tasks: 246 total, 2 running, 244 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.2%sy, 0.0%ni, 95.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 7937924k total, 7662172k used, 275752k free, 708148k buffers Swap: 8546572k total, 6200k used, 8540372k free, 5075036k cached – – PID USER PR NI VIRT RES – 1914 mysql 20 0 670m 95m 1296 S • vmstat SHR S %CPU %MEM 0.7 1.2 TIME+ COMMAND 2:42.14 mysqld
- 20. Disk: diagnostic • df – – – – – – [sveta@delly mysql-5.1]$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 126388036 90639344 29328520 76% / tmpfs 3968960 672 3968288 1% /dev/shm /dev/sda2 341869564 114894920 226974644 34% /mnt/windows /dev/mmcblk0p1 31154688 309376 30845312 1% /media/MYDATA • iostat – – – – – – – – [sveta@delly mysql-5.1]$ iostat -x Linux 2.6.32-131.0.15.el6.x86_64 (delly) 05/03/2012 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.29 0.00 1.47 2.45 0.00 90.79 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util Scd0 2.91 0.00 0.09 0.00 11.99 0.00 129.97 0.00 46.46 24.55 0.23 sda 1.43 56.87 2.10 2.39 201.15 473.77 150.25 0.49 109.38 4.83 2.17 mmcblk0 0.03 0.00 0.00 0.00 0.03 0.00 31.08 0.00 3.14 2.69 0.00
- 21. CPU: diagnostic • top – – – – Tasks: 246 total, 2 running, 244 sleeping, 0 stopped, 0 zombie Cpu(s): 3.2%us, 1.2%sy, 0.0%ni, 95.1%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 7937924k total, 7662172k used, 275752k free, 708148k buffers Swap: 8546572k total, 6200k used, 8540372k free, 5075036k cached – PID USER PR NI VIRT RES SHR S %CPU %MEM – 1914 mysql 20 0 670m 95m 1296 S 0.7 1.2 TIME+ COMMAND 2:42.14 mysqld • iostat • ps – [sveta@delly src]$ ps -eo pid,user,comm,pcpu,pmem,vsz | grep mysqld mysqld_safe 0.0 0.0 108192 – 1817 root mysqld 0.0 1.1 686328 – 1914 mysql mysqld 0.0 0.5 735744 – 14231 sveta
- 22. Network • Important: – Stability – Bandwidth – Speed • RTT • --log-warnings=2 • Send huge file (1G or larger) • tcpdump
- 23. Replication slowdowns • Slow network – Any general diagnostic tool – Connect with mysql command line client • REPLICATION SLAVE • Binary log slowdowns master – InnoDB locks • Slave is slower than master – Multi threaded slave – Tune buffers – SHOW SLAVE STATUS • Seconds_behind_master
- 25. Summary • Tune queries – Check if query is always slow – Ensure query is not affected by concurrency issue • • • • Check if your options are sane Check your hardware and operating system Repeat until you are happy Don't overtune!
- 26. THANK YOU!
- 27. ?
- 28. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.