DPC Tutorial

Optimizing MySQL
Essentials

About Me

• Ligaya Turmelle
• Senior Technical Support Engineer
• MySQL Now (AKA Oracle)
• ~3 years

Agenda
• Basics
• Hardware and OS
• Server
• variables
• Queries
• slow query log and EXPLAIN
• example query

• Deﬁnition of Performance:

Characterized by the amount of useful
work accomplished by a computer
system compared to the time and
resources used.

This can cover a lot of different areas
(potentially at the same time) in a LAMP
stack

• Application

• Network

• Hardware

• Database

Learn how to benchmark

• There are lots of benchmarking tools
out there.
• Use what is appropriate to measure
what you are trying to improve

OS

• Linux vs. Windows
• Linux vs. Linux

Linux

• swappiness = 0
• IO scheduler
• HDD = deadline
• SSD or you have a good RAID
controller = noop
• You can use ext3 - but not the best

Hardware

• RAM
• RAM
• RAM
• Disks
• CPU
• And did I mention RAM

Basic Linux commands

• top
• iostat
• vmstat
• dstat

Server Settings
So what should i tweak?

Knobs, Levers and
Buttons

• mysqld
• MyISAM
• InnoDB
• Others

MySQLD
Many think that yum install / apt-get install

and they’re done.
• No, we’re just beginning!
• Logging
• Memory Usage:
• Global
• Session
• Character Sets

MySQLD
( logging )

• log_bin = <filename>
• general_log = 0
general_log_file = <filename>
• slow_query_log = 1
slow_log_file = <filename>
long_query_time = 1
• log_queries_not_using_indexes
min_examined_row_limit = 100

MySQLD
( memory usage )

• Memory is allocated...
• Globally - server wide, usually allocated
only at server startup and exist only “once”
• Session - each connection or event
• Both - In these cases the global variable
value ( join_buffer_size ) is used if a
new value is not set during the session.

MySQLD
( Memory / global )

• connection_timeout ( 10 )
• expire_log_days ( 7 )
• query_cache_type = 0 ( yes, 0! )
• wait_timeout = 3
• skip_name_resolve = 1
• performance_schema = 1

MySQLD
( memory / global )

• max_heap_table_size /
tmp_table_size
• table_open_cache
• thead_cache_size

}
• read_buffer_size /
read_rnd_buffer_size Also
• sort_buffer_size conﬁgurable
per session
• join_buffer_size

MyISAM
• Switch to using InnoDB

MyISAM
• Not an option?

MyISAM
• Not an option?
• Still not an option?

MyISAM
• Not an option?
• The BLACKHOLE storage engine is really
fast

MyISAM
• Not an option?
• The BLACKHOLE storage engine is really
fast
• Still stuck?

MyISAM
• switch to mongodb, it’s webscale!

http://nosql.mypopescu.com/post/1016320617/mongodb-is-web-scale

MyISAM

• concurrent_insert
• 0 - Never, 1 - Hole free, 2 - Holes
• key_buffer_size
• Key_reads / Key_read_requests
• Key_writes / Key_write_requests

InnoDB
Many options only a few covered
• innodb_buffer_pool_size
- caches both indexes and data
- upwards of 85% available RAM
• innodb_flush_log_at_trx_commit
- 0 - written and flushed to disk once per second
- 1 - written and flushed to disk at each
transaction commit
- 2 - written to disk at each commit, flush one a
second

InnoDB
• innodb_log_buffer_size
- Amount of data written to ib_logfile
- The larger, the less disk I/O
• innodb_log_file_size
- The larger the ﬁle, the more InnoDB can do
- < 5.5 the larger, the longer crash recovery time
• innodb_double_write_buffer
- data integrity checking, costs IOPSs
• innodb_file_per_table
- data and indexes written to their own .ibd ﬁle

Examine, Test
and Measure
“Dr. Watson I Presume”

MySQLD
( quick check )

Quick Check - mysql-tuning-primer
https://launchpad.net/mysql-tuning-primer

MySQLD
( in depth 1/11 )

Getting In depth - SHOW GLOBAL STATUS
+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+ | Com_replace | 37372 |
| Com_alter_table | 184 | | Com_replace_select | 0 |
| Com_begin | 103281460 | | Com_rollback | 148312594 |
| Com_change_db | 631 | | Com_select | 7164448573 |
| Com_commit | 669825867 | | Com_set_option | 1877614377 |
| Com_create_index | 0 | | Com_show_collations | 130652029 |
| Com_create_table | 354 | | Com_show_fields | 694 |
| Com_delete | 1235248406 | | Com_show_grants | 36 |
| Com_delete_multi | 353434291 | | Com_show_processlist | 33184 |
| Com_drop_table | 359 | | Com_show_slave_hosts | 1 |
| Com_flush | 3 | | Com_show_variables | 130885618 |
| Com_insert | 2725147239 | | Com_show_warnings | 99619 |
| Com_insert_select | 9 | | Com_truncate | 11 |
| Com_kill | 3642 | | Com_unlock_tables | 18 |
| Com_load | 15 | | Com_update | 1348570903 |
| Com_lock_tables | 18 | | Com_update_multi | 8 |
+--------------------------+------------+

MySQLD
( in depth 2/11 )

Values with out timeframe are
meaningless.
10:21:08 rderoo@mysql09:mysql [1169]> SHOW GLOBAL STATUS LIKE 'Uptime';
+---------------+----------+
+---------------+----------+
| Uptime | 12973903 |
+---------------+----------+
1 row in set (0.00 sec)

That’s about 150 days. :)

MySQLD
( in depth 3/11 )

By using mysqladmin we can observe
changes over short periods of time:
$ mysqladmin -u rderoo -p -c 2 -i 10 -r extended-status > review.txt

Output is the same as SHOW GLOBAL STATUS
with two copies appearing in review.txt
the ﬁrst has values since the last server
restart / FLUSH COUNTERS the second output
are the delta between the ﬁrst and 10 seconds
later.

MySQLD
( in depth 4/11 )

Temporary Tables:
How big?
17:54:33 rderoo@mysql09:event [1185]> SHOW GLOBAL VARIABLES LIKE '%_table_size';
+---------------------+----------+
+---------------------+----------+
| max_heap_table_size | 67108864 |
| tmp_table_size | 67108864 |
+---------------------+----------+
2 rows in set (0.00 sec)

How many?
17:55:09 rderoo@mysql09:event [1186]> SHOW GLOBAL STATUS LIKE '%_tmp%tables';
+-------------------------+-----------+
+-------------------------+-----------+
| Created_tmp_disk_tables | 156 |
| Created_tmp_tables | 278190736 |
+-------------------------+-----------+

MySQLD
( in depth 5/11 )

The max temporary table size is 64mb.

In 150 days 278 million temp tables where
created.

With only 156 going to disk.

This is well tuned!

MySQLD
( in depth 7/11 )

thread_cache_size
• To determine the effectiveness of the thread_cache
as a hit ratio use the following formula:
100-((Threads_created/Connections)*100)
| Connections | 131877233 |
| Threads_created | 20014624 |

100 - ( ( 20014624 / 131877233 ) * 100 ) = 84.8%

• Increasing the thread_cache_size would seem to be
advisable.

MySQLD
( in depth 8/11 )

read_buffer_size
• The effectiveness can be found by examining the
Select_scan :

| Select_scan | 304864263 |

• This should be set to a small value globally and
increased on a per session basis

• Increase when performing full table scans

MySQLD
( in depth 9/11)

read_rnd_buffer_size
• Used when reading sorted results after a key sort has
taken place


• Usually should be set ( per session ) no high than 8M

MySQLD
( in depth 10/11)

sort_buffer_size
Sort_merge_passes :

| Sort_merge_passes | 7 |


• Increase GROUP BY, ORDER BY, SELECT
DISTINCT and UNION DISTINCT performance

MySQLD
( in depth 11/11 )

join_buffer_size
Select_full_join :

| Select_full_join | 5369 |

• A crutch used when proper indexing is not in place
• Will help in cases of full table scans

• Increase GROUP BY and ORDER BY performance

Tuning Queries
Where the magic happens

Tuning your queries is the single biggest
performance improvement you can do to
your server

• efﬁcient resource usage

• results returned quicker

Per QUERY!

Slow query log

• A log of all the “slow” queries
• Deﬁne “slow”
• took longer then long_query_time
to run AND at least
min_examined_row_limit rows
are looked at
• log_queries_not_using_indexes
• log_slow_admin_statements

Turning it on

• Option is slow_query_log
• no argument or value of 1 - enabled
• argument of 0 - disabled (default)

Output Types

• Use log_output
• write to a ﬁle
• write to a table in the mysql database

slow log at startup

• Place the slow_query_log and
optional log_output options in
my.cnf
• log_output acceptable values
• TABLE, FILE, NONE
• Can also use slow_query_log_file
to specify a log ﬁle name

slow log at runtime

• All the below global variables can be
changed at run time
• log_output
• slow_query_log
• slow_query_log_file
SET GLOBAL slow_query_log = 1;

/usr/sbin/mysqld, Version: 5.0.67-0ubuntu6.1-log ((Ubuntu)). started with:
Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock
Time Id Command Argument
# Time: 110328 13:15:05
# User@Host: homer[homer] @ localhost []
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 4452238
use confs;
select
distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from
user u,
talks t,
talk_speaker ts
where
u.ID <> 6198 and
u.ID = ts.speaker_id and
t.ID = ts.talk_id and
t.event_id in (
select
distinct t.event_id
from
talk_speaker ts,
talks t
where
ts.speaker_id = 6198 and
t.ID = ts.talk_id
)
order by rand()
limit 15;

There are multiple ways of getting
information out of the slow query log.

• just open it up to see the raw data

• usually more helpful to summarize/
aggregate the raw data

Tools for aggregating
the slow log data

• mk-query-digest
• third party
• part of the maatkit utilities
• mysqldumpslow
• comes with MySQL in the bin dir

Macintosh-7:joindin-for-lig ligaya$ perl -laF ~/mysql_installs/maatkit/mk-
query-digest mysql-slow.log

# 23s user time, 240ms system time, 17.53M rss, 86.51M vsz

# Current date: Mon May 16 11:08:24 2011

# Hostname: Macintosh-7.local

# Files: mysql-slow.log

# Overall: 73.72k total, 14 unique, 0.02 QPS, 0.04x concurrency __________
# Time range: 2011-03-28 13:15:05 to 2011-05-13 02:33:52
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 160130s 0 180s 2s 3s 819ms 2s
# Lock time 11s 0 1s 149us 0 12ms 0
# Rows sent 387.64k 0 255 5.38 14.52 6.86 0
# Rows examine 338.66G 0 5.35M 4.70M 4.93M 387.82k 4.70M
# Query size 30.19M 17 8.35k 429.47 420.77 29.33 420.77

# Profile
# Rank Query ID Response time Calls R/Call Apdx V/M Item
# ==== ================== ================= ===== ======= ==== ===== =====
# 1 0x2AAA5EC420D7E2A2 159878.0000 99.8% 73682 2.1698 0.50 0.12 SELECT
user talks talk_speaker talks
# 3 0x0A2584C03C8614A9 50.0000 0.0% 16 3.1250 0.44 0.87 SELECT
wp_comments
# MISC 0xMISC 202.0000 0.1% 20 10.1000 NS 0.0 <12
ITEMS>

Count: 1 Time=180.00s (180s) Lock=0.00s (0s) Rows=1.0 (1), XXXXX[xxxxx]@localhost
SELECT SLEEP(N)

SELECT comment_date_gmt FROM wp_comments WHERE comment_author_IP = 'S' OR
comment_author_email = 'S' ORDER BY comment_date DESC LIMIT N

SELECT comment_ID FROM wp_comments WHERE comment_post_ID = 'S' AND ( comment_author
= 'S' OR comment_author_email = 'S' ) AND comment_content = 'S' LIMIT N

Count: 73682 Time=2.17s (159867s) Lock=0.00s (11s) Rows=5.4 (396223),
XXXXX[xxxxx]@localhost
select
distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from
user u,
talks t,
talk_speaker ts
where
u.ID <> N and
t.event_id in (
select
distinct t.event_id
from
talk_speaker ts,
talks t
where
ts.speaker_id = N and
t.ID = ts.talk_id
)
order by rand()
limit N

Tuning a query

• Tables
• SHOW CREATE TABLE `talks`;
• SHOW TABLE STATUS LIKE
‘talks’;
• Indexes
• SHOW INDEXES FROM ‘talks’;
• EXPLAIN

EXPLAIN Basics

• Syntax: EXPLAIN [EXTENDED]
SELECT select_options
• Displays information from the
optimizer about the query execution
plan
• Works only with SELECT statements

EXPLAIN Output
• Each row provides information on one
table
• Output columns:
id key_length
select_type ref
table rows
type Filtered (new to 5.1)
possible_keys Extra
key

id

• Select identiﬁer
• Only if there are subqueries, derived
tables or unions is this incremented
• Number reﬂects the order that the
SELECT/FROM was done in

select_type

• Type of SELECT
• SIMPLE
• PRIMARY
• DERIVED
• UNION
• SUBQUERY

table

• The table (or alias) that is used
• Read down - these are the order the
tables are JOINed
• DERIVED tables are noted and
numbered

type
• Access type
• Various types:
preference access type
BEST system/const
eq_ref
ref
ref_or_null
index_merge
range
index
WORST ALL

possible_keys

• List of all possible indexes (or NULL)
that optimizer considered using

key

• Index used for the query or NULL
• Look at key and possible_keys to
consider your index strategy

key_len

• shows number of bytes MySQL will use
from the index
• Possible that only part of the index will
be used
NOTE: a character set may use more then
one byte per character

ref

• Very different then the access type ‘ref’
• Show which columns or a constant
within the index will be used for the
access

rows

• Number of rows MySQL expects to ﬁnd
based on statistics
• statistics can be updated with
ANALYZE TABLE

ﬁltered

• New in 5.1.12
• Estimated number of rows ﬁltered by
the table condition
• Shows up if you use EXPLAIN
EXTENDED

Extra

• All the other stuff
• Using index - using a covering index
• Using ﬁlesort - sorting in a temporary
table
• Using temporary - a temporary table
was made.
• Using where - ﬁltering outside the
storage engine.

Query we are working
with:
EXPLAIN select distinct u.ID as user_id,
t.event_id,
u.username,
u.full_name
from user u,
talks t,
talk_speaker ts
where u.ID <> 11945 and
t.event_id in (
select distinct t.event_id
from talk_speaker ts,
talks t
where ts.speaker_id = 11945 and
t.ID = ts.talk_id
)
order by rand()
limit 15G

Full EXPLAIN plan
*************************** 1. row ***************************
id: 1
select_type: PRIMARY
table: t
type: index
possible_keys: PRIMARY
key: idx_event
key_len: 9
ref: NULL
rows: 3119
Extra: Using where; Using index; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
table: ts
type: ref
possible_keys: talk_id
key: talk_id
key_len: 5
ref: joindin.t.ID
rows: 1
Extra: Using where
*************************** 3. row ***************************
id: 1
table: u
type: eq_ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: joindin.ts.speaker_id
rows: 1
Extra:

Full EXPLAIN plan
(con’t)
*************************** 4. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: t
type: ref
possible_keys: PRIMARY,idx_event
key: idx_event
key_len: 5
ref: func
rows: 24
Extra: Using where; Using index; Using temporary
*************************** 5. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: ts
type: ref
possible_keys: talk_id
key: talk_id
key_len: 5
ref: joindin.t.ID
rows: 1
Extra: Using where

What was wrong
with that query
plan?

Hint
EXPLAIN select distinct u.ID as user_id, *************************** 4. row ***********************
t.event_id, id: 2
u.username, select_type: DEPENDENT SUBQUERY
u.full_name table: t
from user u, type: ref
talks t, possible_keys: PRIMARY,idx_event
talk_speaker ts key: idx_event
where u.ID <> 11945 and key_len: 5
u.ID = ts.speaker_id and ref: func
t.ID = ts.talk_id and rows: 24
t.event_id in ( Extra: Using where; Using index; Using temporary
select distinct t.event_id *************************** 5. row **********************
from talk_speaker ts, id: 2
talks t select_type: DEPENDENT SUBQUERY
where ts.speaker_id = 11945 table: ts
and type: ref
t.ID = ts.talk_id possible_keys: talk_id
) key: talk_id
order by rand() key_len: 5
limit 15G ref: joindin.t.ID
rows: 1
Extra: Using where

Solution
1:
SELECT DISTINCT u.ID as user_id,
t.event_id,
u.username,
u.full_name
FROM user u
JOIN talk_speaker ts
ON u.ID = ts.speaker_id
JOIN talks t
ON t.ID = ts.talk_id
JOIN
-- find the events the speaker has talks in
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
ON t1.ID = ts1.talk_id
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

Solution
1: reminder:
CREATE TABLE `talk_speaker` (
SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL,
t.event_id, `speaker_name` varchar(200) DEFAULT NULL,
u.username, `ID` int(11) NOT NULL AUTO_INCREMENT,
u.full_name `speaker_id` int(11) DEFAULT NULL,
FROM user u `status` varchar(10) DEFAULT NULL,
JOIN talk_speaker ts PRIMARY KEY (`ID`),
ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE
JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763
ON t.ID = ts.talk_id DEFAULT CHARSET=utf8;
JOIN
(
SELECT t1.event_id
FROM talk_speaker ts1
JOIN talks t1
WHERE ts1.speaker_id = 11945
) as e
ON t.event_id = e.event_id
WHERE u.ID <> 11945
LIMIT 15G

Solution
1: reminder:
CREATE TABLE `talk_speaker` (
SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL,
t.event_id, `speaker_name` varchar(200) DEFAULT NULL,
u.username, `ID` int(11) NOT NULL AUTO_INCREMENT,
u.full_name `speaker_id` int(11) DEFAULT NULL,
FROM user u `status` varchar(10) DEFAULT NULL,
JOIN talk_speaker ts PRIMARY KEY (`ID`),
ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE
JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763
ON t.ID = ts.talk_id DEFAULT CHARSET=utf8;
JOIN
(
SELECT t1.event_id 2:
FROM talk_speaker ts1 ALTER TABLE `talk_speaker`
JOIN talks t1 ADD INDEX (speaker_id);
WHERE ts1.speaker_id = 11945 OR
) as e ALTER TABLE `talk_speaker`
ON t.event_id = e.event_id ADD INDEX (speaker_id, talk_id);
WHERE u.ID <> 11945
LIMIT 15G

Thank you
http://joind.in/3215

DPC Tutorial

Recommended

More Related Content

What's hot (20)

Similar to DPC Tutorial (20)

More from Ligaya Turmelle (10)

Recently uploaded (20)

DPC Tutorial

Editor's Notes