New features in Performance Schema 5.7 in action

New features
in Performance Schema 5.7 in action
October, 3, 2016 — M¨ovenpick Hotel — Amsterdam
Sveta Smirnova, Alexander Rubin

•Performance Schema Conﬁguration for our
Tutorial
•Locks diagnostic: MDL, table, index
•Memory usage
•Stored routines instrumentation
•Prepared Statements
•Replication: new tables, slave diagnostic, GTID
•Variables
•Errors Summary
Table of Contents
2

Performance Schema Conﬁguration for our Tutorial
3

5.7
• 87 tables
• 1019 instruments
• 42 variables
8.0
• 93 tables
• 1053 instruments
• 43 variables
Performance Schema in versions 5.7 and 8.0
4

• ON by default
• Only global, thread, statements and
transactions instrumentation enabled
• All other consumers are disabled
Performance Schema defaults
5

• Memory allocated on demand
• You don’t need to limit size of tables anymore
• Sys schema included into standard MySQL
distribution
• You can turn statistics on or off for
particular host and/or user
• Size of SQL DIGEST is tunable
Conﬁguraiton improvements in 5.7
6

• We will turn required instrumentation ON
for each example separately
• We will use pattern
update performance_schema.setup_consumers set enabled=’yes’
where name like ’OUR_REQUIREMENT_%’;
update performance_schema.setup_instruments set enabled=’yes’, timed=’yes’
where name like ’OUR_REQUIREMENT_%’;
• Be careful!
• They are memory and CPU intensive
• Do not turn them all ON until needed
Prepare
7

• We will turn required instrumentation ON
for each example separately
• Or easier
call sys.ps_setup_enable_consumer(YOUR_CONSUMER);
call sys.ps_setup_enable_instrument(YOUR_INSTRUMENT);
• Be careful!
• They are memory and CPU intensive
• Do not turn them all ON until needed
Prepare
7

• Table METADATA LOCKS
• Which thread is waiting for a lock
• Which thread holds the lock
• Not only for talbes:
GLOBAL, SCHEMA, TABLE, FUNCTION, PROCEDURE, EVENT, COMMIT, USER LEVEL LOCK,
TABLESPACE
MDL
9

mysql> select processlist_id, object_type, lock_type, lock_status, source
-> from metadata_locks join threads on (owner_thread_id=thread_id)
-> where object_schema=’employees’ and object_name=’titles’G
*************************** 1. row ***************************
processlist_id: 4
object_type: TABLE
lock_type: EXCLUSIVE
lock_status: PENDING -- waits
source: mdl.cc:3263
*************************** 2. row ***************************
processlist_id: 5
object_type: TABLE
lock_type: SHARED_READ
lock_status: GRANTED -- holds
source: sql_parse.cc:5707
METADATA LOCKS: example
10

• Login into EC2 instance
• Login: see your card
• Password: see your card
• Run load
./test1.sh
CALL help_task()G
CALL help_solve()G
CALL task_prepare();
• We need to ﬁnd out what prevents ALTER
from ﬁnishing
MDL: practice
11

• Table TABLE HANDLES
• Not only locks, but also information about
open tables
• FLUSH TABLES removes data from this table
Table locks
12

mysql1> select count(*) from employees where first_name like ’Svet%’;
• While running, check what is going on in
parallel connection:
mysql2> select * from table_handlesG
*************************** 1. row ***************************
OBJECT_TYPE: TABLE
OBJECT_SCHEMA: employees
OBJECT_NAME: employees
OBJECT_INSTANCE_BEGIN: 140544885988272
OWNER_THREAD_ID: 23
OWNER_EVENT_ID: 818320
INTERNAL_LOCK: NULL
EXTERNAL_LOCK: READ EXTERNAL -- Table lock!
1 row in set (0.00 sec)
Table locks: example
13

mysql1> select count(*), sleep(10) from employees where emp_no=10001;
• In parallel connection:
mysql2> select * from table_handlesG
*************************** 1. row ***************************
OBJECT_TYPE: TABLE
OBJECT_SCHEMA: employees
OBJECT_NAME: employees
OBJECT_INSTANCE_BEGIN: 140544885988272
OWNER_THREAD_ID: 23
OWNER_EVENT_ID: 1011419
INTERNAL_LOCK: NULL
EXTERNAL_LOCK: NULL -- Now everything is good: index access
Table locks: example
14

• Run load
./tables.sh
CALL help_task()G
CALL help_solve()G
• We need to ﬁnd out why so many threads
are waiting for a lock and ﬁx the issue
• We can also examine table cache content
Table Handles: practice
15

• You could not diagnose where was memory
gone before version 5.7
• Buffers?
• Temporary tables?
• Internal structures which are out of user
control?
• There is no leak, simply OS did not show
memory as freed yet?
Why these are our favorite improvements?
17

• free
• top
• vmstat
• Investigation
• There was no way to know how exactly
memory was allocated
Diagnostic tools before 5.7
18

• free
$free
total used free shared buffers cached
Mem: 16149184 6223916 9925268 317536 1048 3655160
-/+ buffers/cache: 2567708 13581476
Swap: 2110460 0 2110460
• top
• vmstat
• Investigation
18

• free
• top
$top
Tasks: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.0 us, 0.8 sy, 0.1 ni, 95.4 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 16149184 total, 6231688 used, 9917496 free, 1048 buffers
KiB Swap: 2110460 total, 0 used, 2110460 free. 3670752 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1914 mysql 20 0 670m 95m 1296 S 0.7 1.2 2:42.14 mysqld
• vmstat
• Investigation
18

• free
• top
• vmstat
$vmstat -t 5 3
procs ----------------------memory---------------------- ------swap---- --------
r b swpd free buff cache si so bi bo in cs us sy id wa...
2 0 0 9923160 1048 3662724 0 0 168 86 167 674 3 1 87...
0 0 0 9923252 1048 3662904 0 0 30 122 1168 5264 3 1 96...
0 0 0 9922864 1048 3663120 0 0 25 128 1191 5342 2 1 96...
• Investigation
18

• free
• top
• vmstat
• Investigation
• Total size of buffers
• Number of temporary tables
• Number of parallel connections
18

mysql> select thread_id tid, user, current_allocated ca, total_allocated
-> from sys.memory_by_thread_by_current_bytes;
+-----+-------------------------+-------------+-----------------+
| tid | user | ca | total_allocated |
+-----+-------------------------+-------------+-----------------+
| 1 | sql/main | 2.53 GiB | 2.69 GiB |
| 150 | root@127.0.0.1 | 4.06 MiB | 32.17 MiB |
| 146 | sql/slave_sql | 1.31 MiB | 1.44 MiB |
| 145 | sql/slave_io | 1.08 MiB | 2.79 MiB |
...
| 60 | innodb/io_read_thread | 0 bytes | 384 bytes |
| 139 | innodb/srv_purge_thread | -328 bytes | 754.21 KiB |
| 69 | innodb/io_write_thread | -1008 bytes | 34.28 KiB |
| 4 | innodb/io_log_thread | -2880 bytes | 132.38 KiB |
Memory diagnostic in 5.7
19

mysql> select * from sys.memory_by_thread_by_current_bytes
-> order by current_allocated descG
*************************** 1. row ***************************
thread_id: 152
user: lj@127.0.0.1
current_count_used: 325
current_allocated: 36.00 GiB
current_avg_alloc: 113.43 MiB
current_max_alloc: 36.00 GiB
total_allocated: 37.95 GiB
...
• Finding connections, using too much
memory, now is matter of seconds!
Threads Statistics
20

• memory summary by account by event name
• memory summary by host by event name
• memory summary by thread by event name
• memory summary by user by event name
• memory summary global by event name
• You must enable memory instrumentation!
• sys schema includes user name
RAW Performance Schema tables
21

• NAME@HOST - regular user
• System users
• sql/main
• innodb/*
• ...
• Data comes from table THREADS
Users in sys.memory * tables
22

• Run load
./test2.sh
CALL help_task()G
CALL help_solve()G
• We need to ﬁnd out how much memory
uses SysBench load, running in parallel
• To identify how much RAM used by whole
server run
select * from sys.memory_global_total;
Memory usage: practice
23

Stored routines instrumentation
24

mysql> select * from setup_instruments where name like ’statement/sp%’;
+--------------------------------+---------+-------+
| NAME | ENABLED | TIMED |
+--------------------------------+---------+-------+ ...
| statement/sp/stmt | YES | YES | | statement/sp/hreturn |
| statement/sp/set | YES | YES | | statement/sp/cpush |
| statement/sp/set_trigger_field | YES | YES | | statement/sp/cpop |
| statement/sp/jump | YES | YES | | statement/sp/copen |
| statement/sp/jump_if_not | YES | YES | | statement/sp/cclose |
| statement/sp/freturn | YES | YES | | statement/sp/cfetch |
| statement/sp/hpush_jump | YES | YES | | statement/sp/error |
| statement/sp/hpop | YES | YES | | statement/sp/set_case_expr |
... +----------------------------+
16 rows in set (0.00 sec)
New instruments
25

• What happens inside the routine
• Queries, called from the routine
• statement/sp/stmt
Stored routines instrumentation
26

• We will use this procedure
CREATE DEFINER=‘root‘@‘localhost‘ PROCEDURE ‘sp_test‘(val int)
BEGIN
DECLARE CONTINUE HANDLER FOR 1364, 1048, 1366
BEGIN
INSERT IGNORE INTO t1 VALUES(’Some string’);
GET STACKED DIAGNOSTICS CONDITION 1 @stacked_state = RETURNED_SQLSTATE;
GET STACKED DIAGNOSTICS CONDITION 1 @stacked_msg = MESSAGE_TEXT;
END;
INSERT INTO t1 VALUES(val);
END
• When HANDLER called?
Stored routines: example
27

• Run load
./crazy_timing.sh
CALL help_task()G
CALL help_solve()G
• We need to ﬁnd out why procedure takes
different time each run
• For better output set pager to less:
mysql> P less
Stored routines: practice
30

Prepared statements diagnostics
31

• Contains current prepared statements
• Statistics by
• Which thread owns the statement
• How many times executed
• Optimizer statistics, similar to
events statements *
Table prepared statements instances
32

mysql1> prepare stmt from ’select count(*) from employees where hire_date > ?’;
Query OK, 0 rows affected (0.00 sec)
Statement prepared
mysql1> set @hd=’1995-01-01’;
mysql1> execute stmt using @hd;
+----------+
| count(*) |
+----------+
| 34004 |
+----------+
• Try EXECUTE with different variable values
Example: prepared statement
33

mysql2> select statement_name, sql_text, owner_thread_id, count_reprepare,
-> count_execute, sum_timer_execute from prepared_statements_instancesG
*************************** 1. row ***************************
statement_name: stmt
sql_text: select count(*) from employees where hire_date > ?
owner_thread_id: 22
count_reprepare: 0
count_execute: 3
sum_timer_execute: 4156561368000
mysql1> drop prepare stmt;
mysql2> select * from prepared_statements_instancesG
Empty set (0.00 sec)
Example: diagnosis
34

• Run load
./prepared.sh
CALL help_task()G
CALL help_solve()G
• We need to ﬁnd out how effective is
prepared statement
Prepared statements: practice
35

• Data from SHOW SLAVE STATUS available
in replication * tables
• Support of Replication Channels
(Multi-master slave)
• More instruments for GTID
Major improvements
37

• No need to parse SHOW output
• Conﬁguration
• IO thread
• SQL thread
SLAVE STATUS
38

• Configuration
• replication connection configuration
• replication applier configuration
• IO thread
• SQL thread
SLAVE STATUS
38

• Conﬁguration
• IO thread
• replication connection status
• SQL thread
SLAVE STATUS
38

• Conﬁguration
• IO thread
• SQL thread
• replication applier status
• replication applier status by coordinator - MTS only
• replication applier status by worker
SLAVE STATUS
38

• Conﬁguation
mysql> select * from replication_connection_configuration
-> join replication_applier_configuration using(channel_name)G
*************************** 1. row ***************************
CHANNEL_NAME:
HOST: 127.0.0.1
PORT: 13000
USER: root
NETWORK_INTERFACE:
AUTO_POSITION: 1
SSL_ALLOWED: NO
SSL_CA_FILE:
...
CHANNEL_NAME:
DESIRED_DELAY: 0
SLAVE STATUS
39

• State of IO Thread
mysql> select * from replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME:
GROUP_NAME:
SOURCE_UUID: d0753e78-14ec-11e5-b3fb-28b2bd7442fd
THREAD_ID: 21
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 17
LAST_HEARTBEAT_TIMESTAMP: 2015-06-17 15:49:08
RECEIVED_TRANSACTION_SET:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
SLAVE STATUS
40

• Coordinator thread for multiple workers
mysql> select * from replication_applier_status join
-> replication_applier_status_by_coordinator using(channel_name)G
*************************** 1. row ***************************
CHANNEL_NAME:
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
THREAD_ID: 22
SERVICE_STATE: ON
LAST_ERROR_MESSAGE:
• Other cases
Performance Schema: State of SQL Thread
41

• Other cases
mysql> select * from replication_applier_status join
-> replication_applier_status_by_worker using(channel_name)G
*************************** 1. row ***************************
CHANNEL_NAME: master-1
SERVICE_STATE: OFF
WORKER_ID: 0
THREAD_ID: NULL
SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: ANONYMOUS
LAST_ERROR_MESSAGE: Could not execute Update_rows...
41

• Other cases
*************************** 2. row ***************************
CHANNEL_NAME: master-2
SERVICE_STATE: ON
WORKER_ID: 0
THREAD_ID: 42
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: ANONYMOUS
LAST_ERROR_MESSAGE:
2 rows in set (0,00 sec)
41

• RECEIVED TRANSACTION SET
in table replication connection status
• LAST SEEN TRANSACTION
in replication applier status by worker
GTID diagnostics
42

• Single-threaded slave
mysql> select cs.CHANNEL_NAME, cs.SOURCE_UUID, cs.RECEIVED_TRANSACTION_SET,
-> asw.LAST_SEEN_TRANSACTION, aps.SERVICE_STATE from
-> replication_connection_status cs join replication_applier_status_by_worke
-> asw using(channel_name) join replication_applier_status aps
-> using(channel_name) G
*************************** 1. row ***************************
CHANNEL_NAME:
SOURCE_UUID: 9038967d-7164-11e6-8c88-30b5c2208a0f
RECEIVED_TRANSACTION_SET: 9038967d-7164-11e6-8c88-30b5c2208a0f:1-2
LAST_SEEN_TRANSACTION: 9038967d-7164-11e6-8c88-30b5c2208a0f:2
SERVICE_STATE: ON
1 row in set (0,00 sec)
• Multi-threaded
GTID: all in one place
43

• Single-threaded slave
• Multi-threaded
*************************** 1. row ***************************
THREAD_ID: 30
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION:
...
*************************** 8. row ***************************
THREAD_ID: 37
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: 9038967d-7164-11e6-8c88-30b5c2208a0f:3
GTID: all in one place
43

• Tables in mysql schema
• slave master info
• slave relay log info
• slave worker info
• Join with Performance Schema tables
• New instruments
• memory
• wait
• stage
More diagnostic
44

• Run load
./repl.sh
CALL help_task()G
CALL help_solve()G
• We need to ﬁnd out why replication is
broken and ﬁx it
Replication: practice
45

• Variables
• Status variables
• show compatibility 56 = 0
Variables instrumentation
47

• Variables
• global variables
• session variables
• user variables by thread
• variables by thread
47

• Variables
• global status
• session status
• status by [account|host|thread|user]
47

• Same information which is in
• SHOW [GLOBAL] STATUS
• I S.GLOBAL VARIABLES (deprecated in 5.7)
• I S.SESSION VARIABLES (deprecated in
5.7)
• Helps to watch session variables changes
Global and session variables
48

• Same information which is in
• SHOW [GLOBAL] STATUS
• I S.GLOBAL STATUS (deprecated in 5.7)
• I S.SESSION STATUS (deprecated in 5.7)
Status variables
49

mysql> SELECT ss.variable_name, ss.variable_value FROM session_status ss
-> LEFT JOIN global_status gs USING(variable_name)
-> WHERE ss.variable_value != gs.variable_value OR gs.variable_value IS NULL
-> AND ss.variable_value>0;
+----------------------------+----------------+
| variable_name | variable_value |
+----------------------------+----------------+
| Bytes_sent | 197774 |
| Handler_commit | 0 |
| Handler_external_lock | 44 |
| Handler_read_first | 3 |
| Handler_read_key | 523 |
| Handler_read_next | 0 |
| Handler_read_rnd_next | 7241 |
| Opened_table_definitions | 0 |
...
Status variables
50

• status by
• account
• host
• thread
• user
Possible to group
51

• status by
mysql> select * from status_by_thread where variable_name=’Handler_write’;
+-----------+---------------+----------------+
+-----------+---------------+----------------+
| 71 | Handler_write | 94 |
| 83 | Handler_write | 477 | -- Most writes
| 84 | Handler_write | 101 |
+-----------+---------------+----------------+
Possible to group
51

• Grouped by connection
• Sometimes can help to ﬁnd tricky bugs with
persistent connections
mysql> select * from user_variables_by_thread;
+-----------+---------------+----------------+
+-----------+---------------+----------------+
| 71 | baz | boo |
| 84 | foo | bar |
+-----------+---------------+----------------+
User variables
52

• VARIABLES INFO in 8.0
• Source of variable
COMPILED
EXPLICIT
COMMAND LINE
DYNAMIC
• Path of option ﬁle if speciﬁed
• Minimum and maximum values
Variables info
53

mysql> select * from variables_info G
*************************** 1. row ***************************
VARIABLE_NAME: auto_increment_increment
VARIABLE_SOURCE: COMPILED
VARIABLE_PATH:
MIN_VALUE: 1
MAX_VALUE: 65535
*************************** 2. row ***************************
VARIABLE_NAME: basedir
VARIABLE_SOURCE: EXPLICIT
VARIABLE_PATH: /home/sveta/build/mysql-8.0/mysql-test/var/my.cnf
MIN_VALUE: 0
MAX_VALUE: 0
...
Variables info
53

• Source of variable
COMPILED
EXPLICIT
COMMAND LINE
DYNAMIC
• Path of option ﬁle if speciﬁed
• Minimum and maximum values
• No variable values in this table!
Variables info
53

• Run load
./variables.sh
CALL help_task()G
CALL help_solve()G
• We need to watch progress of INSERT
command, running by stored routine.
• Note what there is parallel load, caused by
SysBench. We are not interested in its
statistics.
Variables: practice
54

• Traditionally aggregated
• events errors summary by account by error
• events errors summary by host by error
• events errors summary by thread by error
• events errors summary by user by error
• events errors summary global by error
• All tables have similar structure
Errors Summary Tables in 8.0
56

mysql> select * from events_errors_summary_by_account_by_error
-> where SUM_ERROR_RAISED > 100G
*************** 1. row ***************
USER: root
HOST: localhost
ERROR_NUMBER: 1213
ERROR_NAME: ER_LOCK_DEADLOCK
SQL_STATE: 40001
SUM_ERROR_RAISED: 221
SUM_ERROR_HANDLED: 0
FIRST_SEEN: 2016-09-28 01:45:09
LAST_SEEN: 2016-09-28 01:47:02
*************** 2. row ***************
USER: root
HOST: localhost
ERROR_NUMBER: 1287
ERROR_NAME: ER_WARN_DEPRECATED_SYNTAX
SQL_STATE: HY000
SUM_ERROR_RAISED: 279
SUM_ERROR_HANDLED: 0
FIRST_SEEN: 2016-09-27 23:59:49
LAST_SEEN: 2016-09-28 01:47:05
Errors Summary: which accounts raise more errors?
57

• Nickolay Ihalainen for practice setup idea
and examples
• Alexander Rubin for EC2 setup and
replication example
Special thanks
58

• Blog of developers team
• Blog of Mark Leith: author of sys schema
• Ofﬁcial reference manual
More informaiton
59

???
Place for your questions
61

http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
https://www.linkedin.com/in/alexanderrubin
Thank you!
62

New features in Performance Schema 5.7 in action

Recommended

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to New features in Performance Schema 5.7 in action (20)

More from Sveta Smirnova (20)

Recently uploaded (11)

New features in Performance Schema 5.7 in action