Using MySQL in Automated Testing
7 likes2,754 views
This document discusses using MySQL in automated testing. It covers various tools that can be used to automate and manage database deployments as part of testing, including pt-online-schema-change, MySQL Sandbox, SYS, Outbrain Propagator, Liquibase, ORM migrations, and libeatmydata. It also discusses considerations for different MySQL versions, such as online DDL support being introduced in MySQL 5.6. The document aims to demonstrate that databases can and should be automated and treated as first-class citizens in testing environments.
![Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL::Sandbox
25
$ make_sandbox mysql-5.6.23-osx10.9-x86_64.tar.gz
unpacking /Users/morgo/Downloads/mysql-5.6.23-osx10.9-x86_64.tar.gz
Executing low_level_make_sandbox --basedir=/Users/morgo/Downloads/
5.6.23
--sandbox_directory=msb_5_6_23
--install_version=5.6
--sandbox_port=5623
--no_ver_after_name
--my_clause=log-error=msandbox.err
The MySQL Sandbox, version 3.0.47
(C) 2006-2013 Giuseppe Maxia
installing with the following parameters:
upper_directory = /Users/morgo/sandboxes
sandbox_directory = msb_5_6_23
sandbox_port = 5623
..
do you agree? ([Y],n) Y
loading grants
.. sandbox server started
Your sandbox server was installed in $HOME/sandboxes/msb_5_6_23](https://image.slidesharecdn.com/automated-testing-150220131520-conversion-gate01/85/Using-MySQL-in-Automated-Testing-25-320.jpg)
![Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
My
Local
Sandboxes
26
morgo@Rbook:~/sandboxes$ ls
clear_all msb_5_5_40 msb_5_6_15 msb_5_6_22
msb_5_7_3 msb_5_7_5.bak sandbox_action status_all
msb_5_5_32 msb_5_6_12 msb_5_6_16 msb_5_6_23
msb_5_7_4 plugin.conf send_kill_all stop_all
msb_5_5_36 msb_5_6_13 msb_5_6_17 msb_5_7_1
msb_5_7_5 restart_all start_all use_all
morgo@Rbook:~/sandboxes$ cd msb_5_6_23
morgo@Rbook:~/sandboxes/msb_5_6_23$ ./use
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 3
Server version: 5.6.23 MySQL Community Server (GPL)
..
Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.
mysql [localhost] {msandbox} ((none)) >
Version
5.6.23
=
Port
5623](https://image.slidesharecdn.com/automated-testing-150220131520-conversion-gate01/85/Using-MySQL-in-Automated-Testing-26-320.jpg)
![Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Examples
of
Incompatibilities
• Renaming
a
column
• $row[4];
//
Using
nth
column.
Can’t
guarantee
new
column
preserves
order.
• INSERT
into
my_table
VALUES
(1,2,3);
//
Non-‐full
inserts
if
column
count
changes
• SELECT
*
FROM
a
UNION
SELECT
*
FROM
b;
//
Column
count
mismatch
if
a
adds
a
column
• INSERT
INTO
a
SELECT
*
FROM
b
//
Column
count
mismatch
if
a
adds
a
column
45](https://image.slidesharecdn.com/automated-testing-150220131520-conversion-gate01/85/Using-MySQL-in-Automated-Testing-45-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Using MySQL in Automated Testing (20)
![[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...](https://cdn.slidesharecdn.com/ss_thumbnails/lqy2zqfxqmif8xeztxtr-signature-44327555e43d6a00760901a1f10c738a3c30da5dd494b81e3f7ae75f088cd2d2-poli-160815043717-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Morgan Tocker (11)
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
Using MySQL in Automated Testing
- 2. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Using MySQL in Automated Testing Morgan Tocker MySQL Community Manager February 2015
- 3. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3
- 4. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 4 Today’s Agenda Introduction Tools Bootstrapping Pushing Database Changes Metrics to Capture 1 2 3 4 5
- 5. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Introduction • Database are not an outlier • Can be automated / scripted / managed as part of a deployment • Many good tools exist 5
- 6. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | History of MySQL specific versions • MySQL 5.1 and below: • No Online DDL • Adding indexes to InnoDB tables causes rebuild • Supports triggers from 5.0 (!) 6
- 7. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | History of MySQL specific versions (cont.) • MySQL 5.5 (2010): • InnoDB as default • Fast ALTER TABLE 7
- 8. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | History of MySQL specific versions (cont.) • MySQL 5.6 (2013): • Online DDL • Performance Schema by default 8
- 9. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | History of MySQL specific versions (cont.) • MySQL 5.7 (In Development): • Multiple triggers per table 9
- 10. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Online DDL (MySQL 5.6+) • Can add indexes, many other changes without blocking READS or WRITES: • http://dev.mysql.com/doc/refman/5.6/en/innodb-‐ create-‐index-‐overview.html • Still requires application to handle two versions of schema. • Does not facilitate READ/WRITE split with slaves. 10
- 11. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 11 In-Place? Copies Table? Allows Concurrent DML? Allows Concurrent Query? CREATE INDEX,ADD INDEX Yes* No* Yes Yes ADD FULLTEXT INDEX Yes No* No Yes DROP INDEX Yes No Yes Yes OPTIMIZE TABLE Yes Yes Yes Yes Set default value for a column Yes No Yes Yes Change auto-increment value for a column Yes No Yes Yes Add a foreign key constraint Yes* No* Yes Yes Drop a foreign key constraint Yes No Yes Yes Rename a column Yes* No* Yes* Yes Add a column Yes Yes Yes* Yes Drop a column Yes Yes Yes Yes Reorder columns Yes Yes Yes Yes Change ROW_FORMAT property Yes Yes Yes Yes Change KEY_BLOCK_SIZE property Yes Yes Yes Yes Make column NULL Yes Yes Yes Yes Make column NOT NULL Yes* Yes Yes Yes Change data type of column No Yes No Yes Add primary key Yes* Yes Yes Yes Drop primary key and add another Yes Yes Yes Yes Drop primary key No Yes No Yes Convert character set No Yes No Yes Specify character set No Yes No Yes Rebuild with FORCE option No Yes No Yes
- 12. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 12 Today’s Agenda Introduction Tools Bootstrapping Pushing Database Changes Metrics to Capture 2 3 4 2 1 5
- 13. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Worth Considering • pt-‐online-‐schema-‐change • MySQL Sandbox • SYS • Outbrain Propagator • Liquibase • ORM specific migrations • libeatmydata 13
- 14. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | pt-‐online-‐schema-‐change • Part of “Percona Toolkit”. • Simulates Online DDL by creating a shadow table and adding triggers to the original table. • Later performs a hot switchover. • Useful for statements that do not support Online DDL. • Required in substitution of Online DDL in the case slaves are required for active use. 14
- 15. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | MySQL Sandbox • Allows you to run multiple copies at once, each self contained. • Can bootstrap a fresh install from a .tar.gz of MySQL. • Can bootstrap common topologies. • i.e. Master with a slave 15
- 16. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | SYS • Common set of views on MySQL’s Performance Schema. • Entirely SQL view of all of MySQL’s Performance. • Could potentially problematic queries in QA during test suite execution, etc… 16
- 17. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Outbrain Propagator • Based on SQL Scripts • Concept of Environments 17
- 18. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 18
- 19. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Liquibase • Specify change in XML/Yaml/JSON/SQL 19
- 20. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | ORM Specific • Rails Migrations • Django Migrations (1.7+) 20
- 21. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | libeatmydata • Disables fsyncs and related functions • Packaged in Ubuntu • apt-‐get install eatmydata 21 root@mysqldb:/usr/local/mysql# eatmydata ./bin/ mysqld —user=mysql
- 22. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 22 Today’s Agenda Introduction Tools Bootstrapping Pushing Database Changes Metrics to Capture 2 3 4 1 5 3
- 23. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Building a new QA Environment • Recommend starting with either MySQL::Sandbox or official repos. • Can restore data from mysqldump, or copying data data directory in place. 23
- 24. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Official Repos 24 sudo yum localinstall http://dev.mysql.com/get/ mysql-community-release-el6-5.noarch.rpm yum install mysql-community-server • More Information: • http://dev.mysql.com/downloads/repo/yum/ • http://dev.mysql.com/downloads/repo/apt/ • http://dev.mysql.com/downloads/repo/suse/
- 25. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | MySQL::Sandbox 25 $ make_sandbox mysql-5.6.23-osx10.9-x86_64.tar.gz unpacking /Users/morgo/Downloads/mysql-5.6.23-osx10.9-x86_64.tar.gz Executing low_level_make_sandbox --basedir=/Users/morgo/Downloads/ 5.6.23 --sandbox_directory=msb_5_6_23 --install_version=5.6 --sandbox_port=5623 --no_ver_after_name --my_clause=log-error=msandbox.err The MySQL Sandbox, version 3.0.47 (C) 2006-2013 Giuseppe Maxia installing with the following parameters: upper_directory = /Users/morgo/sandboxes sandbox_directory = msb_5_6_23 sandbox_port = 5623 .. do you agree? ([Y],n) Y loading grants .. sandbox server started Your sandbox server was installed in $HOME/sandboxes/msb_5_6_23
- 26. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | My Local Sandboxes 26 morgo@Rbook:~/sandboxes$ ls clear_all msb_5_5_40 msb_5_6_15 msb_5_6_22 msb_5_7_3 msb_5_7_5.bak sandbox_action status_all msb_5_5_32 msb_5_6_12 msb_5_6_16 msb_5_6_23 msb_5_7_4 plugin.conf send_kill_all stop_all msb_5_5_36 msb_5_6_13 msb_5_6_17 msb_5_7_1 msb_5_7_5 restart_all start_all use_all morgo@Rbook:~/sandboxes$ cd msb_5_6_23 morgo@Rbook:~/sandboxes/msb_5_6_23$ ./use Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 3 Server version: 5.6.23 MySQL Community Server (GPL) .. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql [localhost] {msandbox} ((none)) > Version 5.6.23 = Port 5623
- 27. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Restoring Data • Some testing environments (i.e. staging) may have a snapshot of actual data. • This is actually a very good practice if your data security policy allows for it. • Ensures highest chance of same performance characteristics as production. 27
- 28. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Restoring Data (cont.) • mysqldump can be very slow to load data into your newly setup environment. • Consider copying data directory in raw format instead. 28
- 29. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | How to Configure MySQL in QA? • It’s up to you! • For CI rule of thumb says that the test suite should execute in less than 5 minutes. 29
- 30. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Alternative #1 -‐ mysql-‐reckless.cnf • Disable MySQL’s durability: • Warning: Not 100% a best practice. 30 sync_frm=0 sync_binlog=0 innodb-flush-log-at-trx-commit=0 innodb-doublewrite=0 innodb_support_xa=0 innodb_checksum_algorithm=none innodb_adaptive_flushing=OFF Source: https://github.com/morgo/mysql-‐compatibility-‐config/blob/master/mysql-‐56/mysql-‐56-‐reckless.cnf
- 31. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Alternative #2 -‐ Disable fsync and friends • At a lower level, kill durability. • Keeps MySQL functionality consistent between QA and Production. • Possibly requires Linux. 31 root@mysqldb:/usr/local/mysql# eatmydata ./bin/ mysqld —user=mysql
- 32. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 32 Today’s Agenda Introduction Tools Bootstrapping Pushing Database Changes Metrics to Capture 2 3 4 1 5 4
- 33. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Two Categories of Change • Data Definition Language • aka DDL • Adding/Dropping Indexes • Adding/Dropping Columns • .. • Data manipulation language • aka DML • One off tasks to modify data • Repairing incorrect values • .. 33
- 34. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DDL Considerations • DDL does not support transactions • Is still atomic on a per-‐table basis • Grouping multiple tables may complicate deployment: 34 ALTER TABLE t1 ADD INDEX (a); ALTER TABLE t2 ADD INDEX (a); // t2 fails ALTER TABLE t3 ADD INDEX (a);
- 35. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DDL Considerations (cont.) • Good practice to group all changes for one table to single chained ALTER command. • Saves on table rebuilds where required: 35 ALTER TABLE my_table ADD my_column1 VARCHAR(255), ADD my_column2 VARCHAR(255);
- 36. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DDL Considerations (cont.) • Don’t play a guessing game with Online DDL. • ALTER TABLE supports LOCK=NONE extension: 36 CREATE TABLE acdc (a int); ALTER TABLE acdc ADD PRIMARY KEY (a), LOCK=NONE; Query OK, 0 rows affected (0.29 sec) Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE acdc DROP PRIMARY KEY, LOCK=NONE; ERROR 1846 (0A000): LOCK=NONE is not supported. Reason: Dropping a primary key is not allowed without also adding a new primary key. Try LOCK=SHARED.
- 37. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DDL Considerations (cont.) • Any time you change indexes: • Existing query plans can change • Diligent DBAs watch for regressions :) 37
- 38. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DML Example • Added a new column called is_paying_user on users table. • New code correctly sets is_paying_user to 0 or 1. • All 5M existing users will need the is_paying_user property set. 38
- 39. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DML Considerations • SQL is not guaranteed idempotent • Best to make sure that any script fails or does nothing if executed twice. 39 UPDATE t SET a = a+1; // potentially bad. UPDATE t SET a = 11; // better UPDATE t SET a = 11 WHERE a = 10; // safest
- 40. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DML Considerations (cont.) • Modifications via temporary tables are unsafe for Statement-‐based replication: 40 CREATE TEMPORARY TABLE tmp_user_payments AS SELECT DISTINCT(user_id) as user_id FROM payments WHERE payment_total > 0; /* Slave can disconnect here and lose tmp table */ UPDATE users, tmp_user_payments SET is_paying_user=1 WHERE user tmp_user_payments.user_id=users.user_id;
- 41. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Suggested Fixes: • Use Row-‐Based replication (proposed default in 5.7) • Use base table instead of temporary table. • Perform change in single step, such as via sub query (really requires 5.6+) 41
- 42. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | DML Considerations (cont.) • Locks are held for the duration of a transaction • Scripts that appear safe in QA may still create production problems • Common practice to chunk-‐split batch operations to yield locks (i.e. 10K row modifications per transaction.) • Not strictly ACID :) 42
- 43. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Distributed Systems • i.e. Sharded of Functional Partitioned Deployments • Presents special challenges • Application typically must talk to two versions of a schema at once. • Requirement still true for non distributed systems… 43
- 44. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Why two versions of a Schema? • Even when DDL is online, can’t control when it finishes and new schema is immediately visible to application. • Guaranteed safe way: • Stop old version of Application • Apply Database Changes • Start new version of Application 44 Painful
- 45. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Examples of Incompatibilities • Renaming a column • $row[4]; // Using nth column. Can’t guarantee new column preserves order. • INSERT into my_table VALUES (1,2,3); // Non-‐full inserts if column count changes • SELECT * FROM a UNION SELECT * FROM b; // Column count mismatch if a adds a column • INSERT INTO a SELECT * FROM b // Column count mismatch if a adds a column 45
- 46. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | How to make an application speak two schemas? • Introduce Social Change • Developers will not always consider deployment issues • Easy way to force them is to make database changes separate from other changes 46
- 47. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | When to add that column? • Make the change happen in the release before. • If it’s easier to explain, have two types of software releases: • Database Changes • Code Changes 47
- 48. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | When to add that column? (cont.) • Can also be handled as push prerequisites or post-‐ requisites. • Adds to complexity. • Release steps should not boil the ocean -‐ hard to keep simple and automated. 48
- 49. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Out of Scope Today • Almost starts discussion on how to refactor databases • This is a good book • Not MySQL specific 49
- 50. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 50 Today’s Agenda Introduction Tools Bootstrapping Pushing Database Changes Metrics to Capture 2 3 4 1 55
- 51. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | “Bad” Things • Not always likely to catch slow queries in QA while tests execute. • Data set can not always reflect production. • May be able to catch symptoms. 51
- 52. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Symptoms • SHOW GLOBAL STATUS LIKE 'Select_full_join'; • Slow queries • Unhandled errors or warnings • In non-‐strict versions of MySQL “warnings” are probably “bugs”. • Not recommended to use deprecated features from Day #1. 52
- 53. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Test suite execution stats • SYS schema with MySQL is very useful. • SQL Interface to fetch runtime data directly. 53
- 54. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Slow Queries 54 mysql> select * from statement_analysisG *************************** 1. row *************************** query: SELECT `sys` . `format_stateme ... cy` , `sys` . `format_time` ( db: sys full_scan: * exec_count: 4 err_count: 0 warn_count: 0 total_latency: 100.45 ms max_latency: 30.83 ms avg_latency: 25.11 ms lock_latency: 1.59 ms rows_sent: 683 rows_sent_avg: 171 rows_examined: 1367 rows_examined_avg: 342 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 683 sort_merge_passes: 0 digest: 0efbdc5d2727eea1b943011cde39e375 first_seen: 2015-02-20 07:53:52 last_seen: 2015-02-20 07:54:10
- 55. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Runtime Stats mysql> select * from wait_classes_global_by_latency; +-------------------+--------------+---------------+-------------+-------------+-------------+ | event_class | total_events | total_latency | min_latency | avg_latency | max_latency | +-------------------+--------------+---------------+-------------+-------------+-------------+ | wait/io/file | 550470 | 46.01 s | 19.44 ns | 83.58 µs | 4.21 s | | wait/io/socket | 228833 | 2.71 s | 0 ps | 11.86 µs | 29.93 ms | | wait/io/table | 64063 | 1.89 s | 99.79 ns | 29.43 µs | 68.07 ms | | wait/lock/table | 76029 | 47.19 ms | 65.45 ns | 620.74 ns | 969.88 µs | | wait/synch/mutex | 635925 | 34.93 ms | 19.44 ns | 54.93 ns | 107.70 µs | | wait/synch/rwlock | 61287 | 7.62 ms | 21.38 ns | 124.37 ns | 34.65 µs | +-------------------+--------------+---------------+-------------+-------------+-------------+ 55
- 56. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 56