MySQL Sharding: Tools and Best Practices for Horizontal Scaling

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Insert Picture Here
2013-09-21
MySQL Sharding:
Tools and Best Practices for
Horizontal Scaling
Mats Kindahl (mats.kindahl@oracle.com)
Alfranio Correia (alfranio.correia@oracle.com)
Narayanan Venkateswaran (v.narayanan@oracle.com)
2013-09-21

2013-09-21
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decision. The development,
release, and timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
Safe Harbor Statement

Program Agenda
 Handling Scaling
 What is sharding?
 Managing a Sharded Database
 Working with a Sharded Database

2013-09-21
What is
this
sharding?
What are
the benefits
of sharding?
How do I
shard my
database?

2013-09-21
It's all about scaling...
●
Start with a single server
The Growing Enterprise

2013-09-21
●
●
More and more page requests
●
...more and more reads
●
What to do?
●
Scale out!

2013-09-21
●
●
●
●
What to do?
●
Scale out!
●
Replicate to read servers

2013-09-21
●
●
●
●
What to do?
●
Scale out!
●
Replicate to read servers
●
Perform a read-write split
●
Writes go to master
●
Reads go to read servers

2013-09-21
●
More and more updates
●
Write load increases
●
What now?

2013-09-21
●
●
●
What now?
●
Add another master?

2013-09-21
●
●
●
What now?
●
Add another master?
●
Doesn't work...
●
Write load is replicated

2013-09-21
●
●
●
What now?
●
Add another master?
●
Doesn't work...
●
Write load is replicated
●
Partition database
●
Distribute writes
●
Called sharding
UID 10000-20000 UID 20001-40000

2013-09-21
Benefits of Sharding
●
Write scalability
●
Can handle more writes
●
Large data set
●
Database too large
●
Does not fit on single server
●
Improved performance
●
Smaller index size
●
Smaller working set
●
Improve performance
UID 10000-20000 UID 20001-40000

Insert Picture Here
Architecture of a
Sharding Solution

What do you need to consider?
 High level architecture
 Transaction and sharding key handling
 Granularity of sharding

2013-09-21
High-level Architecture
●
What components do we need?
●
How are they deployed?

2013-09-21
Components for a Sharding Solution
Shard #2
Shard #1
Shard #3
Switch
State
Store Executor
QUERY
KEY
KEY
QUERY
Contain decision logic
for distributing queries
Contain information
about location of shards
SHARD#

2013-09-21
Deployment of the Sharding Solution
Shard #2
Shard #1
Shard #3
Switch
State
Store Executor
Network hop!
Performance?
Protocol?
Single Points
of Failure!
Caches can be
used to avoid
performance impact

2013-09-21
Deployment of the Sharding Solution
Shard #2
Shard #1
Shard #3
Switch
State
Store Executor
Deployed with application
(e.g., inside connector)
API simple to add

2013-09-21
Transaction and Shard Key Handling
●
How are the transactions handled?
●
How do get the shard key for a transaction?
●
How to compute shard from key?

2013-09-21
BEGIN
SELECT salary INTO @s FROM salaries WHERE emp_no = 20101;
SET @s = 1.1 * @s;
INSERT INTO salaries VALUES (20101, @s);
COMMIT
BEGIN
INSERT INTO ...
COMMIT
Sharding key? Ah, there it is!
Session state?
Hmm... looks like
a read transaction
Oops.. it was a
write transaction!
Transaction done!
Clear session state?
New transaction! Different shard?
What about the session state?
Transaction Handling

2013-09-21
Transaction Handling
●
Detecting transaction boundaries
●
Managing session state
●
Move session state between servers
– Easy to use
– Expensive and error prone
●
Reset state after each transaction
– Transactions start with default session state
What about
crashes?
Where do I store
the session state?

2013-09-21
Mapping the Sharding Key
●
What is a sharding key?
●
Single column
●
Multi column
– Same table?
– Different tables?
●
How is the key transformed?
●
Hash
●
Range
●
User-defined
Compute
Shard#
KeyShard#
(X)
(X,Y,...)
RANGE
HASH
Something else

2013-09-21
Granularity of Sharding
●
Can multiple tables be sharded with the same key?
●
Can we shard different tables different ways?
●
Do we allow global tables?
●
Do we allow cross-database queries?

2013-09-21
Sharded Tables
Table Rows
salaries 284 404 700
titles 44 330 800
employees 30 002 400
dept_emp 33 160 300
dept_manager 2 400
departments 900
In desperate need
of sharding!
Foreign keys

2013-09-21
Multi-table Query with Sharded Tables
SELECT first_name, last_name, salary
FROM salaries JOIN employees USING (emp_no)
WHERE emp_no = 21012
AND CURRENT_DATE BETWEEN from_date AND to_date;
● Referential Integrity Constraint
● Example query joining salaries and employees
● Same key, same shard: co-locate rows for same user
● JOIN normally based on equality
● Using non-equality defeats purpose of foreign key

2013-09-21
Global Tables
Table Rows
salaries 284 404 700
titles 44 330 800
employees 30 002 400
dept_emp 33 160 300
dept_manager 2 400
departments 900
Do not really need
to be sharded

2013-09-21
Multi-table Query with Global Tables
SELECT first_name, last_name, GROUP_CONCAT(dept_name)
FROM employees JOIN dept_emp USING (emp_no)
JOIN departments USING (dept_no)
WHERE emp_no = 21012 GROUP BY emp_no;
● JOIN with departments table
● Has no employee number, hence no sharding key
● Table need to be present on all shards
● How do we update global tables?

Insert Picture Here
Managing a
Sharded Database

Insert Picture Here
An extensible and easy-to-use
framework for managing a farm
of MySQL server supporting
high-availability and sharding
MySQL Fabric

2013-09-21
MySQL Fabric: What is it?
●
“Farm” Management System
●
High-Availability
●
Sharding
●
Distributed
●
Procedure Execution
●
Extensible
●
Written in Python
●
Early alpha
●
Long road ahead
●
Open Source
●
You can participate
●
Suggest features
●
Report bugs
●
Contribute patches
●
MySQL 5.6 is focus

2013-09-21
MySQL Fabric: Features
●
Decision logic in connector
●
Reducing network load
●
Support Transactions
●
API to provide sharding key
●
Global Updates
●
Global Tables
●
Schema updates
●
Procedure Executor
●
Shard Multiple Tables
●
Using same key
●
Sharding Functions
●
Range
●
(Consistent) Hash
●
Shard Operations
●
Using built-in executor
●
Shard move
●
Shard split

2013-09-21
Birds-eye View of a Sharded Database
High Availability Groups
(Shards)
MySQL
Fabric
Node
Application
XML-RPC
SQL
SQL
Connector
Connector
Connector
Operator

2013-09-21
MySQL Fabric Node Architecture
MySQL
MySQL Fabric
Framework
Executor
State Store
(Persister)
Sh
?HA
MySQLAMQP XML-RPC
??
Connector
Connector
Connector
Protocols
Extensions
Backing
Store

2013-09-21
MySQL Fabric: Prerequisites
●
MySQL Servers (version 5.6.10 or later)
●
Server for meta-data backing store
●
Servers being managed
●
Python 2.6 or 2.7
●
No support for 3.x yet
●
MySQL Utilities 1.4.0
●
Available at http://labs.mysql.com/

2013-09-21
MySQL Fabric: Configuration
●
Backing Store
●
MySQL server
●
Persistent storage for state
●
Storage engine-agnostic
●
Protocol
●
Address where node will be
●
Currently only XML-RPC
●
Logging
●
Chatty: INFO (default)
●
Moderate: WARNING
●
URL for rotating log
[storage]
address = localhost:3306
user = fabric
password =
database = fabric
connection_timeout = 6
[protocol.xmlrpc]
address = localhost:8080
threads = 5
[logging]
level = INFO
url = file:///var/log/fabric.log

2013-09-21
MySQL Fabric: Setup and Teardown
●
Create the necessary tables in backing store
mysqlfabric manage setup
●
Remove the tables from backing store
mysqlfabric manage teardown
●
Connect to database server in “storage” section
●
Ensure that you have the necessary users and privileges

2013-09-21
MySQL Fabric: Starting and Stopping
●
Start MySQL Fabric node in foreground – print log to terminal
mysqlfabric manage start
●
Start MySQL Fabric node in background – print log to file
mysqlfabric manage start daemonize
●
Stop MySQL Fabric node
mysqlfabric manage stop

2013-09-21
Sharding Architecture
Shards
MySQL
Fabric Node
Application
Connector
Connector
Connector
Global
Group
Global Updates
Shard
Updates
Replication
Support global update
for off-line shards

2013-09-21
MySQL Fabric: Sharding Setup
●
Set up some groups
●
my_global – for global updates
●
my_group.* – for the shards
●
Add servers to the groups
●
Create a shard mapping
●
A “distributed database”
●
Mapping keys to shards
●
Give information on what tables are sharded
●
Add shards

2013-09-21
MySQL Fabric: Create Groups and add Servers
●
Define a group
mysqlfabric group create my_global
●
Add servers to group
mysqlfabric group add my_global global.example.com
mats xyzzy
mysqlfabric group add my_global …
User + Password
(Likely to go away)

2013-09-21
MySQL Fabric: Create Groups and add Servers
●
Promote one server to be primary
mysqlfabric group promote my_global
●
Tell failure detector to monitor group
mysqlfabric group activate my_global

2013-09-21
MySQL Fabric: Set up Shard Mapping
●
Define shard mapping
mysqlfabric sharding define hash my_global
●
Add tables that should be sharded
mysqlfabric sharding add_mapping 1
employees.employees emp_no
mysqlfabric sharding add_mapping 1
employees.salaries emp_no
●
Tables not added are global
Will show the shard map
identifier (a number)
Shard map identifier

2013-09-21
MySQL Fabric: Add Shards
●
Add shards to shard mapping
mysqlfabric sharding add_shard 1 my_group.1 enabled
    .
    .
    .
mysqlfabric sharding add_shard 1 my_group.N enabled
Shard map identifier

2013-09-21
MySQL Fabric: Moving and Splitting Shards
●
Moving a shard from one group to another
mysqlfabric sharding move 5 my_group.5
●
Splitting a shard into two pieces (hash)
mysqlfabric sharding split 5 my_group.6
Shard ID

Insert Picture Here
Working with a
Sharded Database

2013-09-21
Fabric-aware Connector API
●
Support Transactions
●
Sharding key out of band
●
Fabric-aware Connectors
●
Connector/J
●
Connector/Python
●
Connector/PHP
●
Fabric-aware Frameworks
●
Doctrine
●
Hibernate
●
Focus on Connector/Python

2013-09-21
Fabric-aware Connector API
import mysql.connector.fabric as connector
conn = connector.MySQLFabricConnection(
fabric={"host": "fabric.example.com", "port" : 8080},
user='mats', database="employees")
● Establish a “virtual” connection
● Real server connection established lazily
● Provide connection information for the Fabric node
● Fabric node will provide information about real servers

2013-09-21
Digression: Computing Shards
●
Multiple Mappings
●
Which mapping to use?
●
Application don't care
… but know tables in transaction
●
Currently only one mapping
●
Computing shard requires
●
Tables + sharding key
●
Extended Connector API
●
Extra properties passed out-of-band
Compute
Shard#
Key
Shard#Map#
Compute
Map#
Tables

2013-09-21
Connector API: Shard Specific Query
●
Provide tables in query
●
Property: tables
●
Fabric will compute map
conn.set_property(tables=['employees.employees', 'employees.titles'],
                  key=emp_no)
cur = conn.cursor()
cur.execute("INSERT INTO employees VALUES (%s,%s,%s)",
            (emp_no, first_name, last_name))
cur.execute("INSERT INTO titles(emp_no,title,from_date)"
            “ VALUES (%s,%s,CURDATE())",
            (emp_no, 'Intern'));
conn.commit()
● Provide sharding key
● Property: key
● Fabric will compute shard
Transactions work fine!

2013-09-21
Connector API: Shard Specific Query
●
●
Property: tables
●
conn.set_property(tables=['employees.employees', 'employees.titles'],
                  key=emp_no)
cur = conn.cursor()
cur.execute(
    "SELECT first_name, last_name, title"
    "  FROM employees JOIN titles USING (emp_no)"
    " WHERE emp_no = %d", (emp_no,))
for row in cur:
    print row[0], row[1], “,“, row[2]
● Provide sharding key
● Property: key
● Fabric will compute shard
Join queries are sent to correct
shard and executed there

2013-09-21
Connector API: Global Update
●
●
Property: tables
●
conn.set_property(tables=['employees.titles'], scope='GLOBAL')
cur = conn.cursor()
cur.execute("ALTER TABLE employees.titles ADD nickname VARCHAR(64)")
● Set global scope
● Property: scope
● Query goes to global group

Insert Picture Here
Closing Remarks

2013-09-21
Thoughts for the Future
●
Connector multi-cast
●
Scatter-gather
●
Internal interfaces
●
Improve extension support
●
Improve procedures support
●
Command-line interface
●
Improving usability
●
Focus on ease-of-use
●
More protocols
●
MySQL-RPC Protocol?
●
AMQP?
●
More frameworks?
●
More HA group types
●
DRBD
●
MySQL Cluster
●
Fabric-unaware connectors?

2013-09-21
Thoughts for the Future
●
“More transparent” sharding
●
Single-query transactions
●
Cross-shard joins is a problem
●
Multiple shard mappings
●
Independent tables
●
Multi-way shard split
●
Efficient initial sharding
●
Better use of resources
●
High-availability executor
●
Node failure stop execution
●
Replicated State Machine
●
Fail over to other Fabric node
●
Distributed failure detector
●
Connectors report failures
●
Custom failure detectors

2013-09-21
MySQL Connect Sessions
● MySQL High Availability: Managing Farms of Distributed Servers
● September 22, 5:30pm-6:30pm in Imperial Ballroom B
● Scaling PHP Applications
● September 22, 10:00am-11:00am in Union Square Room ¾
● MySQL Sharding, Replication, and HA
● September 21, 5:30-6:30pm in Imperial Ballroom B

2013-09-21
References
● MySQL Forum: Fabric, Sharding, HA, Utilities
● http://forums.mysql.com/list.php?144
● A Brief Introduction to MySQL Fabric
● http://mysqlmusings.blogspot.com/2013/09/brief-introduction-to-mysql-fabric.html
● MySQL Fabric – Sharding – Introduction
● http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-introduction.html
● Migrating From an Unsharded to a Sharded Setup
● http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html

2013-09-21
Mats Kindahl
Twitter: @mkindahl
Blog: http://mysqlmusings.blogspot.com
Alfranio Correia
Twitter: @alfranio
Blog: http://alfranio-distributed.blogspot.com
Narayanan Venkateswaran
Twitter: @vn_tweets
Blog: http://vnwrites.blogspot.com
Keeping in Touch

2013-09-21
Thank you!

MySQL Sharding: Tools and Best Practices for Horizontal Scaling

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MySQL Sharding: Tools and Best Practices for Horizontal Scaling (20)

More from Mats Kindahl (10)

Recently uploaded (20)

MySQL Sharding: Tools and Best Practices for Horizontal Scaling