How to Manage Scale-Out Environments with MariaDB MaxScale
1 like1,346 views
MaxScale is a database proxy that provides load balancing, connection pooling, and replication capabilities for MariaDB and MySQL databases. It can be used to scale databases horizontally across multiple servers for increased performance and availability. The document provides an overview of MaxScale concepts and capabilities such as routing, filtering, security features, and how it can be used for operational tasks like query caching, logging, and data streaming. It also includes instructions on setting up MaxScale with a basic example of configuring read/write splitting between a master and slave database servers.
![MaxScale Configuration – Threads
• MariaDB MaxScale configuration is held
in an ini file, /etc/maxscale.cnf.
• This must be manually created, a template file does
exist within the /usr/share/maxscale directory.
• A global, maxscale, section is included within
every MariaDB MaxScale configuration file.
• Various MariaDB MaxScale wide parameters
are set in the file.
• The most important of these is the number of threads
that MariaDB MaxScale will use to execute the code that
forwards requests and handles responses for clients.
[maxscale]
threads=4](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-25-320.jpg)
![MaxScale Configuration – Service
• The first step is to create a service
for our Read/Write Splitter.
• Create a section in your MariaDB MaxScale
configuration file and set the type to service.
[Splitter Service]
type=service](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-26-320.jpg)
![MaxScale Configuration readwritesplit Module
• The router we need to use for this configuration
is the readwritesplit module.
• The server names given here are actually the names
of server sections in the configuration file and not the
physical hostnames or addresses of the servers.
[Splitter Service]
type=service
router=readwritesplit
servers=dbserv1,
dbserv2, dbserv3](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-27-320.jpg)
![MaxScale Configuration – Splitter Service
• The username and password, either encrypted
or plain text, are stored in the service section using
the user and passwd parameters.
[Splitter Service]
type=service
router=readwritesplit
servers=dbserv1, dbserv2, dbserv3
user=maxscale
passwd=96F99AA1315BDC3604B006F427
DD9484](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-30-320.jpg)
![MaxScale Configuration – Listener Service
• Listening ports must be associated with the
service in order to allow network connections.
• This is done by creating a series of listener sections.
• A service may have multiple listeners.
[Splitter Listener]
type=listener
service=Splitter Service](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-31-320.jpg)
![MaxScale Configuration – More Listener Service
• A listener must also define the protocol module
it will use for the incoming network protocol,
currently this should be the MySQLClient protocol
for all database listeners.
• The listener may then supply a network port to
listen on and/or a socket within the file system.
[Splitter Listener]
type=listener
service=Splitter Service
protocol=MySQLClient
port=3306
socket=/tmp/ClusterMaster](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-32-320.jpg)
![MaxScale Configuration – Servers
• An address parameter may be given if the listener
is required to bind to a particular network address
when using hosts with multiple network addresses.
• The next stage is the configuration is to define
the server information.
[dbserv1]
type=server
address=192.168.2.1
port=3306
protocol=MySQLBackend](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-33-320.jpg)
![MaxScale Configuration – Servers
• An address parameter may be given if the listener
is required to bind to a particular network address
when using hosts with multiple network addresses.
• The next stage is the configuration is to define
the server information.
[dbserv2]
type=server
address=192.168.2.2
port=3306
protocol=MySQLBackend](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-34-320.jpg)
![MaxScale Configuration – Servers
• An address parameter may be given if the listener
is required to bind to a particular network address
when using hosts with multiple network addresses.
• The next stage is the configuration is to define
the server information.
[dbserv3]
type=server
address=192.168.2.3
port=3306
protocol=MySQLBackend](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-35-320.jpg)
![MaxScale Configuration – Replication Monitor
• In order for MariaDB MaxScale to monitor the servers
using the correct monitoring mechanisms a section
should be provided that defines the monitor to use and
the servers to monitor.
• Once again a section is created with a symbolic name
for the monitor, with the type set to monitor.
• Parameters are added for the module to use, the list of
servers to monitor and the username and password to
use when connecting to the servers with the monitor.
[Replication Monitor]
type=monitor
module=mysqlmon
servers=dbserv1, dbserv2, dbserv3
user=maxscale
passwd=96F99AA1315BDC3604B006F427
DD9484](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-36-320.jpg)
![MaxScale Configuration – Services and Listener
Section
• As with the password definition in the server either
plain text or encrypted passwords may be used.
• The final stage in the configuration is to add the
option service which is used by the maxadmin command
to connect to MariaDB MaxScale for monitoring and
administration purposes. This creates a service section
and a listener section.
[CLI]
type=service
router=cli
[CLI Listener]
type=listener
service=CLI
protocol=maxscaled
socket=default](https://image.slidesharecdn.com/mariadbmaxscale-phoenixroadshow-171218142941/85/How-to-Manage-Scale-Out-Environments-with-MariaDB-MaxScale-37-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to How to Manage Scale-Out Environments with MariaDB MaxScale (20)
Ad
More from MariaDB plc (20)
Ad
Recently uploaded (20)
How to Manage Scale-Out Environments with MariaDB MaxScale
- 1. MariaDB MaxScale James McLaurin Solution Architect - MariaDB
- 2. MariaDB MaxScaleMariaDB Multi-Master Cluster O P E R A T I N G S Y S T E M / F I L E S Y S T E M / S A N / C L O U D Application Connectors MariaDB Server NoSQL CRUD API Original Core MariaDB MariaDB Engineering Community Contribution Replicas Supporting Asynchronous, Semi-Sync & Synchronous replication MariaDB C JDBC ODBC Replication Kernel Production Plugins Parallel Slave GTIDBinLog API Multi-Source SQL Parser Cache/Buffer Optimiser Connection Pool Temporal PL/SQL Audit AWS KMS Authentication Handler Socket Etc. 40+ Plugins SQL Lightweight Transactional Interoperability Performance & Scalability Graph & SearchAnalytics InnoDB XtraDBAria Memory MyISAM CONNECTColumnStoreSpider OQGRAPH MyRocks Mroonga STORAGE LAYER EXTENSIBILITY KERNEL EXTENSIBILITY
- 3. 3 MariaDB MaxScale Latest GA Version 2.1 Part of MariaDB TX and AX Offering Next Generation Database Proxy for SCALABILITY SECURITY HIGH AVAILABILITY DATA STREAMING
- 4. MariaDB MaxScale Concept DATABASE SERVERS Primary Secondary Binlog Cache Insulates client applications from the complexities of backend database cluster Simplify replication from database to other databases CLIENT PROTOCOL SUPPORT AUTHENTICATION DATABASE MONITORING LOAD BALANCING & ROUTING QUERY TRANSFORMATION & LOGGING Flexible, easy to write plug-ins for Generic Core MULTI-THREADED E-POLL BASED STATELESS SHARES THE THREAD POOL
- 5. MaxScale Core and Plugins Filter Client Protocol Protocol Filter Filter Router Server State Monitor Parser Core updates monitors uses Backend
- 6. Scalability Transaction Scaling to support user growth and simplify applications Connection Rate Limitation Query Cache LOAD BALANCING TO MARIADB MASTER/SLAVE MARIADB GALERA CLUSTER AURORA CLUSTER Replication Scaling to support web-scale applications’ user base BINLOG SERVER FOR HORIZONTAL SCALING OF SLAVES IN MASTER/SLAVE ARCHITECTURE Multi-tenant database scaling to transparently grow tenants and data volume SCHEMA SHARDING HANDLE LARGE AMOUNT OF READ OPERATIONS WITH HIGH PERFORMANCE
- 7. Routing based on QUERY TYPES QUERY PATTERNS DATABASE SERVER STATE REPLICATION LAGS Load balancing across database clusters MARIADB MASTER/SLAVE MARIADB GALERA MARIADB MULTI-MASTER AWS AURORA Scaling Scale database environment without application impact Minimize maintenance downtime Connection Based Routing readwrite Primary Node Secondary read write Statement Based Routing (Read- Write Split) readwrite Primary Secondary read & write
- 8. Scaling: Tunability Route all reads to master subsequent to a write for a configurable amount of time or number of operations Route queries to only those slave that are less than configurable replication lag behind master Route all queries that match a pattern(regex) to specific servers Weighted routing
- 9. Scaling: Multi-tenant Database Each tenant with its own schema Multi-tenant database hosting Without impacting existing user base Scale the database with user base and data volume growth Multi-tenant Database Routing (Schema Sharding) shard n...shard 2shard 1
- 10. Scaling: Binlog Replication Large scale Master/Slave Replication Clusters Transparent MariaDB binlog replication relay Horizontally Scale Slaves without Master Overload Secondary Primary Node Secondary Binlog Cache Binlog Cache
- 11. Query Cache in memory LRU cache. Clients Database Servers Query Cache Filter CACHE QUERY RESULTS IN MAXSCALE FOR CONFIGURABLE TIMEOUT FOR CACHED QUERIES RETURN RESULTS FROM CACHE Handle large amount of read operations with high performance
- 12. Security in MaxScale Black & White List Connection Rate Limitation End to End SSL Database Firewall Filter for SQL Injection protection Selective Data Masking HIPPA/PCI Compliance Maximum Rows Returned Limit DDoS Protection Transport layer security between applications, proxy & databases DDoS Protection LDAP/GSSAPI Authentication Support Encrypted Binlog Server Files SSL between binlog server and Master/Slave
- 13. QUERY FAILED: 1141 ERROR: Required WHERE/HAVING clause is missing rule safe_select deny no_where_clause on_queries select rule safe_cust_select deny regex '.*from.*customers.*' user %app-user@% match all rules safe_cust_select safe_select Security DATABASE FIREWALL FILTER SELECT * FROM CUSTOMERS; MaxScale Database Servers 1 2 3 Database Firewall Filter Allow/Block queries that MATCH A SET OF RULES MATCH RULES FOR SPECIFIED USERS MATCH ON • date/time • a WHERE clause • query type • column match • a wildcard or regular expression or function name Protect against SQL injection Prevent unauthorized data access Prevent data damage
- 14. Security SELECT Name, creditcardNum, balance FROM customerTbl WHERE id=1001 Name creditcardNum balance --------------------------------------- John Smith xxxxxxxxxx 1201.07 Database Servers Client HIPPA/PCI Compliance: Selective Data Masking based on column name DATABASE NAME, TABLE NAME CLASSIFIER MAY BE PROVIDED • commerceDb.customerTbl.creditcardNum • customerTbl.creditcardNum • credicardNum
- 15. Security DDoS Protection MAXIMUM ROWS FILTER • Return zero rows to client if number of rows in result set exceeds configured max limit • Return zero rows to client if the size of result set exceeds configured max size in KB Max Rows Limit = 500 NumRows Returned > MaxRows Limit QUERY FAILED: 1141 ERROR: No rows returned 51 QUERY 4 MaxRowsLimit FILTER Clients NumRows returned = 100032 Database Servers QUERY
- 16. Security Secondary Primary Secondary SSL SSL SSL SSL Secured Binlog Server ENCRYPT BINLOG SERVER FILES on MaxScale SSL between binlog server and Master/Slave Secured user access LDAP/GSSAPI for secured single sign-on across OS platforms(windows, linux), applications and databases Binlog Cache Binlog Cache
- 17. Operational Use Cases Dynamically configure server, listener, monitor Dynamically configure database firewall rules Query Duplication Query Cache DUPLICATE QUERY BETWEEN PRODUCTION AND STAGING FOR UPGRADE VERIFICATION DUPLICATE QUERY BETWEEN DATABASE AND EXTERNAL APPLICATIONS FOR QUERY ANALYTICS Query logging CLUSTER WIDER QUERY TUNING Query translation TRANSLATE QUERIES SYNTAX FROM ONE PROTOCOL TO ANOTHER ASYNCHRONOUSLY UPGRADE APPLICATIONS AND DATABASE INVOKE EXTERNAL OPERATIONAL TOOLS FOR SERVER DOWN/UP EVENT MANAGEMENT MANAGE FAILOVER - PROMOTE SLAVE TO MASTER UPON FAILURE
- 18. Binlog conversion to Avro Streaming of Change Data Capture events (binlog) to big data environments Data Streaming Change Data Capture Clients EMR MariaDB ColumnStore Big Data Platforms AVRO or JSON events Streaming MariaDB Primary Binlog to AVRO conversion
- 19. MariaDB MaxScale 2.2 MariaDB 10.2 Support REST based management PAMD Authentication More performance enhancements MaxScale CDC to MariaDB ColumnStore Data Adapter
- 20. Learn more Download https://mariadb.com/downloads/mariadb-tx/maxscale Documentation https://mariadb.com/kb/en/mariadb-enterprise/maxscale/ Blogs https://mariadb.com/blog-tags/maxscale Product Page https://mariadb.com/products/technology/maxscale Webinars https://mariadb.com/resources/webinars?combine=MaxScale Reach me [email protected]
- 21. Thank you
- 23. Read/Write Splitting MariaDB Replication + R/W Split Routing Each application server uses only 1 connection MaxScale monitors the state of each node and only applies operations on available slaves MaxScale creates 2 connections, one for R/W on the master node and one for R/O load balanced on the slave nodes Max Scale R/W Splitting
- 24. Install MariaDB MaxScale How to set up MariaDB MaxScale Directions here for reference: https://mariadb.com/kb/en/mariadb- enterprise/setting-up-mariadb-maxscale/ Install the package relevant to your distribution Step 1 Create the required users in your MariaDB or MySQL Replication cluster Step 2 Create a MariaDB MaxScale configuration file Step 3
- 25. MaxScale Configuration – Threads • MariaDB MaxScale configuration is held in an ini file, /etc/maxscale.cnf. • This must be manually created, a template file does exist within the /usr/share/maxscale directory. • A global, maxscale, section is included within every MariaDB MaxScale configuration file. • Various MariaDB MaxScale wide parameters are set in the file. • The most important of these is the number of threads that MariaDB MaxScale will use to execute the code that forwards requests and handles responses for clients. [maxscale] threads=4
- 26. MaxScale Configuration – Service • The first step is to create a service for our Read/Write Splitter. • Create a section in your MariaDB MaxScale configuration file and set the type to service. [Splitter Service] type=service
- 27. MaxScale Configuration readwritesplit Module • The router we need to use for this configuration is the readwritesplit module. • The server names given here are actually the names of server sections in the configuration file and not the physical hostnames or addresses of the servers. [Splitter Service] type=service router=readwritesplit servers=dbserv1, dbserv2, dbserv3
- 28. MaxScale Configuration – Passwords • The final step in the service sections is to add the username and password that will be used to populate the user data from the database cluster. • There are two options for representing the password, either plain text or encrypted passwords. • In order to use encrypted passwords a set of keys must be generated that will be used by the encryption and decryption process. To generate the keys, use the maxkeys command and pass the name of the secrets file in which the keys are stored. maxkeys /var/lib/ maxscale/.secrets
- 29. MaxScale Configuration – More Passwords • Once the keys have been created the maxpasswd command can be used to generate the encrypted password. maxpasswd plainpassword 96F99AA1315BDC3604B00 6F427DD9484
- 30. MaxScale Configuration – Splitter Service • The username and password, either encrypted or plain text, are stored in the service section using the user and passwd parameters. [Splitter Service] type=service router=readwritesplit servers=dbserv1, dbserv2, dbserv3 user=maxscale passwd=96F99AA1315BDC3604B006F427 DD9484
- 31. MaxScale Configuration – Listener Service • Listening ports must be associated with the service in order to allow network connections. • This is done by creating a series of listener sections. • A service may have multiple listeners. [Splitter Listener] type=listener service=Splitter Service
- 32. MaxScale Configuration – More Listener Service • A listener must also define the protocol module it will use for the incoming network protocol, currently this should be the MySQLClient protocol for all database listeners. • The listener may then supply a network port to listen on and/or a socket within the file system. [Splitter Listener] type=listener service=Splitter Service protocol=MySQLClient port=3306 socket=/tmp/ClusterMaster
- 33. MaxScale Configuration – Servers • An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses. • The next stage is the configuration is to define the server information. [dbserv1] type=server address=192.168.2.1 port=3306 protocol=MySQLBackend
- 34. MaxScale Configuration – Servers • An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses. • The next stage is the configuration is to define the server information. [dbserv2] type=server address=192.168.2.2 port=3306 protocol=MySQLBackend
- 35. MaxScale Configuration – Servers • An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses. • The next stage is the configuration is to define the server information. [dbserv3] type=server address=192.168.2.3 port=3306 protocol=MySQLBackend
- 36. MaxScale Configuration – Replication Monitor • In order for MariaDB MaxScale to monitor the servers using the correct monitoring mechanisms a section should be provided that defines the monitor to use and the servers to monitor. • Once again a section is created with a symbolic name for the monitor, with the type set to monitor. • Parameters are added for the module to use, the list of servers to monitor and the username and password to use when connecting to the servers with the monitor. [Replication Monitor] type=monitor module=mysqlmon servers=dbserv1, dbserv2, dbserv3 user=maxscale passwd=96F99AA1315BDC3604B006F427 DD9484
- 37. MaxScale Configuration – Services and Listener Section • As with the password definition in the server either plain text or encrypted passwords may be used. • The final stage in the configuration is to add the option service which is used by the maxadmin command to connect to MariaDB MaxScale for monitoring and administration purposes. This creates a service section and a listener section. [CLI] type=service router=cli [CLI Listener] type=listener service=CLI protocol=maxscaled socket=default
- 38. Starting MariaDB MaxScale • Upon completion of the configuration process MariaDB MaxScale is ready to be started for the first time. This may either be done manually by running the maxscale command or via the service interface. % maxscale or % service maxscale start
- 39. Services ------------------+----------------+--------+--------------- Service Name | Router Module | #Users | Total Sessions ------------------+----------------+--------+--------------- Splitter Service | readwritesplit | 1 | 1 CLI | cli | 2 | 2 ------------------+----------------+--------+--------------- List Services Check the error log in /var/log/maxscale to see if any errors are detected in the configuration file and to confirm MariaDB MaxScale has been started. Also the maxadmin command may be used to confirm that MariaDB MaxScale is running and the services, listeners etc., have been correctly configured. % maxadmin list services
- 40. Servers List Servers % maxadmin list servers ----------+--------------+-------+-------------+--------------- Server | Address | Port | Connections | Status ----------+--------------+-------+-------------+--------------- dbserv1 | 192.168.2.1 | 3306 | 0 | Running, Slave dbserv2 | 192.168.2.2 | 3306 | 0 | Running, Master dbserv3 | 192.168.2.3 | 3306 | 0 | Running, Slave ----------+--------------+-------+-------------+---------------
- 41. Listeners List Listeners MariaDB MaxScale is now ready to start accepting client connections and routing them to the master or slaves within your cluster. • Other configuration options are available that can alter the criteria used for routing, these include monitoring the replication lag within the cluster and routing only to slaves that are within a predetermined delay from the current master or using weights to obtain unequal balancing operations. • These options may be found in the MariaDB MaxScale Configuration Guide. More detail on the use of maxadmin can be found in the document. % maxadmin list listeners -----------------+-----------------+-----------+-------+-------- Service Name | Protocol Module | Address | Port | State -----------------+-----------------+-----------+-------+-------- Splitter Service | MySQLClient | * | 3306 | Running CLI | maxscaled | localhost | 6603 | Running -----------------+-----------------+-----------+-------+--------
- 42. Testing the Read - Write Splitting A simple select statement routes to one of the slaves A grouped transaction could contain writes, so is considered as one. It should route to the master MariaDB> select @@hostname; +------------+ | @@hostname | +------------+ | dbserve1 | +------------+ MariaDB> start transaction; MariaDB> select @@hostname; +------------+ | @@hostname | +------------+ | dbserve2 | +------------+