Grafana and MySQL - Benefits and Challenges
Download as PPTX, PDF6 likes17,360 views
The document presents a case study on integrating Grafana with MySQL for tracking and visualizing metrics in Ingram Content Group's automated print-on-demand division. It discusses the challenges faced while using InfluxDB, such as incompatibility with SQL and performance issues, which led to the decision to utilize MySQL instead. The benefits of this integration include leveraging existing knowledge of MySQL, handling large data volumes efficiently, and using full SQL capabilities for queries.
Grafana and MySQL - Benefits and Challenges
- 1. Grafana and MySQL Benefits and Challenges Aemelia Presentation Template from http://www.slidescarnival.com
- 2. About me Philip Wernersbach Software Engineer Ingram Content Group https://github.com/philip-wernersbach https://www.linkedin.com/in/pwernersbach 2
- 3. • I work in Ingram Content Group’s Automated Print On Demand division • We have an automated process in which publishers (independent or corporate) request books via a website, and we automatically print, bind, and ship those books to them • This process involves lots of hardware devices and software components 3
- 4. We collect: • Device Status (Online/Offline) • Tracking data for every piece of material for every book • Throughput metrics • Etc. This produces gigabytes of data an hour. 4
- 6. The Problem “How do we aggregate and track metrics from our hardware and software sources, and display those data points in a graph format to the end user?” Grafana! 6
- 7. Which data store should we use with Grafana? ▸ Out of the box, Grafana supports Elasticsearch, Graphite, InfluxDB, KairosDB, OpenTSDB 7
- 8. Which data store should we use with Grafana? ▸ We compared the options and tried InfluxDB ▸ There were several sticking points with InfluxDB, both technical and organizational, that caused us to rule it out 8
- 9. Which data store should we use with Grafana? ▸ We already have a MySQL cluster deployed, System Administrators and Operations know how to manage it ▸ Decided to go with MySQL as a data store for Grafana 9
- 10. The Solution: Ingram Content’s Grafana-MySQL Integration 10
- 11. The Integration ▸Written in Nim ▸Emulates an InfluxDB server ▸Connects to an existing MySQL server ▸Protocol compatible with InfluxDB 0.9.3 ▸Acts as a proxy that converts the InfluxDB protocol to the MySQL 11
- 14. InfluxDB ▸ Time Series database ▸ “time” is a special value, has special meaning ▸ Applies special logic to “time” SMART! The Challenges! Database Engine Comparison MySQL ▸ Generic relational database ▸ “time” is a generic data value ▸ Doesn’t apply special logic DUMB! 14
- 15. The Challenges! Query Comparison InfluxDB “SELECT count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > now() - 7d GROUP BY time(1h)” 15
- 16. The Challenges! Query Comparison MySQL “SELECT time, count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > NOW(6) - INTERVAL 7 DAY GROUP BY YEAR(time), MONTH(time), DAY(time), HOUR(time) ORDER BY time ASC” 16
- 17. The Challenges! InfluxQL != SQL InfluxQL != SQL ▸ InfluxQL is SQL-like, but different enough that it can’t be passed through to MySQL 17
- 18. The Challenges! InfluxQL != SQL ▸ “time” is SELECT’d automatically implicitly in InfluxDB ▹ “SELECT bar FROM foo” “SELECT time, bar FROM FOO” ▸ GROUP’ing on “time” is smart in InfluxDB, and dumb in MySQL ▹ See slides 16 and 17 ▸ “time” in epoch format with millisecond precision is a float in MySQL ▹ “FROM_UNIXTIME(1444667802.145)” 18
- 19. The Challenges! InfluxQL != SQL ▸ InfluxDB series are roughly equivalent to MySQL tables, but the management syntax is sometimes different ▹ “DROP SERIES FROM foo” “DELETE FROM foo” ▸ InfluxDB allows quoted identifiers, MySQL is more restrictive ▹ “SELECT * FROM foo WHERE bar > ‘0’” Silently returns garbage in MySQL ▹ Grafana quotes identifiers aggressively, so we had to patch it 19
- 20. The Challenges! InfluxQL != SQL There’s probably more InfluxQL- SQL incompatibilities, but these are the ones that we’ve run into with Grafana at Ingram Content Group. 20
- 21. The Benefits! 21
- 22. The Benefits! ▸ Allows us to leverage System Administrator’s and Operation’s existing knowledge of MySQL ▸ Allows us to process large amounts of data ▹ InfluxDB had issues when we tried to insert about a million points at a time, MySQL handles this with no problem 22
- 23. The Benefits! ▸ We can use full, regular SQL for our Grafana queries in addition to InfluxQL ▹ Think JOINs, etc. ▸ We’re doing massive batch inserts, and MySQL’s query cache makes Grafana queries significantly faster ▹ This is obviously use case specific, but shows how you can tune MySQL to your specific use case 23