How the Automation of a Benchmark Famework Keeps Pace with the Dev Cycle at InfluxData
0 likes100 views
The team at InfluxData needed to review benchmark data on InfluxDB during their development cycle to ensure any new changes continue to improve performance. However, using the existing benchmarking framework, InfluxDB-comparison, was manual and time consuming and not used consistently. To change this, InfluxData asked the team at Bonitoo to enhance the benchmark framework to easily incorporate new use cases, add new versions of the software quickly and easily and provide benchmark data on a cadence that works for the development (daily, weekly, monthly) cycle. In this webinar the team from Bonitoo will share how they were able to accomplish this as well as build automation into the existing framework. In addition, they will share the benchmark results generated from the framework that highlights how performant a time series database like InfluxDB is compared to the latest versions of products like MongoDB, Cassandra, Elasticsearch, and OpenTSDB.
![Few words about Time Series Database
A time series database is a software system that is optimized for handling time series
data, arrays indexed by time.
In relational DB world, time serie is often a table with Date field as primary key.
Relational DBs are not optimized for more complex use cases.
● High frequency of INSERT commands, >90% of all traffic
● Aggregated queries to compute metrics over specific time range
● Low or no occurrences of UPDATEs and DELETEs
● Communication over HTTP rather than [O/J]DBC
● Need flexibility in schema design as new devices insert data on the go](https://image.slidesharecdn.com/howtheautomationofabenchmarkframeworkkeepspacewiththedevcycleatinfluxdata2-180307190810/85/How-the-Automation-of-a-Benchmark-Famework-Keeps-Pace-with-the-Dev-Cycle-at-InfluxData-3-320.jpg)
![The Mission
Our journey started in November 2017… The mission was to address questions from
InfluxData as quickly as possible…
From [reactive] To [proactive]
Random Repeated in fixed intervals, cadency
Manual runs All automated to prevent from errors & variability
No coverage of non-InfluxData DBs Test all new releases asap
Various environment configurations and tests Constant comparable results
Pair of fixed hardware boxes Apply tests to various hardware combinations](https://image.slidesharecdn.com/howtheautomationofabenchmarkframeworkkeepspacewiththedevcycleatinfluxdata2-180307190810/85/How-the-Automation-of-a-Benchmark-Famework-Keeps-Pace-with-the-Dev-Cycle-at-InfluxData-5-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to How the Automation of a Benchmark Famework Keeps Pace with the Dev Cycle at InfluxData (20)
Ad
More from DevOps.com (20)
Ad
Recently uploaded (20)
How the Automation of a Benchmark Famework Keeps Pace with the Dev Cycle at InfluxData
- 1. How the automation of a benchmark framework keeps pace with the Dev cycle at InfluxData Vlastimil Hájek, Tomáš Klapka, Ivan Kudibal
- 2. Technology partner with InfluxData Experienced software engineers Asked to help with benchmarks Complete time series platform provider Modern engine for events and metrics InfluxDB database at the core About Companies InfluxData Bonitoo
- 3. Few words about Time Series Database A time series database is a software system that is optimized for handling time series data, arrays indexed by time. In relational DB world, time serie is often a table with Date field as primary key. Relational DBs are not optimized for more complex use cases. ● High frequency of INSERT commands, >90% of all traffic ● Aggregated queries to compute metrics over specific time range ● Low or no occurrences of UPDATEs and DELETEs ● Communication over HTTP rather than [O/J]DBC ● Need flexibility in schema design as new devices insert data on the go
- 4. Perform benchmarks for latest releases of InfluxDB and 4 other DBMSs Refresh the existing benchmarking methodology & fwk last used in 2016 Use public repo https://github.com/influxdata/influxdb-comparisons New releases during 2017 not covered by black box benchmarks The Beginning
- 5. The Mission Our journey started in November 2017… The mission was to address questions from InfluxData as quickly as possible… From [reactive] To [proactive] Random Repeated in fixed intervals, cadency Manual runs All automated to prevent from errors & variability No coverage of non-InfluxData DBs Test all new releases asap Various environment configurations and tests Constant comparable results Pair of fixed hardware boxes Apply tests to various hardware combinations
- 6. Compared performance of InfluxDB, ElasticSearch, Cassandra, MongoDB, OpenTSDB This selection of technologies was driven by typical use cases The dataset used generates “DevOps” data monitoring hosts in the network 1. Ingestion Rate 2. Query Performance 3. Efficiency in disk usage 4. CPU utilization Initial Methodology
- 7. DevOps Data Example What a system administrator would do when operating fleet of VMs
- 8. Automation
- 9. High Level Design Picture Use AWS in order to have flexibility in images Use Jenkins as Job Manager Use Ansible scripts for automated installation Store results in InfluxDB Visualize results in Chronograf and Google Spreadsheet Copy results to Influxdata performance DB for R&D
- 10. AWS Technology Layer No hw physical boxes at the office. No significant difference between physical and virtual. Same results. AWS provides various instance types giving CPU and memory options. Spot instances found better for batch tasks such as performance tests. EBS volumes charged by IOPS vs MongoDB HP HW:Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, 32vCPU (2x8cores, 2 threads per core), 32GB RAM, 300GB SCSI 15000 rpm AWS c4.4xlarge: Intel Xeon E5-2666 v3 2.9GHz, 16 vCPU, 30GB RAM, 1x EBS Provisioned 6000 IOPS SSD 120GB Traditional
- 11. Jenkins What it solved for us: ● Parallelization of multiple runs with different parameters ● Clean & functional UI ● Storing logs for every run Big number of combination of input parameters
- 12. Ansible Playbook Creates and removes instances in EC2 Manages DB installations (InfluxDB OSS, InfluxDB Enterprise, Cassandra, ElasticSearch, OpenTSDB, MongoDB) Manages various configurations Can use different cli tools Handles reporting “All-in-one” tool Easy-to-learn, self-documenting Easy Amazon EC2 management & provisioning Loves repetitive tasks Does not require agents installed at the endpoints Written in Python - community support, variety of additional modules, extendable modular system Reuse of existing roles Why Ansible?
- 13. Ansible solution Use EC2 for provisioning VMs Automate benchmarking cli tools written in Go Report measurements, system and disk usage, deals with different tags Contain ansible roles for DB installations, client machine and supporting tools Support parallel runs
- 14. InfluxData Platform Telegraf is installed on each database host to report hw utilisation InfluxDB stores benchmarks results, as well as hw utilisation metrics Chronograf to show results trend and hw utilisation by database engines Future improvement: Kapacitor to detect anomalies in results
- 15. Google Spreadsheets Why Google Spreadsheets? ● Visualisation apps like Grafana or Chronograf relies on data measure over time ● We need various variables on the X axis ● Flexible for further analysis ● Easily scriptable (JS) ● Publishable and linkable charts ● Smooth integration with InfluxDB (benchmarks results db) What it does? ● Checks new benchmark jobs on Jenkins server ● Downloads respective results from InfluxDB ● Creates tables with results ● Creates charts
- 16. Benefits
- 17. First Presented Results pre-flight InfluxDB 1.4 Elasticsearch 5.6.3 *) Cassandra 3.11.1 MongoDB 3.6 OpenTSDB 2.3.0 Ingestion rate / s 1,432,630 143,711 103,912 24,897 143,867 Size on disk 145 MB 540MB 2.3GB 16.7GB 1.23 GB Query rate / s 820 1,019 630 71.27 111 Peak CPU usage 500% 1,500% 500% 600% 500% Test set: Write/Query 100 servers, 100 values, 10s, 24h, 4 workers AWS c4.4xlarge: Xeon E5-2666 v3 2.9GHz, 16 vCPU, 32GB RAM, 1x EBS Provisioned IOPS SSD *) aggregation template
- 18. Scalable devops system Ops are automated. Engineers can focus on extending the use cases and methodology Follow new external releases (eg. MongoDB) New blackbox tests on top of R&D nightly builds Improved reporting in benchmarking tools 13 hrs to run all 25 result sets (manually impossible), including clustered environments Before Random tests, low automation Engineers locked in operations of benchmark tests Low capacity to keep pace with the new releases of competitors Manual testing, error prone Lack of visibility 6 weeks to test one result set After
- 19. Financial Benefits of Used Tech. Infrastructure Traditional physical hw: Typical price of 16 vCPU, 32GB RAM box is $11 per day. Tests need at least 2 machines: daily costs start at $22. Regardless of real usage. Automated benchmark system runs complete InfluxDB nightly test sets on ~50 m4 spot instances in parallel at a price of < $10 a day, in all single and clustered combinations. No run, no charges.