How we switched to columnar at SpendHQ

Editor's Notes

• #2: Thanks for coming! Hi I’m Allen Herrera. I’m an Engineer with SpendHQ, Most recently encharged with the migration from our prior database to MariaDB Columnstore in the later half of 2018. I’m excited and nervous to be here speaking, sharing the results of our journey as this is a first for me. So how we’ll do this is Ill start with a business level summary/ justification and then jump into a time lined story of our process ,challenges and results of switching to MariaDB Columnstore.
• #3: So lets start with some high level background information about SpendHQ! We are In the business of cleaning up client data and helping them identify savings opportunities from your data We do this through our sister consulting company ISG and our Data Analytic / Visualization web application It all starts with the client sending us raw data in any format they have. This includes excel files, csv files and more. We then take this data and consolidate, normalize it into a single schema . Part of this process includes normalizing vendor/company names Next the data is categorized against a custom taxonomy defined by the client. We then have internal experts review the results with the client in case further data processing is necessary All this to result in clean data being uploaded into our production web application for our clients to browse their data and drive conversations around potential savings opportunities. This final part , step 7, is where we’ve migrated from our old columnar database Infobright to MariaDB Columnstore.
• #4: So why change? At SpendHQ, we’ve been going through some massive growth that’s exposed scalability flaws with our legacy architecture. One of those was our database. Over the last two plus years as we’ve over doubled in size as a company but the data we get is 10 to 20 times greater than before. Naturally, as data grew, we realized we needed to address query performance and modernization.
• #5: So why specifically was our old database flawed. Simple our prior DB was old. It was based of MySQL 5.1 ( similar to infiniDB actually which Columstore was created out of but our old database stopped giving updates). This older version translates to slower performance compared to modern databases. Infobrights columnar db was single threaded which didnt help performance. We couldn’t clusterize it. We didn’t have access to Innodb tables for transactions, thus we were left with MyIsam. Furthermore we couldn’t scale vertically anymore to marginally improve performance either like in years past. All that said, when defining what to move to, these were our top priorities. Ease of transition, Scalability , Lower Cost and Support.
• #6: With that said, I’ve set the stage to take us back to December 2017 when we began considering other databases. Now when going through this, we didn’t plan on three sections but when looking backwards this is how I see. We had three phases, analyze, prepare and refine.
• #7: Step one of analyzing was to identify options
• #8: To do this, we engaged with pythian on a consulting engagement to identify and recommend a database that fit our needs. Taking into consideration our wants from a couple slides ago, (ease of transition, scalability , Cost and Support ) and our business model, they identified 7 column-oriented databases and recommended one. MariaDB Columnstore. By the way thank you John Shults for your work on this here.
• #9: MariaDB Columnstore met all our need to haves. Its ANSI SQL, apart from some special columnstore commands and intricacies, Its open source helping keep costs down and community support big There’s enterprise support for those who want it, which we at SpendHQ definitely take advantage of There’s Professional services to ramp up team education and to be a partner in any project. Plus MariaDB Columnstore is scalable and performant.
• #10: Next was to quantify what we aimed to accomplish by switching to MariaDB so we could sell Bussiness Folks that this is a good decision and so that we could measure our criteria of success.
• #11: Working with our data team of Robert Little and Dan Mackey, they identified roughly 25 problematic queries that we wanted to see improved. In our final report from pythian they estimated we could achieve a 71% reduction in query time by simply switching databases without significantly refactoring the queries or the schema. Management was blown away. 71% reduction in reads. That means 19 second queries in only 3 and half . Furthermore, if we were to refactor the schema to de-normalize the tables, we could achieve 95%. (blue is for our old db, red was mariadb columnstore, green is a denormalized tables in columnstore) So these became our goals.
• #12: It took us until March/April to begin actual work for the preparation of the migration. We hired professional services to come out and setup out first Columnstore instance so we could connect to it and do some minor performance tuning of the database.
• #13: Our consultant from MariaDB was Todd Stoffel. Thank you Todd, great and knowledgeable guy. He has a git repo that we used to easily install Columnstore using ansible. It auto tunes the configs to the hardware better than the Columnstore defaults. So it was a great place to start. However the challenges we faced were NOT on behalf of mariadb, but rather our legacy app.
• #14: Fail fail fail when making queries to the database. Let me quickly summarize the 3 challenges we had to overcome to keep moving forward.
• #15: Our framework’s ORM picked up the use of vtables on Columnstore and modified the data objects returns to include them. The root cause of the issue had to do with a specific php function returning the vtable value within our frameworks ORM that we had to wrap around custom logic tying values to tables they were queried from.
• #16: The next challenge was improper SQL. Somehow our prior database let queries like this above to execute.
• #17: The 3rd challenge was minor but revolved around case sensitive group bys relative to our old database.
• #18: Once we got past our application level challenges, we were ready to move from a standalone Columnstore instance to a cluster and actually benchmark performance. We brought Todd back in for a 2nd time to look at the cluster we had setup on our own and help drive optimizations.
• #19: The first thing we did was look at our hard drives with FIO testing. We identified our storage solution was HDD and had low IOPS speeds. Thus Todd recommend faster hard drives. After switching to SSDs we noticed better concurrency performance from the cluster as well as better individual performance. Next we looked at the configuration files The two key changes that yielded the best results were increasing both our innodb buffer size in the my.cnf and PmMaxMemorySmallSide from the columnstore.xml
• #20: After other minor adjustments, we went back to the original 25 ish queries and re-benchmarked them all again. Results were great!
• #21: Next we moved on to making cluster creation faster with some automation
• #22: [ start playing video then speak] Todd built the original version that we then modified for subsequent deployments. Here is a small video of a 1 um 3 pm setup.
• #23: Next we worked on the process to move the actual data from our old database to MariaDB Columnstore. At a high level what we chose to do was essentially export all our data as CSVs onto a hard drive, move it to MariaDB and import with cpimport.
• #24: To optimize this performance, we adjusted key sql variables and split CSVs for tables that were InnoDB. This took our migration time down from 35 hours to 8 hours as Innodb tables were the slow ones to insert. [Talk about slide numbers]
• #25: With the migration process in place, we were ready to test deploying production.
• #26: And it failed. The reason for it had to do with our hard drives. We didn’t have enough storage which was shocking to us. The other was a silly mistake of having iSCSI volumes in the fstab without having authenticated with the target first. Simple solutions, more storage and adding some logic to rc.local
• #27: Furthermore, once we felt ready to deploy again, we noticed performance issues in writes.
• #28: We then benchmarked 6 core data changing processes that write performance wasn’t good. So I sought out to refactor some of the critical write performance areas. This included the ETL.
• #29: Utilizing a new multi process architecture to take advantage of innodb row level locking. This resulted in our ETL being 7 times faster on the same hardware as Infobright, but also opened the door to more vertical scaling resulting in 25 times faster ETL uploads.
• #30: With that and 2 more refactors, we made a huge leap of performance.
• #31: Now we could deploy
• #32: The main issue we had when releasing was concurrency and our hard drives not working fast enough. When we asked our storage provider, they recommended we use multipath to open additional sessions from the server to the storage, opening up additional bandwidth.
• #33: So what do next!
• #34: Next to do is to de-normalize our schema but instead of trying to refactor our existing app, we’ll be starting a new app. We’ll also want to pilot a GPU database to see the results of the Brytlyt partnership with MariaDB. Overall we are happy with MariaDB Columnstore. Performance is great, the only issues we’ve come across with it are really just our own.
• #35: So to summarize Where we stand is great compared to where we were. Faster writes, significantly faster reads, significantly faster ETL. Storage needs took a hit and our application needed quite a bit of work given its age. Concurrency isn’t any better for us at the moment but can be solved if we utilize maxscale with additional UMs. Before concluding I want to give a special shout out to all the support we’ve received from MariaDB, specifically Todd Stoffel, Geoff Montee, David Hill plus more