Using SAS GRID v 9 with Isilon F810

Editor's Notes

• #2: Hello, my name is Boni Bruno, Chief Solutions Architect for Dell Technologies. I focus on analytics solutions for our UDS products and have developed various collateral around using our storage products with various technologies like Hadoop, Spark, Kafka, ML, running analytics on Isilon in Google Cloud, etc. I’ve been working extensively on testing SAS GRID with our All-Flash Isilon Systems, specifically our Isilon F800 and F810 models. I recently gave a tech jam session on running SAS GRID with our All-Flash Isilon F800/F810 models with great interest and feedback. I’ve been asked to do a technical deep dive on my testing so that’s exactly what I will be covering in this presentation. With that said, let’s dive right into the presentation.
• #3: SAS has been around for over 40 years with an amazing history and growth as a company, not just financially speak, they also provide a comprehensive suite of analytics products covering business intelligence, advance analytics, data management, predictive analysis and more. It’s important to understand that SAS is not a relational database, rather SAS provides an interpretive programming language and stores data in proprietary SAS formatted files. SAS also provides native access to a variety of databases as well as big data platforms like Hadoop. Nearly every enterprise customer we have is using SAS in one form or another so many of you will likely be engaged to present why Isilon is a good fit for SAS. I highlight why we are a good fit as we progress through this presentation.
• #4: So why consider Dell EMC for SAS analytics. At a high level, Dell EMC has provided infrastructure solutions for many SAS customers already and we know our solutions work well, we are also fortunate to hold the #1 market position in converged infrastructure, virtualized data center infrastructure, and both traditional and all-flash storage, this is based IDC reports. Dell EMC makes numerous storage solutions that has worked well with SAS for example our VMAX and PowerMax products and well as XIO, or VxFlex have been deployed with SAS in the field, but lately we’ve seen customers looking to our scale-out Isilon NAS products to house SAS data. This is the primary reason for me doing a formal performance validation for SAS GRID with Isilon. [CLICK] The validation is did focuses on our Isilon all-flash storage systems and why using Isilon all-flash systems with SAS for data storage make sense. I’ll cover design considerations, performance numbers, and some new features introduced with our F810 model and how these new features can benefit SAS customers.
• #5: The F810 model is the latest model we have in the F800 series. I’m exciting to say this model has produced some excellent performance results with SAS GRID. Those of you not familiar with our F800 line. These are the all-flash models. All of our F800 series models are 4U in size, you can equip them with 3.8 TB, 7.7TB, or 15.4TB SSD drives. This translates into SAS customers being able to get just under a 1 PB of data storage in a 4U form factor when using the 15.4TB SSD drives. As with all of our Isilon models, this is a true SCALE-OUT solution providing SAS customers an easy ability to add storage nodes to support more capacity and performance as needed. What’s unique about this F810 model specifically is that this model comes with a HW acceleration FPGA card that provides in-line data compression. This is a key value proposition to SAS customers as this significantly saves on storage space and increases I/O performance. Again, ,we will get in the details in the up coming slides.
• #6: Before we dive into the tested architecture, it’s important to understand the criticality of storage for Analytics and related workloads. Clearly the massive amounts of data to meet business needs is growing daily in many cases. This has lead our Isilon business unit into developing enhanced compression techniques with our newer products as well as the need to make them more scalable and higher performing than ever before. Isilon clusters can now grow to 244 nodes in a single cluster with a single name space, truly amazing. Even if a lot of analytics is done in RAM, customers are having to store more and more data as time goes on to shared storage. BTW – The F810 model I mentioned early now has a new feature to allow our customers the ability to dedup data as needed. I will get into the dedup results later in the presentation.
• #7: So let’s talk about SAS GRID. SAS provides a lot of products as mentioned earlier. For our testing, we specifically wanted to test SAS GRID. A typical SAS GRID environment has users that run SAS desktop clients or thin clients or clients can simply ssh into the SAS grid to submit various jobs. The SAS GRID Resource Manager distributes these jobs across the numerous grid nodes in the SAS GRID network. While these jobs are running, there is a lot of I/O generated for the creation of temporary and staging files as well as I/O going to and from the shared storage environment. It’s important to understand that SAS refers to this temp/staging environment as the SAS WORK environment and the shared storage environment as the SAS DATA Environment. An important design consideration and best practice to strictly adhere to is that SAS WORK should always be fast local block storage only, Isilon should never be used for SAS WORK, rather use Isilon for SAS DATA only. If any of you have seen my presentations on using Hadoop with Isilon, putting SAS WORK on Isilon is equivalent to putting Hadoop SCRATCH SPACE on ISILON, you simply never want to do it.
• #8: In speaking with D4T4, they recommended not using the Load Sharing Facility for our test lab and performance testing. LSF is not good when you want to control the job spread and ensure repeatable job launches. Not using LSF is a common practice for validation and I/O performance testing as we did in our SAS GRID/Isilon test lab.
• #9: With that being said, I can now discuss the test lab systems and network I built for SAS GRID and Isilon. The specific SAS GRID software version tested is version 9.4M6 with both our Isilon F800 and F810 models. Each SAS Grid Node has 40 cores and 256GB RAM with dual 10GbE connections to the network. Each Isilon node is connected 40GbE to the access network and the private Isilon backend network is also 40GbE. Note: For 40 cores, you really need 25GbE connections, but I digress. Testing ranged from using a single SAS compute node in the GRID to scaling up to12 SAS compute nodes in the GRID. The backend Isilon system stayed as a single 4-node chassis as SAS GRID compute nodes increased from 1 to 12.
• #10: Before we dive into the tested architecture, it’s important to understand the criticality of storage for Analytics and related workloads. Clearly the massive amounts of data to meet business needs is growing daily in many cases. This has lead our Isilon business unit into developing enhanced compression techniques with our newer products as well as the need to make them more scalable and higher performing than ever before. Isilon clusters can now grow to 244 nodes in a single cluster with a single name space, truly amazing. Even if a lot of analytics is done in RAM, customers are having to store more and more data as time goes on to shared storage. BTW – The F810 model I mentioned early now has a new feature to allow our customers the ability to dedup data as needed. I will get into the dedup results later in the presentation.
• #11: Before we dive into the tested architecture, it’s important to understand the criticality of storage for Analytics and related workloads. Clearly the massive amounts of data to meet business needs is growing daily in many cases. This has lead our Isilon business unit into developing enhanced compression techniques with our newer products as well as the need to make them more scalable and higher performing than ever before. Isilon clusters can now grow to 244 nodes in a single cluster with a single name space, truly amazing. Even if a lot of analytics is done in RAM, customers are having to store more and more data as time goes on to shared storage. BTW – The F810 model I mentioned early now has a new feature to allow our customers the ability to dedup data as needed. I will get into the dedup results later in the presentation.
• #12: The testing focus areas for this lab environment are as follows: 1. We wanted to see how well the F800’s performed with a multi-user mix workload. I’ll talk about the workload in the next slide. We also wanted to understand the value of using the new in-line compression capabilities that comes with the newer F810 model. 2. Historically speaking, most NFS clusters are deployed with SAS using a ratio of 1 SAS compute node to one NFS storage node. We wanted to see if we can increase this ratio to 2 to 1 or even 3 to 1 using the F810 without decreasing job runtimes as SAS compute nodes increased. 3. SAS offers software compression, we wanted to see what happens when you add Isilon’s HW compression and the benefits in performance and space savings. 4. We also wanted to see the effectiveness of Isilon’s new dedup feature. Lastly, if Isilon performs well in these four areas of focus then the overall TCO will be better for our SAS customers and we want happy SAS customers.
• #13: Now I had the option to just do a basic SAS benchmark with Isilon, but instead I decided to engage our go to SAS partner D4t4. D4T4 has ex-SAS employees on staff with over 20 years of SAS experience. They developed an comprehensive multiuser analytics workload representative of various jobs typically run by our financial services customers. In working with the senior SAS architects at D4T4, we were able to simulate a lot of users submitting real-world mix workloads to stress test the storage I/O environment which is a top concern for many SAS customers. I’ll get into more details in upcoming slides.
• #14: At a high level, SAS requires certain I/O performance for SAS GRID. Specifically SAS GRID Sizing guidelines specify a total I/O per CPU core to be in the range of 100-150 MBps. This is divided among the data sources and targets, namely SAS WORK, SAS DATA, and other network connections pertaining to database connectivity, streams, etc. The DATA on DISK, also referred to as SAS DATA, shown here in purple is were Isilon fits in. Many of our customers may have petabytes of SAS DATA consisting of long term project tables and storage performance and scalability is vital. SAS DATA represents 40-50% of the overall I/O requirement on average for SAS. As I mentioned earlier, SAS WORK should never be on Isilon, SAS WORK represents 40-60% of the overall I/O and typically will leverage local NVMe or high speed fiber connected storage. The other network traffic makes up the rest of the I/O percentage and typically is in the 10-20% range. The key thing to remember here is that SAS wants around 150MBps of sustained read and write throughput per core. I’ll get into what we were able to sustain with our deployed SAS GRID using Isilon shortly.
• #15: As far as running the D4T4 workload, it was easy running batches of the workload on each node as we scaled up the number of SAS nodes. Each batch was executed on each grid SAS compute node and the results were recorded to determine if repeatable and predictable I/O throughput can be achieved with Isilon. You we see in upcoming slides we actually achieved that with no problem. .
• #16: Each batch consisted of 33 SAS programs and each batch had 1.3 TB of uncompressed SAS input dataset. We didn’t create an RDBMS which is normal, but by not having an RDBMS which would typically offload some of the I/O away from Isilon, this method actually put more I/O load on Isilon which is good and is a point in our favor. The 33 jobs were launched through a script. As jobs run it uses that scratch area called SAS WORK for temp storage on the PowerEdge 730’s for combines, sorts, merges data that comes from Isilon to local SAS WORK and the output data goes back to Isilon. This is pretty normal for bank environments were users pull data from various sources and work heaving in the SAS work environment then put it back on the permanent storage which is perfect for our scale-out all-flash Isilon nodes. For those of you familiar with SAS, the 33 jobs simulate everything from a modeler to a report user that comes in and out over a period of time to someone doing an ETL inbound data build with an analytics table, the code does some sorts, merges, and other common things you find with SAS analytics.
• #17: The workload manipulates data to running logistic regressions, jobs blows through files, merges, and sorts, a majority of the jobs was sequential but some of the jobs did some random reads and writes from Isilon. BTW, the data generation was patterned after a SAS modeler that works at financial services organization. Again, many kudos to D4T4 for providing this dataset and workload scripts, it made the testing much more comprehensive and representative of actual SAS production environments. We have a joint webinar coming up on May 19th, our marketing teams should be sending out registration links to that event next week. So keep a lookout for that.
• #18: This slide shows how the jobs were launched. The key thing I want to point out here is that when SAS read and writes data, it uses predefined block sizes. On average using a block size of 64K, 128K, or 256K is typical in production environments as databases aren’t streaming large amounts of data. So unlike Hadoop workloads where Isilon is configured to use 128MB or 256MB block sizes over HDFS typically, with SAS 128KB or 256KB block sizes is much more common with NFS.
• #19: SAS GRID nodes typically have from 8 to 12 CPU cores when with dual 10 GbE configurations. So going by SAS recommended guidelines of 150MBps/CPU core, a 12 core node needs a total of 1.8Gps of sustained I/O throughput. Recall that ~50% of the I/O comes from SAS WORK, ~40% of the I/O comes from SAS DATA, i.e. Isilon, and the remainder of the I/O comes from database connections. Based on that, a 12 node SAS Grid with 12 CPU cores each would need a aggregate sustained R+W throughput of ~GBps from Isilon or ~720 MBps of sustained read and write throughput per SAS node from Isilon. [CLICK] Isilon was able to maintain 9GBps with our deployed 12 node SAS GRID with the small block sizes. When D4T4 saw this they were very happy. They have sold H500’s in the past to SAS customers. Based on these results, D4T4 is not moving to looking to use F810’s with H5600’s as a recommended storage architecture for SAS customers moving forward. This is great news coming from SAS experts who live and breathe SAS analytics day in and day out. Note: The CPU utilization on Isilon never exceeded 70% during all this testing. There is still room to grow, but I recommend not going beyond a 3 SAS compute to 1 Isilon node ratio.
• #20: This slide talks about the testing methodology. I’m writing a white paper with Tom Keefer from D4T4 covering all our SAS Grid testing and findings with using Isilon for SAS DATA. The white paper will be available by the end of this month.
• #21: This is really an exciting slide. Before I get into the results, it’s important to note that SAS users or analytics people in general don’t care about Gigabytes or I/O throughput, they just care about the time it takes to run their SAS jobs. It’s funny that In many cases they don’t even know the size of their datasets, rather they know how many billions of records are in a table or how wide their tables are. Keeping that in mind, this slide covers the response time for a batch run of 33 jobs and how the runtimes varied with SAS DATA being on an Isilon F800 with no SAS compression, an F800 with SAS compression, and our newer F810 model that provides HW compression along with the SAS compression. SAS compression is software based and customers will typically have this turned on. The software compressions does but a little more load on the compute node but it also sends less traffic over the network which benefits shared storage solutions like Isilon. When you add in the hw compression capabilities of the F810, you can see the sum of all the runtimes decreased from 12 hours and 10 min with SAS SW based compression to 7 hours 21min when using SAS SW compression with the F810 HW compression. That’s a 40% decrease which is fantastic! From an individual user perspective, the average individual job runtime with SAS compression was 22min, this when down to 13min when using the F810, again a 40% decrease in runtime. Our SAS partner D4T4 are very happy with these results.
• #22: Our next focus area is scalability. SAS GRID customers shrink and grow their GRID SIZES all the time, when you grow the SAS GRID at peak times to deal with end of month jobs and the like what’s critical is to get predictable runtimes as you scale. [CLICK] If you look at the 3rd column in this table, you can see we grew the grid size from 1 to 2 to 4 to 8 and finally to 12 SAS GRID nodes. [CLICK] Correspondingly the mix workload increased from 33 jobs to 66, to 132, to 264, and finally 396 simultaneous jobs. As we scaled the SAS nodes and aggregate job count, we recorded both the [CLICK] average job runtimes and [CLICK] max job runtimes. We never increased the Isilon node count during testing. Historically speaking, we typically recommend having a ratio of one Isilon storage node to one SAS compute node. This is typical with our H500 models that we have deployed in the field. [CLICK] However, with the F810 results , for the first time ever, I can confidently say the F810 easily breaks the 1 to 1 ratio barrier. As the results show, the run times stayed consistent as we scaled from 33 jobs running on 1 SAS node to 396 jobs running across 12 SAS nodes in the grid while using just a single F810 4-node chassis. Absolutely Beautiful! Our SAS integration partner is now looking to standardize on F810 models for SAS deployments in the financial services sector.
• #23: During the 12 node SAS GRID workload testing I took Isilon statistics at different points of the testing to make sure the I/O distribution on Isilon stayed even. [CLICK] Here you can see a nice even distribution across the 4 nodes in the F810 chassis. [CLICK] This remained consistent. [CLICK] Throughout the various batch job runs. [CLICK] What was very interesting was the fact that the CPU utilization peaked at only 68% under the 12 node testing with 365 simultaneous jobs running. This means the single f810 still have room to support more load. Again, very pleased with these results!
• #24: Using NMON we can see the I/O throughput for both SAS WORK and SAS DATA on individual nodes as we scaled up the SAS nodes. The chart on the left shows the I/O throughput on SAS node 2 when the GRID size had 2 nodes with a single batch run. NMON shows both the NFS traffic and the local I/O traffic. You can see that the mix workload generates a lot of I/O traffic for both SAS WORK and SAS DATA. The chart on the right shows the I/O throughput on the same SAS node 2 when the GRID size was increased to 12 nodes. What is nice here is that both NMON graphs show similar I/O patterns which mean consistent i/o throughput, if the I/O subsystem had problems you would see longer run times, weird I/O wait times, but that wasn’t the case, we had a good balanced system here with consistent i/o patterns.
• #25: This slide shows the CPU utilization on a sas node with a single batch run when using the F800 with no compression, when using the F800 with SAS compression, and when using the SAS compression combined with the HW compression of the F810. The patterns are similar but notice how the runtimes are better with the f810. There is some wait time shown and that’s because of SAS WORK and the 10GbE nics, the local drives were spinning disks and not NVMe drives and 40 core systems really should be using 25GbE nics, but overall the value of the F810 can easily be observed here which is good.
• #26: This slide just provides more technical evidence that both the F800 and the F810 scale well as the SAS nodes increase , in this case we are highlighting the results of a specific Bank2 job. For these kind of workloads, the HW compression of the F810 clearly makes a difference.
• #27: All the input data for the workload was ~ 1.4TB uncompressed. Most SAS customers will have SAS SW compression turn on so the input dropped down to ½ a TB with SAS SW compression as a lot of analytic tables compress really well. But when you add the HW compression of the F810, the input data was further reduced to 149GB providing roughly a 3:1 compression ratio over using just SAS SW compression which is very good. You have the ability with SAS to turn on SW compression on and off and it’s common to experiment with compression when doing say ETL jobs. Again, a lot of analytic tables are very repetitive and compress well, whether you get 2 to 1 or 3 to 1 or 4 to 1 will vary from customer to customer, but overall we are very pleased with these results.
• #28: Here we show details of the compression results on the output data from the job runs highlighted here in the last column of the table, due to running several merges and sorts on the data, even with SAS compression turn on, the output data grew to 748GB, but when adding the HW compression of the F810, the output data was significantly lower at 119GB, that an 84% reduction in output data. This was checked three times as this was really impressive. Again, results will vary from SAS customer to SAS customer, but this is very promising.
• #29: One thing I noticed is that SAS SW compression can sometimes increase runtimes of some jobs. If you notice this with some of your jobs, just turn off SAS SW compression for those specific jobs. Note: This has nothing to do with Isilon, this is just a SAS thing and SAS is aware of it, again the good news is that you can turn SAS compression off on individual jobs.
• #30: The F810 also includes the ability to run dedup on the filesystem. In cases where you want to save even more space, you can run a dedup assessment to give you an idea o the space savings you can potential obtain. The left side of this slide shows an example output of the dedup assessment job and the right side shows the results of an actual dedup job run on Isilon OneFS. I just chose a sample SAS node to see the impact of running dedup, if you look at the top right upper corner of this slide you can see that prior to running dedup, the multi-user directory used 4.6TB of space, after the dedup, this went down to 2.7TB which is a reduction of 41%.
• #31: In summary, we are very pleased with the results of our SAS GRID performance testing with the F810. For the first time ever, we were able to observe space savings using compression on average of 3 to 1 and performance gains that allow us to support 3 SAS compute nodes to 1 Isilon storage node. Considering many banks may have 100’s of nodes, this can provide significant cost savings with respect to storage costs. And lastly, the F810 deduplication feature can potentially save an additional 20-40% in storage space further decreasing storage costs.
• #32: That concludes the deep dive session. I’m currently working with D4T4 on the publication of a whitepaper based on this work. This will be available by the end of May 2020. Thank you.