What it takes to run Hadoop at Scale: Yahoo! Perspectives

Editor's Notes

• #2: (30 secs) T: 30 secs Let’s get started. I always wanted to document and talk about some of the top considerations that go into making Hadoop a scalable platform for the entire business, and this was a perfect opportunity to do so.
• #3: (30 secs) T: 1 min My name is Sumeet Singh, and I am a Sr. Director of Products for Hadoop and Big Data platforms at Yahoo. I have been working at Yahoo for nearly three and a half / four years now, and had few different roles in my time there. The top 10 list I am presenting is what I came up with. You may come up with a different list, or think of other things that may be more important for your businesses. Let me set some context first on Hadoop at Yahoo, and then we will jump right into it.
• #4: (1 min) T: 2 min Hadoop is a secure shared platform at Yahoo, lot of people call it the public grid, but it’s a private cloud. Yahoo products and properties across devices generate a lot of data that is of immense value to us in driving new and interesting use experiences across devices. All that data comes to Hadoop that acts as a single source of truth for all data at Yahoo. A wide variety of other data also gets pulled into the Hadoop Grid from various sources as shown. The idea is to consolidated data from all over was the company from disparate sources at once place so that it can be (a) shared (b) enriched (c) de-duped (d) kept up to date. That data once processed is applied back or served as value to our consumers in the form of personalized experiences across our products and properties. And of course is used for reporting and analytics. All of this is done while keeping the web scale economics and cost in mind.
• #5: (30 secs) T: 2 min 30 secs With improving platform capabilities and robustness, the workload over the years have systematically moved from research into production and revenue systems. The use cases running on platform have continued to grow and we reached a point quickly where we started to depend on it. The growth is evident from the storage we have added at roughly 20-25% CAGR in the last five years.
• #6: (1 min) T: 3 min 30 secs Ok, so how did we get here? In the rest of the talk, I want to walk you through some of the top considerations that go into making the Hadoop a scalable platform to run your entire business on. First one, of course is where you run and how you should think about it (2) Second is the cost considerations and knowing who and what costs how much (3) Third, of course is how to choose and think about hardware configurations (4) Network is a big piece of the puzzle, and we will talk about that (5) Software stack that you want to run on your platform (6) Security is of course a big area, we will not get too detailed into any one of these (7) Data movement or lifecycle management, including BCP provisions (8) Then, we will get into audit and governance (9) How well Hadoop works with other/ external systems (10) And, finally, talking about some of the issues or non-issues when you make architectural choices on the platform.
• #7: (1 min) T: 4 min 30 secs Perhaps the first consideration would be where you want to run your workloads, on an on-premise infrastructure or public cloud. With on-premise, you can go with two different deployment models, (1) private cluster that you would often setup for large demanding use cases, newer technology or for regulatory reasons, (2) and then there is shared grid which is source of truth for most data at the company. Public cloud models also come in two flavors. (1) You acquire hosted compute clusters and run Hadoop on them mostly for setup time or cost reasons or if your data is already generated and stored in public cloud. (2) Purpose-built big data clouds are also becoming popular with more intense out of the box integration among offered stack components
• #8: (2 min) T: 6 min 30 secs There is no right or wrong answer when you are thinking about which model to go after. Depends on your particular situation, your workloads, your plans for the future, adjacencies and other investments in cloud etc. Let’s consider some of the aspects that makes you sway one way or the other. (1) Cost – On-premise requires investments in setup and maintenance, public cloud has a convenience aspect to it as well. (2) Data – You need to think about data movement and data serving. One of the most important aspects to consider (3) SLA – There are multiple mechanism to make sure a variety of SLAs are covered, not just uptime (4) Tech Stack – You can deploy what you need at both places, but you have better control over the stack and integration with other aspects of cloud, such as monitoring, serving stack (in our case cloud/ data serving, ad serving, and content serving) (5) Security – one of the deepest topic you need to think about. (6) Security and multi-tenancy is hard. If your plans are to scale up your infrastructure, you need to think about it. Company policies also impacts your software and operations.
• #9: (30 secs) T: 7 min Cost and SLAs (which is price and performance) is something you can evaluate quantitatively, but the other factors require careful consideration and choices. A single factor can sway you in one direction or the other.
• #10: (1 min) T: 8 min So when does an on-premise infrastructure makes sense. When you amortize your investments over a higher usage of resources. In a nutshell, as you benchmark, you will realize that as utilization/ consumption increase, eventually, you will start to look fairly attractive compared with a public cloud model. Do not stop at the cost, conduct a sensitivity analysis based on your expected/ target utilization before you make a decision.
• #11: (2 min) T: 10 min Knowing your true cost of operating a platform is not only important for metering but also giving projects running on the platform a good idea of their ROI and profitability. This is perhaps one of the most difficult exercises as it requires running around to gather data which is often hard to get. Nonetheless, this is the first step and required. We will walk our way up. Hardware – not just the namenodes and datanodes, but all kinds of other things involved. Also, most hadoop servers have compute and jbods, so remember this that is all things hardware. R&D HC – this is not for writing or developing apps on top of hadoop, this is HC needed for platform development Ops Cost – power is one of the biggest elements here. Network – can be substantial depending on the backplane capacity and architectural choices Acq./ Install – one time costs Ops – this is where Yahoo excels. Number of people needed to maintain the infra. Network bandwidth – can be significant depending on your operation. As you understand your total cost of ownership, you can get to a monthly figure. Remember to properly depreciate assets and bring it to a monthly run rate (opex + capex).
• #12: (2 min) T: 12 min Once you have your monthly TCO, it is time to understand your unit costs. So, let’s talk about resources you are going to consume first. Compute (Memory) is for the YARN containers where your Map or Reduce tasks run. The unit is $/GB-hr/month. You need to know your monthly compute TCO and total compute capacity Compute (CPU) is for YARN containers in terms of CPU vCores. 10 vCores per hyper threaded core or 20 vCores per physical CPU core. Storage – usable storage space instead of raw storage space. Bandwidth – calculated at the cross-colo link level, but you get your monthly bill based on your portion of data in and out. So, your unit cost is $/(data in+out). Namespace – while we do not cost it out separately for the namenode, it is important to manage it. We have allocations/ quota, simply the sum of all the files and directories.
• #13: (2 min) T: 14 min Remember the monthly TCO we calculated earlier. I am splitting my monthly TCO in 30/30/40 among compute memory, compute CPU and storage. You can choose based on your situation. If you buy them separately, then you already know the split. My total slots or memory in the infrastructure is summed up and converted to hours. You need the notion of time as you will associate the units to usage by the users. So, total cost/ total capacity (in slot hours) or in usable storage (good so that you can benchmark effectively should you go to cloud systems that charge for usable storage of data, not raw storage capacity). And similarly for the bandwidth.
• #14: (30 secs) T: 15 min There are only four resources that are consumed in Hadoop operations when it comes to hardware. Memory, CPU, Storage/ Disk, and Network. We will talk about networks within datacenter at length as our fourth consideration. The best way to think of these resources as configurable when it comes to deciding the type of server hardware you want to setup for hadoop. So, combining these resources into a configuration you can certify for performance evaluation.
• #15: (1 min) T: 16 min Here are some of the possible combinations, as you can see, even today, our clusters have generations of memory and CPU as well as disk drives that are configured as JBODs giving us well over 10 configurations live in production. Beyond recognizing the configuration for overall pool, we let the framework make decisions when managing a cluster with heterogeneous configuration. As far as storage is concerned, Hadoop has support for multiple types of storage, although we are chugging along find with disk drives. We are now also composing clusters with large memory boxes and GPUs particularly for Machine Learning and let the applications choose those configurations for specific needs based on assigned labels to them.
• #16: (2 min) T: 18 min Beyond servers, network is another important aspect of the overall setup. We host all shared compute clusters on a dedicated backplane. Within a datacenter, bandwidth between all compute racks is consistent, independent of the clusters. This allows for inter-cluster access with similar transfer rates as intra- cluster racks. Nodes (racks) can be moved between clusters without network reconfiguration or physical moves.
• #17: (1 min) T: 19 min Big Data Applications can be demanding on the network as they often present many-to-one traffic flows, otherwise known as in-cast. For e.g. you may be joining or accessing data between two Hadoop clusters. Large data extractions between HBase and Hadoop is common for web applications such as in search. Between Storm and HBase where incremental processing works great, fast bulk updates become an issue. Similarly when moving data between Hadoop and storm clusters. We have seen saturations, although mostly TOR switches before overwhelming anything else.
• #18: (2 min) T: 21 min One way to compose your network among racks is what we called as through BAS switches a.k.a. big ass switches or big box fabric design. (core – distribution – access layers). Some call it aggregation and or core. The host connect is 1G to TOR. Switch uplink is 10G. Minimum all-to-all guarantee across 15,000+ servers this backplane can support is 500Gpbs with a 2:1 oversubscription. Oversubscription is the ratio of contention should all devices send traffic at the same time. (1G copper – 8way – 200Mbps, 10G Fiber – 2-way – 500Mbps). This typical datacenter design can get inflexible to scale and $/server for 10G may be expensive. Also, given that nodes in Hadoop have IP addresses, you rely on L3 or routing protocols and STP that can again get inflexible and hard.
• #19: (2 min) T: 23 min A new leaf and spine based architecture on old circuit switched networks (1950s) is becoming popular now for better latency and $/server cost. Now, the RSW can be 48x10G, with 4x40G uplinks. The 4x40G uplinks can be broken down into 16x10G (upto 160G). The number of VCs you construct defines the uplinks, for e.g., 2 VCs with 2x40G. The composition of the VCs define the TORs you can have. Each leaf here can support 16 TOR switches giving you 20,000+ (= 16 * 32 = 512 == 20,480 hosts) hosts that can be connected. Each leaf supports 16 TORs. So, total TOR Each VC is 40 G, 2 VCs. – 80G. 5:1 == 400G / 80G, to decrease oversubscription build more VCs. Number of leaf is always 2x the number of spines for non-blocking. ½ to spine and ½ to racks. Each TOR has 4 x 10G up.
• #20: (1 min) T: 24 min
• #21: (1 min) T: 25 min
• #22: (1 min) T: 26 min
• #23: (1 min) T: 27 min User accounts in LDAP and netgroups for access control. Automated jobs/workflows via headless accounts. Two Kerberos REALM with one-way trusts - Active Directory for individual users and Hadoop specific REALM for headless users and service principals. The architecture is SOX compliant. Kerberos provides Generic Security Service APIs for RPC auth, and Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) mechanism is used for web UIs for HTTP auth.
• #24: (1 min) T: 28 min
• #25: (1 min) T: 29 min
• #26: (2 min) T: 31 min Each site has a corresponding prod / non-prod cluster for user applications to failover to. Feeds are kept in sync using replication. Applications develop BCP strategy using native Hadoop services (e.g. async replication)
• #27: (1 min) T: 32
• #28: 1 min T: 33 min
• #29: 1 min T: 34 min
• #30: 1 min T: 35 mins
• #31: (5 mins) T: 40 mins