Plugging the Holes: Security and Compatability in Hadoop
3 likes1,699 views
My presentation from Hadoop World in NYC on 2 October 2009. I covered the plans for adding security into Hadoop in 0.22.
Ad
More Related Content
What's hot (19)
Viewers also liked (20)
Ad
Similar to Plugging the Holes: Security and Compatability in Hadoop (20)
Ad
More from Owen O'Malley (9)
Recently uploaded (20)
Plugging the Holes: Security and Compatability in Hadoop
- 1. Plugging the Holes: Security and Compatibility Owen O’Malley Yahoo! Hadoop Team [email protected]
- 2. Who Am I? • Software Architect working on Hadoop since Jan 2006 – Before Hadoop worked on Yahoo Search’s WebMap – My first patch on Hadoop was Nutch-197 – First Yahoo Hadoop committer – Most prolific contributor to Hadoop (by patch count) – Won the 2008 1TB and 2009 Minute and 100TB Sort Benchmarks • Apache VP of Hadoop – Chair of the Hadoop Project Management Committee – Quarterly reports on the state of Hadoop for Apache Board Hadoop World NYC - 2009
- 3. What are the Problems? • Our shared clusters increase: – Developer and operations productivity – Hardware utilization – Access to data • Yahoo! wants to put customer and financial data on our Hadoop clusters. – Great for providing access to all of the parts of Yahoo! – Need to make sure that only the authorized people have access. • Rolling out new versions of Hadoop is painful – Clients need to change and recompile their code Hadoop World NYC - 2009
- 4. Hadoop Security • Currently, the Hadoop servers trust the users to declare who they are. – It is very easy to spoof, especially with open source. – For private clusters, we will leave non-security as option • We need to ensure that users are who they claim to be. • All access to HDFS (and therefore MapReduce) must be authenticated. • The standard distributed authentication service is Kerberos (including ActiveDirectory). • User code isn’t affected, since the security happens in the RPC layer. Hadoop World NYC - 2009
- 5. HDFS Security • Hadoop security is grounded in HDFS security. – Other services such as MapReduce store their state in HDFS. • Use of Kerberos allows a single sign on where the Hadoop commands pick up and use the user’s tickets. • The framework authenticates the user to the Name Node using Kerberos before any operations. • The Name Node is also authenticated to the user. • Client can request an HDFS Access Token to get access later without going through Kerberos again. – Prevents authorization storms as MapReduce jobs launch! Hadoop World NYC - 2009
- 6. Accessing a File • User uses Kerberos (or a HDFS Access Token) to authenticate to the Name Node. • They request to open a file X. • If they have permission to file X, the Name Node returns a token for reading the blocks of X. • The user uses these tokens when communicating with the Data Nodes to show they have access. • There are also tokens for writing blocks when the file is being created. Hadoop World NYC - 2009
- 7. MapReduce Security • Framework authenticates user to Job Tracker before they can submit, modify, or kill jobs. • The Job Tracker authenticates itself to the user. • Job’s logs (including stdout) are only visible to the user. • Map and Reduce tasks actually run as the user. • Tasks’ working directories are protected from others. • The Job Tracker’s system directory is no longer readable and writable by everyone. • Only the reduce tasks can get the map outputs. Hadoop World NYC - 2009
- 8. Interactions with HDFS • MapReduce jobs need to read and write HDFS files as the user. • Currently, we store the user name in the job. • With security enabled, we will store HDFS Access Tokens in the job. • The job needs a token for each HDFS cluster. • The tokens will be renewed by the Job Tracker so they don’t expire for long running jobs. • When the job completes, the tokens will be cancelled. Hadoop World NYC - 2009
- 9. Interactions with Higher Layers • Yahoo uses a workflow manager named Oozie to submits MapReduce jobs on behalf of the user. • We could store the user’s credentials with a modifier (oom/oozie) in Oozie to access Hadoop as the user. • Or we could create Token granting Tokens for HDFS and MapReduce and store those in Oozie. • In either case, such proxies are a potential source of security problems, since they are storing large number of user’s access credentials. Hadoop World NYC - 2009
- 10. Web UIs • Hadoop and especially MapReduce make heavy use of the Web Uis. • These need to be authenticated also… • Fortunately, there is a standard solution for Kerberos and HTTP, named SPNEGO. • SPNEGO is supported by all of the major browsers. • All of the servlets will use SPNEGO to authenticate the user and enforce permissions appropriately. Hadoop World NYC - 2009
- 11. Remaining Security Issues • We are not encrypting on the wire. – It will be possible within the framework, but not in 0.22. • We are not encrypting on disk. – For either HDFS or MapReduce. • Encryption is expensive in terms of CPU and IO speed. • Our current threat model is that the attacker has access to a user account, but not root. – They can’t sniff the packets on the network. Hadoop World NYC - 2009
- 12. Backwards Compatibility • API • Protocols • File Formats • Configuration Hadoop World NYC - 2009
- 13. API Compatibility • Need to mark APIs with – Audience: Public, Limited Private, Private – Stability: Stable, Evolving, Unstable @InterfaceAudience.Public @InterfaceStability.Stable public class Xxxx {…} – Developers need to ensure that 0.22 is backwards compatible with 0.21 • Defined new APIs designed to be future-proof: – MapReduce – Context objects in org.apache.hadoop.mapreduce – HDFS – FileContext in org.apache.hadoop.fs Hadoop World NYC - 2009
- 14. Protocol Compatibility • Currently all clients of a server must be the same version (0.18, 0.19, 0.20, 0.21). • Want to enable forward and backward compatibility • Started work on Avro – Includes the schema of the information as well as the data – Can support different schemas on the client and server – Still need to make the code tolerant of version differences – Avro provides the mechanisms • Avro will be used for file version compatibility too Hadoop World NYC - 2009
- 15. Configuration • Configuration in Hadoop is a string to string map. • Maintaining backwards compatibility of configuration knobs was done case by case. • Now we have standard infrastructure for declaring old knobs deprecated. • Also have cleaned up a lot of the names in 0.21. Hadoop World NYC - 2009
- 16. Questions? • Thanks for coming! • Mailing lists: – [email protected] – [email protected] – [email protected] • Slides posted on the Hadoop wiki page – http://wiki.apache.org/hadoop/HadoopPresentations Hadoop World NYC - 2009