Inside hadoop-dev
Download as PPTX, PDF6 likes4,381 views
An overview of the development of the Apache Hadoop software stack, including some of the barriers to participation -and how and why to overcome them. It closes with some open discussion points/ideas of how the existing process can be improved.
Ad
More Related Content
What's hot (20)
Viewers also liked (7)
Ad
Similar to Inside hadoop-dev (20)
Ad
More from Steve Loughran (20)
Inside hadoop-dev
- 1. Inside hadoop-dev Steve Loughran– Hortonworks @steveloughran Apachecon EU, November 2012 © Hortonworks Inc. 2012
- 2. [email protected] • HP Labs: –Deployment, cloud infrastructure, Hadoop-in-Cloud • Apache – member and committer –Ant (author, Ant in Action), Axis 2 –HadoopJoined Hortonworks in 2012 –UK based R&D Page 2 © Hortonworks Inc. 2012
- 3. Hadoop is the OS for the datacentre Page 3 © Hortonworks Inc. 2012
- 4. Page 4 © Hortonworks Inc. 2012
- 5. History: ASF releases slowed 0.20.0 0.20.1 0.20.2 0.21.0 0.20.20{3,4,5}.0 • 64 Releases from 2006-2011 • Branches from the last 2.5 years: –0.20.{0,1,2} – Stable release without security –0.20.2xx.y – Stable release with security –0.21.0 – released, unstable, deprecated –0.22.0 – orphan, unstable, lack of community –0.23.x • Cloudera CDH: fork w/ patches pushed back Page 5
- 6. Now: 2 ASF branches Hadoop 1.x • Stable, used in production systems • Features focus on fixes & low-risk performance Hadoop 2.x/trunk • The successor • Alpha-release. Download and test • Where features & fixes first go in • Your new code goes here. Page 6
- 7. Loosely coupled projects form the stack Page 7 © Hortonworks Inc. 2012
- 8. Incubating & graduate projects Kafka Giraph HCatalog templeton Ambari Page 8 © Hortonworks Inc. 2012
- 9. Integration is a major undertaking Latest ASF artifacts Stable, tested ASF artifacts ASF + own artifacts Page 9 © Hortonworks Inc. 2012
- 10. What does all this mean? Page 10 © Hortonworks Inc. 2012
- 11. There is more work than we can cope with Page 11 © Hortonworks Inc. 2012
- 12. Hadoop is CS-Hard • Core HDFS, MR and YARN – Distributed Computing – Consensus Protocols & Consistency Models – Work Scheduling & Data Placement – Reliability theory – CPU Architecture; x86 assembler • Others – Machine learning – Distributed Transactions – Graph Theory – Queue Theory – Correctness proofs Page 12 © Hortonworks Inc. 2012
- 13. If you have these skills, come and play! http://hortonworks.com/careers/ Page 13 © Hortonworks Inc. 2012
- 14. But there are barriers Page 14 © Hortonworks Inc. 2012
- 15. Your time & cluster • Full time core business @ Hortonworks + Cloudera • Full time projects at others: LinkedIn, IBM, MSFT, VMWare • Single developers can't compete • Small test runs take too long • Your cluster probably isn't as big as Yahoo!'s • Review-then-Commit neglects everyone's patches Page 15 © Hortonworks Inc. 2012
- 16. Fear of damage The worth of Hadoop is the data in HDFS the worth of all companies whose data it is cost to individuals of data loss cost to governments of losing their data ∴ resistance to radical changes in HDFS Scheduling performance worth $100Ks to individual organisations ∴ resistance to radical work in compute layer except by people with track record Page 16 © Hortonworks Inc. 2012
- 17. Fear of support and maintenance costs • What will show up on Yahoo!-scale clusters? • Costs of regression testing • Who maintains the code if the author disappears? • Documentation? The 80%-done problem Page 17 © Hortonworks Inc. 2012
- 18. How to get your code in • Trust: get known in the -dev lists, meet-ups • Competence: help with patches other than your own. • Don't attempt rewrites of the core services • Help develop plugin-points • Test across the configuration space • Test at scale, complexity, “unusualness” Page 18 © Hortonworks Inc. 2012
- 19. Testing: not just for the 1% Page 19 © Hortonworks Inc. 2012
- 20. youTesting: not just for scale issues have network and the 1% Page 20 © Hortonworks Inc. 2012
- 21. Documentation & Books Page 21 © Hortonworks Inc. 2012
- 22. Challenge: Major Works • YARN and HDFS HA – Branch then final review at merge – Agile; merge costs scale w/ duration of branch • Independent works – Things that didn't get in -my lifecycle work, … – VMWare virtualisations –initial failure topology how best to get this stuff in • Postgraduate Research – How to get the next generation of postgraduate researchers developing in and with Apache Hadoop? Page 22 © Hortonworks Inc. 2012
- 23. A mentoring program? Guided support for associated projects, the goal to be to merge into the Hadoop codebase. Who has the time to mentor? Page 23 © Hortonworks Inc. 2012
- 24. Better Distributed Development • Regional developer workshops – with local university participation? • Online meet-ups: google+ hangouts? – Shared IDEA or other editor sessions – Remote presentations and demos Page 24 © Hortonworks Inc. 2012
- 25. Git + Gerrit Page 25 © Hortonworks Inc. 2012
- 26. Get involved! svn.apache.org issues.apache.org {hadoop,hbase, mahout, pig, oozie, …}.apache.org Page 26 © Hortonworks Inc. 2012
- 27. hortonworks.com Page 27 © Hortonworks Inc. 2012
Editor's Notes
- #3: This is my background: key point until 2012 I was working on my own things inside a large organisation; now I am FTE on Hadoop
- #7: There's a CoI here between trunk features and branch-1 commits -the latter get into people's hands faster, but threaten the very feature -stability- that justifies branch-1's existence.All the interesting stuff goes into trunk, which is where I push most of my patches (it's easier to avoid backporting)
- #10: Bigtop is ±Fedora: bleeding edge -but also defines RPM installation layout and startup scripts for everyone, for consistency.Hortonworks -trails with the stable artifacts, team manages the Apache Hadoop releases and QA team tests all.Cloudera do a mix of ASF + Apache; got own fork of Hadoop with different set/ordering of patches,.CDH vs HDP is a matter of argument. One thing to know is that everyone now tends to use Git to manage their individual branches
- #14: If you thinjk
- #15: If you thinjk
- #19: Plugin points: yes, I think googleguice would be the alternative, but, well…
- #20: Most people here do not have 500+ clusters with double digit PB of storage. Those clusters are the best for the stress testing of the storage and computer layers -but only a few people have them at this scale: Y! FB. We use Y!'s test clusters for all the apache & Hortonworks releases,
- #21: you have your own issues. Does it scale down enough? does it assume the LAN is well managed, clocks in sync, DNS andrDNS works. Your problems -especially the networking ones -are your own. This is why testing them matters
- #22: I'm proposing people write books for the benefit of the project, not the fame and money with comes with writing a book, Anyone else who has written a book will know precisely why I'm doing that.
- #24: We do have this for the Apache Incubator -but they are projects above and alongside the existing codebase. I'm wondering here how to get medium-sized bits of work done in a way that is timely, not wasted.
- #25: There's no easy answers here, but here are some things I think could be goodGit workflow support. Stops people having to resubmit patches all the time; git pull can be used to grab and apply a patch.Gerrit code review -makes reviewing much, much easier. We have HUG events -but they tend to not normally delve into the codebase. I'm proposing doing exactly that -in regions other than just the Bay Area. I will back this up by offering to host an all day one at a bar/café near me in Bristol if enough people are interested., I'm also advocating university involvement so that they get more of an idea of Hadoop internals.For those of outside the Bay Area, remote events are good. We've had some good webex'd events recently (e.g. the YARN one), but could do with more. I'd like to see something more interactive, and think we could/should try with an online only google+ hangout coding event, possibly using a shared IDE.