Intro to Mahout -- DC Hadoop
Download as PPTX, PDF12 likes4,136 views
Machine learning is used widely on the web today. Apache Mahout provides scalable machine learning libraries for common tasks like recommendation, clustering, classification and pattern mining. It implements many algorithms like k-means clustering in a MapReduce framework allowing them to scale to large datasets. Mahout functionality includes collaborative filtering, document clustering, categorization and frequent pattern mining.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Intro to Mahout -- DC Hadoop (20)
Ad
More from Grant Ingersoll (20)
Recently uploaded (20)
Intro to Mahout -- DC Hadoop
- 1. Intro to Apache MahoutGrant IngersollLucid Imaginationhttp://www.lucidimagination.com
- 2. Anyone Here Use Machine Learning?Any users of:Google?Search?Priority Inbox?Facebook?Twitter?LinkedIn?
- 3. TopicsBackground and Use CasesWhat can you do in Mahout?Where’s the community at?ResourcesK-Means in Hadoop (time permitting)
- 4. Definition“Machine Learning is programming computers to optimize a performance criterion using example data or past experience”Intro. To Machine Learning by E. AlpaydinSubset of Artificial IntelligenceLots of related fields:Information RetrievalStatsBiologyLinear algebraMany more
- 5. Common Use CasesRecommend friends/dates/productsClassify content into predefined groupsFind similar contentFind associations/patterns in actions/behaviorsIdentify key topics/summarize textDocuments and CorporaDetect anomalies/fraudRanking search resultsOthers?
- 6. Apache MahoutAn Apache Software Foundation project to create scalable machine learning libraries under the Apache Software Licensehttp://mahout.apache.orgWhy Mahout?Many Open Source ML libraries either:Lack CommunityLack Documentation and ExamplesLack ScalabilityLack the Apache LicenseOr are research-orientedDefinition:http://dictionary.reference.com/browse/mahout
- 7. What does scalable mean to us?Goal: Be as fast and efficient as possible given the intrinsic design of the algorithmSome algorithms won’t scale to massive machine clustersOthers fit logically on a Map Reduce framework like Apache HadoopStill others will need different distributed programming modelsOthers are already fast (SGD)Be pragmatic
- 8. Sampling of Who uses Mahout?https://cwiki.apache.org/confluence/display/MAHOUT/Powered+By+Mahout
- 9. What Can I do with Mahout Right Now?3C + FPM + O = Mahout
- 10. Collaborative FilteringExtensive framework for collaborative filtering (recommenders)RecommendersUser basedItem basedOnline and Offline supportOffline can utilize HadoopMany different Similarity measuresCosine, LLR, Tanimoto, Pearson, others
- 11. ClusteringDocument levelGroup documents based on a notion of similarityK-Means, Fuzzy K-Means, Dirichlet, Canopy, Mean-Shift, EigenCuts (Spectral)All Map/ReduceDistance MeasuresManhattan, Euclidean, otherTopic Modeling Cluster words across documents to identify topicsLatent Dirichlet Allocation (M/R)
- 12. CategorizationPlace new items into predefined categories:Sports, politics, entertainmentRecommendersImplementationsNaïve Bayes (M/R)Compl. Naïve Bayes (M/R)Decision Forests (M/R)Linear Regression (Seq. but Fast!)See Chapter 17 of Mahout in Action for Shop It To Me use case:
- 13. http://awe.sm/5FyNeFreq. Pattern MiningIdentify frequently co-occurrent itemsUseful for:Query RecommendationsApple -> iPhone, orange, OS XRelated product placementBasket AnalysisMap/Reducehttp://www.amazon.com
- 14. OtherPrimitive Collections!Collocations (M/R)Math libraryVectors, Matrices, etc.Noise Reduction via Singular Value Decomp (M/R)
- 15. Prepare Data from Raw contentData Sources:Lucene integrationbin/mahout lucene.vector…Document Vectorizerbin/mahout seqdirectory …bin/mahout seq2sparse …ProgrammaticallySee the Utils module in Mahout and the Iterator<Vector> classesDatabaseFile system
- 16. How to: Command LineMost algorithms have a Driver program$MAHOUT_HOME/bin/mahout.shhelps with most tasksPrepare the DataDifferent algorithms require different setupRun the algorithmSingle NodeHadoopPrint out the results or incorporate into applicationSeveral helper classes: LDAPrintTopics, ClusterDumper, etc.
- 17. What’s Happening Now?Unified Framework for Clustering and Classification0.5 release on the horizon (May?)Working towards 1.0 release by focusing on:Tests, examples, documentationAPI cleanup and consistencyGearing up for Google Summer of CodeNew M/R work for Hidden Markov Models
- 18. SummaryMachine learning is all over the web todayMahout is about scalable machine learningMahout has functionality for many of today’s common machine learning tasksMany Mahout implementations use Hadoop
- 20. Resources“Mahout in Action” Owen, Anil, Dunning and Friedmanhttp://awe.sm/5FyNe“Introducing Apache Mahout” http://www.ibm.com/developerworks/java/library/j-mahout/“Taming Text” by Ingersoll, Morton, Farris“Programming Collective Intelligence” by Toby Segaran“Data Mining - Practical Machine Learning Tools and Techniques” by Ian H. Witten and Eibe Frank“Data-Intensive Text Processing with MapReduce” by Jimmy Lin and Chris Dyer
- 22. K-Means in Map-ReduceInput:Mahout Vectors representing the original contentEither:A predefined set of initial centroids (Can be from Canopy)--k – The number of clusters to produceIterateDo the centroid calculation (more in a moment)Clustering Step (optional)OutputCentroids (as Mahout Vectors)Points for each Centroid (if Clustering Step was taken)
- 23. Map-Reduce IterationEach Iteration calculates the Centroids using:KMeansMapperKMeansCombinerKMeansReducerClustering StepCalculate the points for each Centroid using:KMeansClusterMapper
- 24. KMeansMapperDuring Setup:Load the initial Centroids (or the Centroids from the last iteration)Map PhaseFor each inputCalculate it’s distance from each Centroid and output the closest oneDistance Measures are pluggableManhattan, Euclidean, Squared Euclidean, Cosine, others
- 25. KMeansReducerSetup:Load up clustersConvergence informationPartial sums from KMeansCombiner (more in a moment)Reduce PhaseSum all the vectors in the cluster to produce a new CentroidCheck for ConvergenceOutput cluster
- 26. KMeansCombinerJust like KMeansReducer, but only produces partial sum of the cluster based on the data local to the Mapper
- 27. KMeansClusterMapperSome applications only care about what the Centroids are, so this step is optionalSetup:Load up the clusters and the DistanceMeasure usedMap PhaseCalculate which Cluster the point belongs toOutput <ClusterId, Vector>