Text Classification Powered by Apache Mahout and Lucene
12 likes•7,714 views
The document discusses text classification using Apache Mahout and Lucene, detailing the processes involved in converting text to vectors and the associated challenges, such as handling noise and varying terms. It highlights different methods for evaluating model performance, including accuracy, precision, recall, and ROC curves, emphasizing the importance of proper data partitioning for training and testing. Additionally, it provides a range of libraries and frameworks for machine learning and collaborative filtering, as well as recommendations for further reading and community involvement.
![Discuss ideas and problems online.
November 16, 2005 [phil h]
http://www.flickr.com/photos/hi-phi/64055296](https://image.slidesharecdn.com/lucenemahoutdrostfrommcopy-131204210407-phpapp02/85/Text-Classification-Powered-by-Apache-Mahout-and-Lucene-76-320.jpg)
Text Classification Powered by Apache Mahout and Lucene
- 1. Text classification With Apache Mahout and Lucene
- 2. Isabel Drost-Fromm Software Engineer at Nokia Maps* Member of the Apache Software Foundation Co-Founder of Berlin Buzzwords and Berlin Apache Hadoop GetTogether Co-founder of Apache Mahout *We are hiring, talk to me or mail [email protected]
- 5. TM
- 6. https://cwiki.apache.org/confluence/display/MAHOUT/Powered+By+Mahout … provide your own success story online.
- 7. TM
- 10. January 8, 2008 by Pink Sherbet Photography http://www.flickr.com/photos/pinksherbet/2177961471/
- 15. How a linear classifier sees data
- 16. Image by ZapTheDingbat (Light meter) http://www.flickr.com/photos/zapthedingbat/3028168415
- 17. Instance* (sometimes also called example, item, or in databases a row)
- 18. Feature* (sometimes also called attribute, signal, predictor, co-variate, or column in databases)
- 19. Label* (sometimes also called class, target variable)
- 23. Image taken in Lisbon/ Portugal.
- 26. ● Remove noise.
- 28. ● Remove noise. ● Convert text to vectors.
- 29. Text consists of terms and phrases.
- 30. Encoding issues? Chinese? Japanese? “New York” vs. new York? “go” vs. “going” vs. “went” vs. “gone”? “go” vs. “Go”?
- 33. Now we have terms – how to turn them into vectors?
- 34. If we looked at two phrases only: Sunny weather High performance computing
- 35. Aaron Zuse
- 36. Binary bag of words ● Imagine a n-dimensional space. ● Each dimension = one possible word in texts. ● Entry in vector is one, if word occurs in text. ● Problem: – bi , j = { 1 ∀ x i ∈d j 0 else } How to know all possible terms in unknown text?
- 37. Term Frequency ● Imagine a n-dimensional space. ● Each dimension = one possible word in texts. ● Entry in vector equal to the words frequency. bi , j =ni , j ● Problem: – Common words dominate vectors.
- 38. TF with stop wording ● Imagine a n-dimensional space. ● Each dimension = one possible word in texts. ● Filter stopwords. ● Entry in vector equal to the words frequency. ● Problem: – bi , j =ni , j Common and uncommon words with same weight.
- 39. TF- IDF ● Imagine a n-dimensional space. ● Each dimension = one possible word in texts. ● Filter stopwords. ● Entry in vector equal to the weighted frequency. ● Problem: – bi , j =ni , j ×log ∣D∣ ∣{ d : t i ∈d }∣ Long texts get larger values.
- 40. Hashed feature vectors ● Imagine a n-dimensional space. ● Each word in texts = hashed to one dimension. ● Entry in vector set to one, if word hashed to it.
- 42. <
- 43. How a linear classifier sees data
- 46. Image by ZapTheDingbat (Light meter) http://www.flickr.com/photos/zapthedingbat/3028168415
- 47. Goals ● Did I use the best model parameters? ● How well will my model perform in the wild?
- 48. Tune model Parameters, Experiment with Tokenization, Experiment with Vector Encoding Compute expected performance
- 50. Performance ● Use same data for training and testing. ● Problem: – Highly optimistic. – Model generalization unknown.
- 51. Performance ● Use same data for training and testing. DON'T ● Problem: – Highly optimistic. – Model generalization unknown.
- 52. Performance ● Use just a fraction for training. ● Set some data aside for testing. ● Problems: – Pessimistic predictor: Not all data used for training. – Result may depend on which data was set aside.
- 53. Performance ● Partition your data into n fractions. ● Each fraction set aside for testing in turn. ● Problem: – Still a pessimistic predictor.
- 54. Performance ● Use just a fraction for training. ● Set some data aside for tuning and testing. ● Problems: – Highly optimistic. – Parameters manually tuned to testing data.
- 55. Performance ● Use just a fraction for training. ● Set some data aside for tuning and testing. DON'T ● Problems: – Highly optimistic. – Parameters manually tuned to testing data.
- 56. Performance ● Use just a fraction for training. ● Set some data aside for tuning. ● Set another set of data aside for testing. ● Problems: – Pretty pessimistic as not all data is used. – May depend on which data was set aside.
- 58. Correct prediction: negative Model prediction: negative Model prediction: positive Correct prediction: positive
- 59. Accuracy ACC= ● true positivetrue negative true positive false positive false negativetrue negative Problems: – What if class distribution is skewed?
- 60. Precision/ Recall true positive Precision= true positive false positive true positive Recall= true positive false negative ● Problem: – Depends on decision threshold.
- 61. ROC Curves
- 63. ROC Curves True orange rate False orange rate
- 64. ROC Curves True orange rate False orange rate
- 65. ROC Curves True orange rate False orange rate
- 66. ROC Curves True orange rate False orange rate
- 67. ROC Curves True orange rate False orange rate
- 68. AUC – area under ROC True orange rate False orange rate
- 69. Foto taken by fras1977 http://www.flickr.com/photos/fras/4992313333/
- 70. Image by Medienmagazin pro http://www.flickr.com/photos/medienmagazinpro/6266643422
- 73. Apache Hadoop-ready Recommendations/ Collaborative filtering kNN and matrix factorization based Collaborative filtering Classification/ Naïve Bayes, random forest Frequent item sets/ (P)FPGrowth Classification/ Logistic Regression/ SGD Clustering/ Mean shift, k-Means, Canopy, Dirichlet Process, Co-Location search Sequence learning/ HMM Math libs/ Mahout collections LDA
- 74. Libraries to have a look at: Vowpal Wabbit Mallet LibSvm LibLinear Libfm Incanter GraphLab Skikits learn Where to get more information: “Mahout in Action” - Manning “Taming Text” - Manning “Machine Learning” - Andrew Ng https://cwiki.apache.org/confluence/dis play/MAHOUT/Books+Tutorials+and+T alks https://cwiki.apache.org/confluence/dis play/MAHOUT/Reference+Reading Image by pareeerica http://www.flickr.com/photos/pareeerica/3711741298/ Frameworks worth mentioning: Apache Mahout Matlab/ Otave Shogun RapidI Apache Giraph R Weka MyMedialight Get your hands dirty: http://kaggle.com https://cwiki.apache.org/confluence/dis play/MAHOUT/Collections Where to meet these people: RecSys NIPS KDD PKDD ApacheCon O'Reilly Strata ICML ECML WSDM JMLR Berlin Buzzwords
- 75. Get started today with the right tools. January 8, 2008 by dreizehn28 http://www.flickr.com/photos/1328/2176949559
- 76. Discuss ideas and problems online. November 16, 2005 [phil h] http://www.flickr.com/photos/hi-phi/64055296
- 77. Images taken at Berlin Buzzwords 2011/12/13 by Philipp Kaden. See you there end of May 2014. Discuss ideas and problems in person.
- 79. Become a committer yourself
- 80. BerlinBuzzwords.de – End of May 2014 in Berlin/ Germany. http:// Online – user/[email protected], [email protected], [email protected] Interest in solving hard problems. Being part of lively community. Engineering best practices. Bug reports, patches, features. Documentation, code, examples. Image by: Patrick McEvoy