NLTK in 20 minutes
Download as KEY, PDF124 likes69,819 views
A sprint thru Python's Natural Language ToolKit, presented at SFPython on 9/14/2011. Covers tokenization, part of speech tagging, chunking & NER, text classification, and training text classifiers with nltk-trainer.
![Sentence Tokenization
>>> from nltk.tokenize import sent_tokenize
>>> sent_tokenize("Hello SF Python. This is NLTK.")
['Hello SF Python.', 'This is NLTK.']
>>> sent_tokenize("Hello, Mr. Anderson. We missed you!")
['Hello, Mr. Anderson.', 'We missed you!']](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-5-320.jpg)
![Word Tokenization
>>> from nltk.tokenize import word_tokenize
>>> word_tokenize('This is NLTK.')
['This', 'is', 'NLTK', '.']](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-6-320.jpg)
![What's a Word?
>>> word_tokenize("What's up?")
['What', "'s", 'up', '?']
>>> from nltk.tokenize import wordpunct_tokenize
>>> wordpunct_tokenize("What's up?")
['What', "'", 's', 'up', '?']
Learn More: http://text-processing.com/demo/tokenize/](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-7-320.jpg)
![Part-of-Speech Tagging
>>> words = word_tokenize("And now for something
completely different")
>>> from nltk.tag import pos_tag
>>> pos_tag(words)
[('And', 'CC'), ('now', 'RB'), ('for', 'IN'),
('something', 'NN'), ('completely', 'RB'), ('different',
'JJ')]
Tags List: http://www.ling.upenn.edu/courses/
Fall_2003/ling001/penn_treebank_pos.html](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-8-320.jpg)
![Chunking & NER
>>> from nltk.chunk import ne_chunk
>>> ne_chunk(pos_tag(word_tokenize('My name is Jacob
Perkins.')))
Tree('S', [('My', 'PRP$'), ('name', 'NN'), ('is', 'VBZ'),
Tree('PERSON', [('Jacob', 'NNP'), ('Perkins', 'NNP')]),
('.', '.')])](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-10-320.jpg)
![NER not perfect
>>> ne_chunk(pos_tag(word_tokenize('San Francisco is
foggy.')))
Tree('S', [Tree('GPE', [('San', 'NNP')]), Tree('PERSON',
[('Francisco', 'NNP')]), ('is', 'VBZ'), ('foggy', 'NN'),
('.', '.')])](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-11-320.jpg)
![Text Classification
def bag_of_words(words):
return dict([(word, True) for word in words])
>>> feats = bag_of_words(word_tokenize("great movie"))
>>> import nltk.data
>>> classifier = nltk.data.load('classifiers/
movie_reviews_NaiveBayes.pickle')
>>> classifier.classify(feats)
'pos'](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-12-320.jpg)
![Train a Sentiment Classifier
$ ./train_classifier.py movie_reviews --instances paras
loading movie_reviews
2 labels: ['neg', 'pos']
2000 training feats, 2000 testing feats
training NaiveBayes classifier
accuracy: 0.967000
neg precision: 1.000000
neg recall: 0.934000
neg f-measure: 0.965874
pos precision: 0.938086
pos recall: 1.000000
pos f-measure: 0.968054
dumping NaiveBayesClassifier to ~/nltk_data/classifiers/
movie_reviews_NaiveBayes.pickle](https://image.slidesharecdn.com/nltkoverviewsfpython-110915160025-phpapp02/85/NLTK-in-20-minutes-15-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to NLTK in 20 minutes (20)
Ad
Recently uploaded (20)
Ad
NLTK in 20 minutes
- 1. NLTK in 20 minutes A sprint thru Python's Natural Language ToolKit
- 2. Jacob Perkins Co-founder/CTO @ Weotta (we're hiring :) "Python Text Processing with NLTK 2.0 Cookbook" NLTK Contributor Blog: http://streamhacker.com NLTK Demos & APIs: http://text-processing.com @japerk
- 3. Why Text Processing? sentiment analysis spam filtering plagariasm detection / document similarity document categorization / topic detection phrase extraction, summarization smarter search simple keyword frequency analysis
- 4. Some NLTK Features sentence & word tokenization part-of-speech tagging chunking & named entity recognition text classification many included corpora
- 5. Sentence Tokenization >>> from nltk.tokenize import sent_tokenize >>> sent_tokenize("Hello SF Python. This is NLTK.") ['Hello SF Python.', 'This is NLTK.'] >>> sent_tokenize("Hello, Mr. Anderson. We missed you!") ['Hello, Mr. Anderson.', 'We missed you!']
- 6. Word Tokenization >>> from nltk.tokenize import word_tokenize >>> word_tokenize('This is NLTK.') ['This', 'is', 'NLTK', '.']
- 7. What's a Word? >>> word_tokenize("What's up?") ['What', "'s", 'up', '?'] >>> from nltk.tokenize import wordpunct_tokenize >>> wordpunct_tokenize("What's up?") ['What', "'", 's', 'up', '?'] Learn More: http://text-processing.com/demo/tokenize/
- 8. Part-of-Speech Tagging >>> words = word_tokenize("And now for something completely different") >>> from nltk.tag import pos_tag >>> pos_tag(words) [('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')] Tags List: http://www.ling.upenn.edu/courses/ Fall_2003/ling001/penn_treebank_pos.html
- 9. Why Part-of-Speech Tag? word definition lookup (WordNet, WordNik) fine-grained text analytics part-of-speech specific keyword analysis chunking & named entity recognition (NER)
- 10. Chunking & NER >>> from nltk.chunk import ne_chunk >>> ne_chunk(pos_tag(word_tokenize('My name is Jacob Perkins.'))) Tree('S', [('My', 'PRP$'), ('name', 'NN'), ('is', 'VBZ'), Tree('PERSON', [('Jacob', 'NNP'), ('Perkins', 'NNP')]), ('.', '.')])
- 11. NER not perfect >>> ne_chunk(pos_tag(word_tokenize('San Francisco is foggy.'))) Tree('S', [Tree('GPE', [('San', 'NNP')]), Tree('PERSON', [('Francisco', 'NNP')]), ('is', 'VBZ'), ('foggy', 'NN'), ('.', '.')])
- 12. Text Classification def bag_of_words(words): return dict([(word, True) for word in words]) >>> feats = bag_of_words(word_tokenize("great movie")) >>> import nltk.data >>> classifier = nltk.data.load('classifiers/ movie_reviews_NaiveBayes.pickle') >>> classifier.classify(feats) 'pos'
- 13. Classification Algos in NLTK Naive Bayes Maximum Entropy / Logistic Regression Decision Tree SVM (coming soon)
- 14. NLTK-Trainer https://github.com/japerk/nltk-trainer command line scripts train custom models analyze corpora analyze models against corpora
- 15. Train a Sentiment Classifier $ ./train_classifier.py movie_reviews --instances paras loading movie_reviews 2 labels: ['neg', 'pos'] 2000 training feats, 2000 testing feats training NaiveBayes classifier accuracy: 0.967000 neg precision: 1.000000 neg recall: 0.934000 neg f-measure: 0.965874 pos precision: 0.938086 pos recall: 1.000000 pos f-measure: 0.968054 dumping NaiveBayesClassifier to ~/nltk_data/classifiers/ movie_reviews_NaiveBayes.pickle
- 16. Notable Included Corpora movie_reviews: pos & neg categorized IMDb reviews treebank: tagged and parsed WSJ text treebank_chunk: tagged and chunked WSJ text brown: tagged & categorized english text 60 other corpora in many languages
- 17. Other NLTK Features clustering metrics parsing stemming WordNet ... and a lot more
- 18. Other Python NLP Libraries pattern: http://www.clips.ua.ac.be/pages/pattern scikits.learn: http://scikit-learn.sourceforge.net/stable/ fuzzywuzzy: https://github.com/seatgeek/fuzzywuzzy
- 20. NLTK Tutorial @ PyCon What would you want to learn in 3 hours? What kinds of NLP problems do you face at work? What do you want to do with text?
Editor's Notes
- #2: \n
- #3: \n
- #4: text processing is very useful in a number of areas, and there's tons of unstructured text flooding the internet nowadays, and NLP/ML is one of the best ways to deal with it\n
- #5: this is what I'll cover today, but there's a lot more I won't be covering\n
- #6: loads a trained sentence tokenizer, then calls its tokenize() method. has sentence tokenizers for 16 languages. Smarter than just splitting on punctuation.\n
- #7: loads a word tokenizer trained on treebank, then calls the tokenize() method\n
- #8: non-ascii characters are also a problem for word_tokenize(). wordpunct_tokenize() can often be better, but you need to first decide what a word is for your specific case. do contractions matter? can you replace them with two words? Demo shows the results from 4 different tokenizers\n
- #9: loads a pos tagger trained on treebank - first call will take a few seconds to load the pickle file off disk, every subsequent call will use in-memory tagger. can find tables of pos tag definitions online.\n
- #10: pos tags might not be useful by themselves, but they are useful metadata for other NLP tasks like dictionary lookup, pos specific keyword analysis, and they are essential for chunking & NER\n
- #11: every Tree has a draw() method that uses TKinter\n
- #12: \n
- #13: bag-of-words is the simplest model, but ignores frequency. good for small text, but frequency can be very important for larger documents. other algorithms, like SVM, create sparse arrays of 1 or 0 depending on word presence, but require knowning full vocabulary beforehand. this classifier is one I trained with nltk-trainer, and can be used for sentiment analysis because it's categories are "pos" and "neg".\n
- #14: \n
- #15: can train taggers, chunkers, and text classifiers, and is great for analyzing corpora and how a model performs against a labeled corpus. I use nltk-trainer to train all my models nowadays.\n
- #16: this trains a very basic sentiment analysis classifier on the movie_reviews corpus, which has reviews categorized into pos or neg\n
- #17: treebank is a very standard corpus for testing taggers and chunkers\n
- #18: NLP isn't black magic, but you can treat it as a black box until the defaults aren't good enough. Then you need to dig in and learn how it works so you can make it do what you want. At that point, the best thing you can do is find/make good data, then use existing algos to learn from it.\n
- #19: \n
- #20: the original NLTK is very good, available for free online, but takes "textbook" approach. I tried to be a lot more practical in my cookbook. nltk-users mailing list is pretty active, and you can also try stackoverflow\n
- #21: \n