Tutorial of Sentiment Analysis
Download as PPTX, PDF•39 likes•36,274 views
This document provides an overview of sentiment analysis techniques including AFINN-111, SentiWordNet, and document classification. It describes analyzing sentiment at the word level using lexicons and at the document level. Key steps are outlined such as tokenization, part-of-speech tagging, word sense disambiguation, and assigning sentiment scores. Issues with analyzing short texts like tweets are also discussed. The document provides references and links to related projects and APIs.
![• SentiWordNet is an extension of WordNet that adds
for each synset 3 measures:
• PosScore [0,1] : positivity measure
• NegScore [0,1]: negativity measure
• ObjScore [0,1]: objective measure
ObjScore
a
a
00016135
00016247
0
0.125
=
1
– (PosScore + NegScore )
0.25 rank#5
0.5
superabundant#1
growing profusely; "rank jungle vegetation"
most excessively abundant
• SentiWordNet 3.0: An Enhanced Lexical Resource for
Sentiment Analysis and Opinion Mining
• http://sentiwordnet.isti.cnr.it/](https://image.slidesharecdn.com/20131001bigdataanalysistweetsentiment-131125063458-phpapp02/85/Tutorial-of-Sentiment-Analysis-5-320.jpg)
Tutorial of Sentiment Analysis
- 1. TUTORIAL OF SENTIMENT ANALYSIS Fabio Benedetti
- 2. Outline • Introduction to vocabularies used in sentiment analysis • Description of GitHub project • Twitter Dev & script for download of tweets • Simple sentiment classification with AFINN-111 • Define sentiment scores of new words • Sentiment classification with SentiWordNet • Document sentiment classification
- 3. AFINN-111 • AFINN is a list of English words rated for sentiment score. • between -5 (negative) to +5 (positive). • AFINN-111: Newest version with 2477 words and phrases. … Abilities 2 Ability 2 Aboard 1 Absentee -1 …
- 4. WordNet • WordNet is lexical database for the English language that groups English word into set of synonyms called synset • WordNet distinguishes between : • nouns • verbs • adjectives • adverbs SYNSET# SYNSET4 SYNSET2 SYNSET1
- 5. • SentiWordNet is an extension of WordNet that adds for each synset 3 measures: • PosScore [0,1] : positivity measure • NegScore [0,1]: negativity measure • ObjScore [0,1]: objective measure ObjScore a a 00016135 00016247 0 0.125 = 1 – (PosScore + NegScore ) 0.25 rank#5 0.5 superabundant#1 growing profusely; "rank jungle vegetation" most excessively abundant • SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining • http://sentiwordnet.isti.cnr.it/
- 6. Project on GitHub • https://github.com/linkTDP/BigDataAnalysis_TweetSentim ent • AFINN-111.txt • SentiWordNet_3.0.0_20130122.txt • config.json • ExtractTweet.py • DeriveTweetSentimentEasy.py • NewTermSentimentInference.py • SentiWordnet.py • DocumentSentimentClassification.py
- 7. config.json & ExtractTweet.py (1) This script can be used to download tweets in a csv file and is configurable through config.json The authentication fields that must be set are: • consumer_key • consumer_secret • access_token • access_token_secret These fields can be retrieved from https://dev.twitter.com creating an account and an application
- 8. Twitter Developers • Create an account on the site: https://dev.twitter.com/
- 10. config.json & ExtractTweet.py (2) Other fields: • file_name (name of the .cvs output file) • count (number of tweet to download) • filter (a word used to filter the tweet in output) The CSV file produced in output can be used as input of the other three script.
- 11. DeriveTweetSentimentEasy.py This script use AFINN-111 as vocabulary In AFINN-111 the score is negative and positive according to sentiment of the word. Therefore a very rudimental sentiment score of the tweet can be calculated summing the score of each word. Issue: In AFINN-111 not all the words are present.
- 13. SentiWordnet.py This script use SentiWordNet as vocabulary and an the algorithm that is implemented is inspired by : Hamouda, Alaa, and Mohamed Rohaim. "Reviews classification using sentiwordnet lexicon." World Congress on Computer Science and Information Technology. 2011. http://www.academia.edu/1336655/Reviews_Classific ation_Using_SentiWordNet_Lexicon
- 15. Tokenization & Speech Tagging • Tokenization process: splits the text into very simple tokens such as numbers, punctuation and words of different types. • Speech Tagging process: produces a tag as an annotation based on the role of each word in the tweet. noun verb noun adverb Francesco speaks English well
- 16. Word Sense Disambiguation The techniques of WSD are aimed at the determination of the meaning of every word in his context. In this case the disambiguation happens selecting for each words in a tweet the synset in WordNet that best represents this word in his context.
- 17. Word Sense Disambiguation (2) I have implemented a simple (and inaccurate) algorithm of WSD using NLTK (Python's library for NLP). Each synset in WordNet has a textual a brief description called Gloss. Very intuitively this algorithm choose as synset of the word the one whose Gloss contains the largest number of words present in the tweet. If no Gloss has a match with the tweet's words, the algorithm choose the first synset, that usually is the most used. Issue: The corpus of a tweet is very small (max 140 character), so this algorithm could produce a bad disambiguation of the word's sense.
- 18. SentiWordNet Interpretation Given a synset (after the phase of WSD) we can search in SentiWordNet the sentiment score associated to this synset tweet @BonksMullet @chet_sellers This is very accurate and hilarious. Well done :) WSD synset accurate#1 conforming exactly or almost exactly to fact or to a standard or performing with total accuracy; "an accurate reproduction"; "the accounting was accurate"; "accurate measurements"; "an accurate scale" SentiWordNet score Pos_score 0.5 Neg_score 0 Obj_score 0.5
- 23. Open issues • the tweet's corpus is too short to use the great part of the WSD techniques • In this kind of short texts (tweet or Facebook's comments) is used a particular slang that needs ad hoc techniques to be processed. Insights: • Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of Twitter data. In Proceedings of the Workshop on Languages in Social Media (LSM '11) • Gokulakrishnan, B.; Priyanthan, P.; Ragavan, T.; Prasath, N.; Perera, A., "Opinion mining and sentiment analysis on a Twitter data stream," Advances in ICT for Emerging Regions (ICTer), 2012 International Conference on.
- 24. Example of Documents Sentiment Classification DocumentSentimentClassification.py Implementation of the algorithm for Document Classification see at lesson Turney, Peter D., and Michael L. Littman. "Measuring praise and criticism: Inference of semantic orientation from association." ACM Transactions on Information Systems (TOIS) 21.4 (2003): 315-346.
- 25. Parameters Parameters (at the start of the code): • FILE_NAME = “ name of the file .txt on which you want execute the classification” • API_KEY_BING = “Api Key Bing” • API_KEY_GOOGLE = “Api Key for Custom Search Api” • USE_GOOGLE = (Boolean) Enable (True) or Disable (False) the use of the Google Api for Custom Search The number of free queries per day using Google Api are limited to 100!!
- 26. Libraries • NLTK – Natural Language Toolkit • tokenizers/punkt/english.pickle Module • Requests • Math • Urllib2 • google-api-python-client • https://code.google.com/p/google-api-python-client/ This libraries could be installed using Pip: pip install <library name>
- 28. Bing API - Key
- 29. Google API – Custom Search • https://cloud.google.com/console#/project
- 30. Google API – Custom Search • https://cloud.google.com/console#/project
- 31. Google API – Custom Search (1)
- 32. Google API – Custom Search (1)
- 33. Google API – Custom Search (1)
- 34. References • AFFIN-111 - • • • • • http://www2.imm.dtu.dk/pubdb/views/publication_details.php ?id=6010 SentiWordNet - http://sentiwordnet.isti.cnr.it/ SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining http://nmis.isti.cnr.it/sebastiani/Publications/LREC06.pdf Reviews ClassificationUsing SentiWordNet Lexicon http://www.academia.edu/1336655/Reviews_Classification_Usi ng_SentiWordNet_Lexicon Using SentiWordNet and Sentiment Analysis for Detecting Radical Content on Web Forums http://www.jeremyellman.com/jeremy_unn/pdfs/1_____Chaloth orn_Ellman_SKIMA_2012.pdf From tweets to polls: Linking text sentiment to public opinion time series http://www.aaai.org/ocs/index.php/ICWSM/ICWSM10/paper/vi ewFile/1536/1842
- 35. References • Natural Language Toolkit - http://nltk.org/ • Twitter Developers - https://dev.twitter.com/ • Tweepy - https://github.com/tweepy/tweepy • Python csv - http://www.pythonforbeginners.com/systems -programming/using-the-csv-module-inpython/