Best Practices

for
Sentiment
Analysis

Presented by:
John Hoskins
Amazon Mechanical Turk

Max Yankelevich
Freedom OSS CrowdControl
 Introductions

John Hoskins
Amazon Mechanical Turk
Sr. Manager, Business Development

Max Yankelevich
FreedomOSS
Chief Architect

© 2011 Amazon.com, Inc. or its Affiliates.

 Welcome
 Logistics
 How to ask a question:.
 Ask a question with the question
panel.
 We will moderate questions.

 Agenda
 Introductions
 Sentiment Analysis
 Applying Mechanical Turk
 CrowdControl – optimized for Sentiment
 Q&A

© 2011 Amazon.com, Inc. or its Affiliates.

 What is Sentiment Analysis?
 Keeping your finger on the pulse of your market – in
near real time
With the explosion in use of Facebook, Twitter, and blogging – it is
essential to hear the true Voice of the Customer. Yet keeping up with
the round-the-clock information torrent is impossible with old,
manual methods. As businesses look to automate the process of
filtering out the noise, understanding the conversations, identifying
the relevant content and actioning it appropriately, many are now
looking to the field of sentiment analysis[

© 2011 Amazon.com, Inc. or its Affiliates.

 Why Sentiment Analysis?
The ability to lead the competition
 in customer satisfaction and support,

 brand and reputation management

 product design and marketing

With the proliferation of reviews, ratings,

recommendations and other forms of online expression,
online opinion has turned into a kind of virtual currency
for businesses looking to market their products, identify
new opportunities and manage their reputations.

© 2011 Amazon.com, Inc. or its Affiliates.

 How to properly analyze
Sentiment.

© 2011 Amazon.com, Inc. or its Affiliates.

 Context & Cognitive Recognition
are the keys to an accurate analysis
Human analysis provides more accurate assessment.
 Complex emotions such as sarcasm are presented in tweets and
blog posts.
 Wit, sarcasm and complex emotions are difficult to analyze with
technology alone
 Processing the unnatural language of text messaging: lol, omg
 Positive and negative emotions are difficult to accurately assess
 Avoid misleading classification of SPAM

© 2011 Amazon.com, Inc. or its Affiliates.

 Is human judgment
affordable and scalable?
Yes.

© 2011 Amazon.com, Inc. or its Affiliates.

 Introducing Mechanical Turk
An affordable solution for human judgment.

© 2011 Amazon.com, Inc. or its Affiliates.

 Mechanical Turk is a
marketplace for work.
Mechanical Turk gives businesses and developers access to
an on-demand, scalable workforce.
 Flexibility: Scale your workforce up and down quickly
 Accuracy: Get high quality, efficient and cost effective
results.
 Price: Pay only when you are satisfied with the results.
 Speed: Start receiving results in minutes

© 2011 Amazon.com, Inc. or its Affiliates.

 Workforce
500,000 Workers
190+ Countries

 Who are the Workers?

 Workers are global: 24X7 Follow the Sun

 Managing Your Workforce

 Can narrow to US based (i.e. when you need western
culture competence)
 Leverage qualifications to find your best Workers

© 2011 Amazon.com, Inc. or its Affiliates.

 How it Works

The Worker
• Design Your Task • Validate the Results
(HIT) • Pay Workers
• Accepts your HIT
• Publish your HIT to • Go!
• Submits an answer
the marketplace

You – The You – The

Requester Requester

© 2011 Amazon.com, Inc. or its Affiliates.

 Popular Use Cases
 Data Management  Categorization
 Data Verification  Classification
 Data Entry & Collection  Tagging
 Data De-duplication  Sentiment Analysis
 Algorithm Training

 Content & Media  Business Services

 Moderate Photos & Content  Search Relevancy

 Content Creation & Editing  Product Usability Testing

 Transcription  Research

© 2011 Amazon.com, Inc. or its Affiliates.

 Case Study – Sentiment Analysis
 Problem:
A large consumer brand reporting customer need 10’s of
thousands of human coded samples to establish a baseline –
for each project

 Solution:
Code them through Mechanical Turk.

 Details:
 Projects are completed in hours providing faster to market
production of broad analysis.
 By integrating Mechanical Turk into their process, they freed up
analysts time to do value add work and can start projects in a day
instead of weeks.

© 2011 Amazon.com, Inc. or its Affiliates.

 Strategies for Analyzing Results
 Plurality
Asking multiple Workers to do assignments (if
their answers agree then the result is validated)
 Qualifying & Training Workers
Assess competence on your coding instructions
before allowing them to work for you
 Known Data Sets
Include work with known answers to quickly
assess worker accuracy.

© 2011 Amazon.com, Inc. or its Affiliates.

 Implementing a Sentiment
Analysis Workflow

© 2011 Amazon.com, Inc. or its Affiliates.

 Freedom at a Glance
 Long term AWS Partner
 Enterprise Cloud Business
 CrowdControl – fastest growing division

 Corporate Office: Newtown PA

 Regional Offices: Newark, NJ. Reston, VA, Seattle, WA
 Offshore Engineering Centers
 Global operations centers

 Privately held
 Established in 2008
 300% annual growth

© 2011 Amazon.com, Inc. or its Affiliates.

 Workflow, Adjudication,
Worker Management.
Programmatic connection to crowdsourcing.
 How do I know I have the best answer?
 How do I break my workflow into discrete tasks
 Who are my best workers?
 How do I put it all together?

© 2011 Amazon.com, Inc. or its Affiliates.

 CrowdControl for Sentiment Analysis
 Provides High Sentiment Quality Data at Lower Cost

 Ability to Process Large Number of UGC

 Combines best of breed Artificial Intelligence to handle most

challenging nuances of “Crowdsourcing” for efficient
Sentiment Analysis
 Worker Management
 Adjudication strategies
 Workflows
 Can easily retrieve and send data to and from any data
source (e.g. Database, File ,etc.)

© 2011 Amazon.com, Inc. or its Affiliates.

 What Can CrowdControl™ Do for Me?
Customer Customer
IT IT
Systems Systems
Step
• Create sentiment coding templates
Quality
Step
• Come up with opinion adjudication rules
Information
Manual Steps

• Extract data from Internal and External IT Systems

Automation
Step

Complete
Step
• Review responses

Step
• Ban “bad” workers

Step
• Accept or reject answers

Step
• Pay workers

Step
• Assign special worker qualifications

Step
• Import data back into IT Systems

Mechanical Turk Portal Mechanical Turk Portal

Turkers Turkers

© 2011 Amazon.com, Inc. or its Affiliates.

 Sentiment Analysis Process Setup

© 2011 Amazon.com, Inc. or its Affiliates.

 Brand Sentiment Coding Template

© 2011 Amazon.com, Inc. or its Affiliates.

© 2011 Amazon.com, Inc. or its Affiliates.
 John Hoskins, Amazon Web Services:
[email protected]
Max Yankelevich, FreedomOSS:
[email protected]
@amazonmturk

Facebook.com/amazonmturk

Mechanicalturk.typepad.com
© 2011 Amazon.com, Inc. or its Affiliates.
 Q&A
Your Questions. Answered.

© 2011 Amazon.com, Inc. or its Affiliates.