BDACA1617s2 - Lecture7

Recap Unsupervised Machine Learning Supervised Machine Learning Next meetings
Big Data and Automated Content Analysis
Week 7 – Monday
»Machine Learning«
Damian Trilling
d.c.trilling@uva.nl
@damian0604
www.damiantrilling.net
Afdeling Communicatiewetenschap
Universiteit van Amsterdam
20 March 2017
Big Data and Automated Content Analysis Damian Trilling

Today
1 Recap: Types of Automated Content Analysis
2 Unsupervised Machine Learning
PCA
LDA
3 Supervised Machine Learning
You have done it before!
Applications
An implementation
4 Next meetings

Recap: Types of Automated Content Analysis

Methodological approach
deductive inductive
Typical research interests
and content features
Common statistical
procedures
visibility analysis
sentiment analysis
subjectivity analysis
Counting and
Dictionary
Supervised
Machine Learning
Unsupervised
Machine Learning
frames
topics
gender bias
frames
topics
string comparisons
counting
support vector machines
naive Bayes
principal component analysis
cluster analysis
latent dirichlet allocation
semantic network analysis

Top-down vs. bottom-up
Some terminology
Supervised machine learning
You have a dataset with both
predictor and outcome
(independent and dependent
variables; features and labels) —
a labeled dataset.

Some terminology
a labeled dataset. Think of
regression: You measured x1, x2, x3
and you want to predict y, which you
also measured

Some terminology
a labeled dataset.
Unsupervised machine learning
You have no labels.

Some terminology
a labeled dataset.
You have no labels. (You did not
measure y)

Some terminology
You have no labels.
Again, you already know some
techniques to ﬁnd out how x1,
x2,. . . x_i co-occur from other
courses:
• Principal Component Analysis
(PCA)
• Cluster analysis
• . . .

inductive and bottom-up:
unsupervised machine learning

inductive and bottom-up:
unsupervised machine learning
(something you aready did in your Bachelor – no kidding.)

PCA
Principal Component Analysis? How does that ﬁt in here?

PCA
In fact, PCA is used everywhere, even in image compression

PCA
PCA in ACA
• Find out what word cooccur (inductive frame analysis)
• Basically, transform each document in a vector of word
frequencies and do a PCA

PCA
A so-called term-document-matrix
1 w1,w2,w3,w4,w5,w6 ...
2 text1, 2, 0, 0, 1, 2, 3 ...
3 text2, 0, 0, 1, 2, 3, 4 ...
4 text3, 9, 0, 1, 1, 0, 0 ...
5 ...

PCA
A so-called term-document-matrix
1 w1,w2,w3,w4,w5,w6 ...
2 text1, 2, 0, 0, 1, 2, 3 ...
3 text2, 0, 0, 1, 2, 3, 4 ...
4 text3, 9, 0, 1, 1, 0, 0 ...
5 ...
These can be simple counts, but also more advanced metrics, like
tf-idf scores (where you weigh the frequency by the number of
documents in which it occurs), cosine distances, etc.

PCA
PCA: implications and problems
• given a term-document matrix, easy to do with any tool
• probably extremely skewed distributions
• some problematic assumptions: does the goal of PCA, to ﬁnd
a solution in which one word loads on one component match
real life, where a word can belong to several topics or frames?

LDA
Enter topic modeling with Latent Dirichlet Allocation (LDA)

LDA
LDA, what’s that?
No mathematical details here, but the general idea
• There are k topics, T1. . . Tk
• Each document Di consists of a mixture of these topics,
e.g.80%T1, 15%T2, 0%T3, . . . 5%Tk
• On the next level, each topic consists of a speciﬁc probability
distribution of words
• Thus, based on the frequencies of words in Di , one can infer
its distribution of topics
• Note that LDA (like PCA) is a Bag-of-Words (BOW)
approach

LDA
Doing a LDA in Python
You can use gensim (Řehůřek & Sojka, 2010) for this.
Let us assume you have a list of lists of words (!) called texts:
1 articles=[’The tax deficit is higher than expected. This said xxx ...’,
’Germany won the World Cup. After a’]
2 texts=[art.split() for art in articles]
which looks like this:
1 [[’The’, ’tax’, ’deficit’, ’is’, ’higher’, ’than’, ’expected.’, ’This’,
’said’, ’xxx’, ’...’], [’Germany’, ’won’, ’the’, ’World’, ’Cup.’, ’
After’, ’a’]]
Řehůřek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. Proceedings of the
LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45–50. Valletta, Malta: ELRA.

1 from gensim import corpora, models
2
3 NTOPICS = 100
4 LDAOUTPUTFILE="topicscores.tsv"
5
6 # Create a BOW represenation of the texts
7 id2word = corpora.Dictionary(texts)
8 mm =[id2word.doc2bow(text) for text in texts]
9
10 # Train the LDA models.
11 lda = models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=
NTOPICS, alpha="auto")
12
13 # Print the topics.
14 for top in lda.print_topics(num_topics=NTOPICS, num_words=5):
15 print ("n",top)
16
17 print ("nFor further analysis, a dataset with the topic score for each
document is saved to",LDAOUTPUTFILE)
18
19 scoresperdoc=lda.inference(mm)
20
21 with open(LDAOUTPUTFILE,"w",encoding="utf-8") as fo:
22 for row in scoresperdoc[0]:
23 fo.write("t".join(["{:0.3f}".format(score) for score in row]))
24 fo.write("n")

LDA
Output: Topics (below) & topic scores (next slide)
1 0.069*fusie + 0.058*brussel + 0.045*europesecommissie + 0.036*europese +
0.023*overname
2 0.109*bank + 0.066*britse + 0.041*regering + 0.035*financien + 0.033*
minister
3 0.114*nederlandse + 0.106*nederland + 0.070*bedrijven + 0.042*rusland +
0.038*russische
4 0.093*nederlandsespoorwegen + 0.074*den + 0.036*jaar + 0.029*onderzoek +
0.027*raad
5 0.099*banen + 0.045*jaar + 0.045*productie + 0.036*ton + 0.029*aantal
6 0.041*grote + 0.038*bedrijven + 0.027*ondernemers + 0.023*goed + 0.015*
jaar
7 0.108*werknemers + 0.037*jongeren + 0.035*werkgevers + 0.029*jaar +
0.025*werk
8 0.171*bank + 0.122* + 0.041*klanten + 0.035*verzekeraar + 0.028*euro
9 0.162*banken + 0.055*bank + 0.039*centrale + 0.027*leningen + 0.024*
financiele
10 0.052*post + 0.042*media + 0.038*nieuwe + 0.034*netwerk + 0.025*
personeel
11 ...

LDA
Visualization with pyldavis
1 import pyLDAvis
2 import pyLDAvis.gensim
3 % first estiate gensim model, then:
4 vis_data = pyLDAvis.gensim.prepare(lda,mm,id2word)
5 pyLDAvis.display(vis_data)

predeﬁned categories, but no predeﬁned rules:
supervised machine learning

predeﬁned categories, but no predeﬁned rules:
supervised machine learning
(something you aready did in your Bachelor – no kidding.)

Recap: supervised vs. unsupervised
Unsupervised
• No manually coded data
• We want to identify patterns
or to make groups of most
similar cases

Unsupervised
similar cases
Example: We have a dataset of
Facebook-massages on an
organizations’ page. We use clustering
to group them and later interpret these
clusters (e.g., as complaints, questions,
praise, . . . )

Unsupervised
similar cases
praise, . . . )
Supervised
• We code a small dataset by
hand and use it to “train” a
machine
• The machine codes the rest

Unsupervised
similar cases
praise, . . . )
Supervised
• We code a small dataset by
hand and use it to “train” a
machine
• The machine codes the rest
Example: We have 2,000 of these
messages grouped into such categories
by human coders. We then use this
data to group all remaining messages
as well.

Regression

Regression
1 Based on your data, you estimate some regression equation
yi = α + β1xi1 + · · · + βpxip + εi

Regression
yi = α + β1xi1 + · · · + βpxip + εi
2 Even if you have some new unseen data, you can estimate
your expected outcome ˆy!

Regression
yi = α + β1xi1 + · · · + βpxip + εi
3 Example: You estimated a regression equation where y is
newspaper reading in days/week:
y = −.8 + .4 × man + .08 × age

Regression
yi = α + β1xi1 + · · · + βpxip + εi
3 Example: You estimated a regression equation where y is
newspaper reading in days/week:
y = −.8 + .4 × man + .08 × age
4 You could now calculate ˆy for a man of 20 years and a woman
of 40 years – even if no such person exists in your dataset:
ˆyman20 = −.8 + .4 × 1 + .08 × 20 = 1.2
ˆywoman40 = −.8 + .4 × 0 + .08 × 40 = 2.4

This is
Supervised Machine Learning!

. . . but. . .
• We will only use half (or another fraction) of our data to estimate the
model, so that we can use the other half to check if our
predictions match the manual coding (“labeled
data”,“annotated data” in SML-lingo)

. . . but. . .
• e.g., 2000 labeled cases, 1000 for training, 1000 for testing —
if successful, run on 100,000 unlabeled cases

. . . but. . .
• We use many more independent variables (“features”)

. . . but. . .
• We use many more independent variables (“features”)
• Typically, IVs are word frequencies (often weighted, e.g.
tf×idf) (⇒BOW-representation)

Applications
Applications

Applications
Applications
In other ﬁelds
A lot of diﬀerent applications
• from recognizing hand-written characters to recommendation
systems

Applications
Applications
In other fields
A lot of different applications
• from recognizing hand-written characters to recommendation
systems
In our field
It starts to get popular to measure latent variables
• frames
• topics

Applications
SML to code frames and topics
Some work by Burscher and colleagues
• Humans can code generic frames (human-interest, economic,
. . . )
• Humans can code topics from a pre-deﬁned list

Applications
. . . )
• But it is very hard to formulate an explicit rule
(as in: code as ’Human Interest’ if regular expression R is
matched)

Applications
. . . )
• But it is very hard to formulate an explicit rule
(as in: code as ’Human Interest’ if regular expression R is
matched)
⇒ This is where you need supervised machine learning!
Burscher, B., Odijk, D., Vliegenthart, R., De Rijke, M., & De Vreese, C. H. (2014). Teaching the computer to
code frames in news: Comparing two supervised machine learning approaches to frame analysis. Communication
Methods and Measures, 8(3), 190–206. doi:10.1080/19312458.2014.937527
Burscher, B., Vliegenthart, R., & De Vreese, C. H. (2015). Using supervised machine learning to code policy issues:
Can classiﬁers generalize across contexts? Annals of the American Academy of Political and Social Science, 659(1),
122–131.

http://commons.wikimedia.org/wiki/File:Precisionrecall.svg
Some measures of accuracy
• Recall
• Precision
• F1 = 2 · precision·recall
precision+recall
• AUC (Area under curve)
[0, 1], 0.5 = random
guessing

Applications
What does this mean for our research?

Applications
What does this mean for our research?
It we have 2,000 documents with manually coded frames and
topics. . .
• we can use them to train a SML classiﬁer
• which can code an unlimited number of new documents
• with an acceptable accuracy
Some easier tasks even need only 500 training documents, see Hopkins, D. J., & King, G. (2010). A method of
automated nonparametric content analysis for social science. American Journal of Political Science, 54(1), 229–247.

An implementation
An implementation
Let’s say we have a list of tuples with movie reviews and their
rating:
1 reviews=[("This is a great movie",1),("Bad movie",-1), ... ...]
And a second list with an identical structure:
1 test=[("Not that good",-1),("Nice film",1), ... ...]
Both are drawn from the same population, it is pure chance
whether a speciﬁc review is on the one list or the other.
Based on an example from http://blog.dataquest.io/blog/naive-bayes-movies/

An implementation
Training a A Naïve Bayes Classiﬁer
1 from sklearn.naive_bayes import MultinomialNB
2 from sklearn.feature_extraction.text import CountVectorizer
3 from sklearn import metrics
4
5 # This is just an efficient way of computing word counts
6 vectorizer = CountVectorizer(stop_words=’english’)
7 train_features = vectorizer.fit_transform([r[0] for r in reviews])
8 test_features = vectorizer.transform([r[0] for r in test])
9
10 # Fit a naive bayes model to the training data.
11 nb = MultinomialNB()
12 nb.fit(train_features, [r[1] for r in reviews])
13
14 # Now we can use the model to predict classifications for our test
features.
15 predictions = nb.predict(test_features)
16 actual=[r[1] for r in test]
17
18 # Compute the error.
19 fpr, tpr, thresholds = metrics.roc_curve(actual, predictions, pos_label
=1)
20 print("Multinomal naive bayes AUC: {0}".format(metrics.auc(fpr, tpr)))

An implementation
And it works!
Using 50,000 IMDB movies that are classiﬁed as either negative or
positive,
• I created a list with 25,000 training tuples and another one
with 25,000 test tuples and
• trained a classiﬁer
• that achieved an AUC of .82.
Dataset obtained from http://ai.stanford.edu/~amaas/data/sentiment, Maas, A.L., Daly, R.E., Pham, P.T.,
Huang, D., Ng, A.Y., & Potts, C. (2011). Learning word vectors for sentiment analysis. 49th Annual Meeting of
the Association for Computational Linguistics (ACL 2011)

An implementation
Playing around with new data
1 newdata=vectorizer.transform(["What a crappy movie! It sucks!", "This is
awsome. I liked this movie a lot, fantastic actors","I would not
recomment it to anyone.", "Enjoyed it a lot"])
2 predictions = nb.predict(newdata)
3 print(predictions)
This returns, as you would expect and hope:
1 [-1 1 -1 1]

An implementation
But we can do even better
We can use different vectorizers and different classifiers.

An implementation
Diﬀerent vectorizers
• CountVectorizer (=simple word counts)
• TﬁdfVectorizer (word counts (“term frequency”) weighted by
number of documents in which the word occurs at all
(“inverse document frequency”))
• additional options: stopwords, thresholds for minimum
frequencies etc.

An implementation
Diﬀerent classiﬁers
• Naïve Bayes
• Logistic Regression
• Support Vector Machine (SVM)
• . . .
Typical approach: Find out which setup performs best (see
example source code in the book).

Next (last. . . ) meetings
Wednesday
Playing around with machine learning! Choose what you ﬁnd
interesting (SML or LDA).
Monday: Guest lectures
• Joanna Strycharz: From research questions to analysis:
Integration of data collection, preparation and statystical tests
in Python
• Marthe Möller: Automated content analyses in entertainment
communication: A YouTube study
Wednesday
Final chance for questions regarding ﬁnal project

BDACA1617s2 - Lecture7

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to BDACA1617s2 - Lecture7 (20)

More from Department of Communication Science, University of Amsterdam (10)

Recently uploaded (20)

BDACA1617s2 - Lecture7