Intro to Vectorization Concepts - GaTech cse6242
Download as PPTX, PDF6 likes4,415 views
Vectorization is the process of converting text into numeric vectors that can be used by machine learning algorithms. There are several common techniques for vectorization, including the bag-of-words model, TF-IDF, and n-grams. The bag-of-words model represents documents as vectors counting the number of times each word appears. TF-IDF improves on this by weighting words based on their frequency in documents and inverse frequency in the corpus. N-grams consider sequences of words, such as bigrams like "Coca Cola", as single units. Kernel hashing allows vectorization in a single pass by mapping words to a fixed-sized vector using a hash function.
![Wait. What’s a Vector Again?
• An array of floating point numbers
• Represents data
– Text
– Audio
– Image
• Example:
–[ 1.0, 0.0, 1.0, 0.5 ]](https://image.slidesharecdn.com/vectorization20140915-140916135606-phpapp02/85/Intro-to-Vectorization-Concepts-GaTech-cse6242-6-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (15)
Ad
Similar to Intro to Vectorization Concepts - GaTech cse6242 (20)
Ad
More from Josh Patterson (20)
Recently uploaded (20)
Intro to Vectorization Concepts - GaTech cse6242
- 1. Vectorization Core Concepts in Data Mining
- 2. Topic Index • Why Vectorization? • Vector Space Model • Bag of Words • TF-IDF • N-Grams • Kernel Hashing
- 3. “How is it possible for a slow, tiny brain, whether biological or electronic, to perceive, understand, predict, and manipulate a world far larger and more complicated than itself?” --- Peter Norvig, “Artificial Intelligence: A Modern Approach” WHY VECTORIZATION?
- 4. Classic Scenario: “Classify some tweets for positive vs negative sentiment”
- 5. What Needs to Happen? • Need each tweet as some structure that can be fed to a learning algorithm – To represent the knowledge of “negative” vs “positive” tweet • How does that happen? – We need to take the raw text and convert it into what is called a “vector” • Vector relates to the fundamentals of linear algebra – “Solving sets of linear equations”
- 6. Wait. What’s a Vector Again? • An array of floating point numbers • Represents data – Text – Audio – Image • Example: –[ 1.0, 0.0, 1.0, 0.5 ]
- 7. “I am putting myself to the fullest possible use, which is all I think that any conscious entity can ever hope to do.” --- Hal, 2001 VECTOR SPACE MODEL
- 8. Vector Space Model • Common way of vectorizing text – every possible word is mapped to a specific integer • If we have a large enough array then every word fits into a unique slot in the array – value at that index is the number of the times the word occurs • Most often our array size is less than our corpus vocabulary – so we have to have a “vectorization strategy” to account for this
- 9. Text Can Include Several Stages • Sentence Segmentation – can skip straight to tokenization depending on use case • Tokenization – find individual words • Lemmatization – finding the base or stem of words • Removing Stop words – “the”, “and”, etc • Vectorization – we take the output of the process and make an array of floating point values
- 10. “A man who carries a cat by the tail learns something he can learn in no other way.” --- Mark Twain TEXT VECTORIZATION STRATEGIES
- 11. Bag of Words • A group of words or a document is represented as a bag – or “multi-set” of its words • Bag of words is a list of words and their word counts – simplest vector model – but can end up using a lot of columns due to number of words involved. • Grammar and word ordering is ignored – but we still track how many times the word occurs in the document • has been used most frequently in the document classification – and information retrieval domains.
- 12. Term frequency inverse document frequency (TF-IDF) • Fixes some issues with “bag of words” • allows us to leverage the information about how often a word occurs in a document (TF) – while considering the frequency of the word in the corpus to control for the facet that some words will be more common than others (IDF) • more accurate than the basic bag of words model – but computationally more expensive
- 13. TF-IDF Formula • wi = TFi * IDFi • TFi(t) – = (Number of times term t appears in a document) / (Total number of terms in the document). • IDFi = log (N / Dfi) – N is total documents in corpus – Dfi is documents containing the term t
- 14. N-grams • A group of words in a sequence is called an n-gram • A single word can be called a unigram • Two words like “Coca Cola” can be considered a single unit and called a bigram • Three and more terms can be called trigrams, 4-grams, 5-grams and so on and so forth
- 15. N-Grams Usage • If we combine the unigrams and bigrams from a document and generate weights using TF-IDF – will end up with large vectors with many meaningless bigrams – having large weights on account of their large IDF • Can pass n-gram through something called a log-likelihood test – which can determine whether two words occurred together rather by chance, or because they form a significant unit – It selects the most significant ones and prunes away the least significant ones • Using the remaining n-grams, TF-IDF weighting scheme is applied and vectors are produced – In this way, significant bigrams like “Coca Cola” can be more properly accounted for in a TF-IDF weighting.
- 16. Kernel Hashing • When we want to vectorize the data in a single pass – making it a “just in time” vectorizer. • Can be used when we want to vectorize text right before we feed it to our learning algorithm. • We come up with a fixed sized vector that is typically smaller than the total possible words that we could index or vectorize – Then we use a hash function to create an index into the vector.
- 17. More Kernel Hashing • Advantage to use kernel hashing is that we don’t need the pre-cursor pass like we do with TF-IDF – but we run the risk of having collisions between words • The reality is that these collisions occur very infrequently – and don’t have a noticeable impact on learning performance • For more reading: – http://jeremydhoon.github.com/2013/03/19/abusing-hash- kernels-for-wildly-unprincipled-machine-learning/