Introduction to Information Retrieval

Editor's Notes

• #11: Not only the data is different, also the queries, and the results we get from it!
• #13: To the surprise of many, the search box has become the preferred method of information access. Customers ask: Why can’t I search my database in the same way?
• #17: Archie is a tool for indexing FTP archives, allowing people to find specific files. It is considered to be the first Internet search engine. In the summer of 1993, no search engine existed for the web, just catalog One of the first "all text" crawler-based search engines was WebCrawler, which came out in 1994. Unlike its predecessors, it allowed users to search for any word in any webpage, which has become the standard for all major search engines since. It was also the first one widely known by the public. Also in 1994, Lycos (which started at Carnegie Mellon University) was launched and became a major commercial endeavor.
• #18: In 1996, Netscape was looking to give a single search engine an exclusive deal as the featured search engine on Netscape's web browser. There was so much interest that instead Netscape struck deals with five of the major search engines: for $5 million a year, each search engine would be in rotation on the Netscape search engine page. The five engines were Yahoo!, Magellan, Lycos, Infoseek, and Excite.[7][8] Google adopted the idea of selling search terms in 1998, from a small search engine company named goto.com. This move had a significant effect on the SE business, which went from struggling to one of the most profitable businesses in the internet.[6]
• #20: Aardvark was a social search service that connected users live with friends or friends-of-friends who were able to answer their questions, also known as a knowledge market. Bought by google 2010 Kaltix Corp., commonly known as Kaltix is a personalized search engine company founded at Stanford University in June 2003 by Sep Kamvar, Taher Haveliwala and Glen Jeh.[1][2] It was acquired by Google in September 2003.
• #44: How do we communicate with search engines
• #45: Information needs must be expressed as a query – But users don’t often know what they want ASK   Hypothesis Belkin et al (1982) Proposed a model called Anomalous State of Knowledge ASK  hypothesis: – difficult for people to define exactly what their information need is, because that information is a gap in their knowledge - Search Engines should look for information that fills those gaps Interesting ideas, little practical impact (yet)
• #49: Under specified Ambiguous Context sensitive  represent different types of search –  E.g. decision making –  background search –  fact search
• #50: Need to have fairly deep knowledge... –  What sites are possible –  What’s in a given site (what’s likely to be there) –  Authority of source / site –  Index structure (time, place, person, ...) what kinds of searches? –  How to read a SERP critically
• #51: Commonplace book
• #52: Start with the simplest search you can think of: [ upper lip indentation ] If it’s not right, you can always modify it. • When I did this, I clicked on the first result, which took me to Yahoo Answers. There’s a nice article there about something called the philtrum.
• #53: Ghost town vs abandoned 1750 Search for images with creative commons attributions
• #59: The need is verbalized mentally
• #60: Queries and documents must share a (at least comparable if not the same) representation
• #64: SCC – single connected component IN – pages not discovered yet OUT – sites that contain only in-host link Tendrils – can’t reach or be reached from the SCC
• #74: creation of indefinitely deep directory structures like http://foo.com/bar/foo/bar/foo/bar/foo/bar/..... dynamic pages like calendars that produce an infinite number of pages for a web crawler to follow. pages filled with a large number of characters, crashing the lexical analyzer parsing the page. pages with session-id's based on required cookies.
• #80: Data: ; this type of data is conventionally dealt with a database management system. Structure: With this view, documents are not treated as flat entities, so a document and its components (e.g. sections) can be retrieved
• #83: How do we arrive to the content representation of a document?
• #85: Nontrivial issues. Requires some design decisions.
• #86: Nontrivial issues. Requires some design decisions. Matches are then more likely to be relevant, and since the documents are smaller it will be much easier for the user to find the relevant passages in the document. But why stop there? We could treat individual sentences as mini-documents. It becomes clear that there is a precision/recall tradeoff here. If the units get too small, we are likely to miss important passages because terms were distributed over several mini-documents, while if units are too large we tend to get spurious matches and the relevant information is hard for the user to find. The problems with large document units can be alleviated by use of explicit or implicit proximity search
• #88: A simple strategy is to just split on all non-alphanumeric characters – bad you always want to do the exact same tokenization of document and query words, generally by processing queries with the same tokenize Conceptually, splitting on white space can also split what should be re- garded as a single token. This occurs most commonly with names (San Fran- cisco, Los Angeles) but also with borrowed foreign phrases (au fait)
• #89: Index numbers -> (One answer is using n-grams: IIR ch. 3)
• #90: Methods of word segmentation vary from having a large vocabulary and taking the longest vocabulary match with some heuristics for unknown words to the use of machine learning sequence models, such as hidden Markov models or condi- tional random fields, trained over hand-segmented words
• #91: No unique tokenization + completely different interpretation of a sequence depending on where you split
• #93: Nevertheless: “Google ignores common words and characters such as where, the, how, and other digits and letters which slow down your search without improving the results.” (Though you can explicitly ask for them to remain.)
• #94: Token normalization is the process of canonicalizing tokens so that matches occur despite superficial differences in the character sequences of the to- kens.4 The most standard way to normalize is to implicitly create equivalence classes, which are normally named after one member of the set. For instance, if the tokens anti-discriminatory and antidiscriminatory are both mapped onto the term antidiscriminatory, in both the document text and queries, then searches for one term will retrieve documents that contain either. The advantage of just using mapping rules that remove characters like hyphens is that the equivalence classing to be done is implicit, rather than being fully calculated in advance: the terms that happen to become identical as the result of these rules are the equivalence classes. It is only easy to write rules of this sort that remove characters. Since the equivalence classes are implicit, it is not obvious when you might want to add characters. For instance, it would be hard to know to turn antidiscriminatory into anti-discriminatory.
• #95: An alternative to creating equivalence classes is to maintain relations between not normalized tokens. This method can be extended to hand-constructed lists of synonyms such as car and automobile, a topic we discuss further in
• #96: Too much equivalence class
• #97: Why not the reverse?
• #101: Also stemmers based on N-grams-based For example trigrams: information => {inf, nfo, for, etc}
• #104: careses parties separational -> separate factional -> faction
• #111: Compression Cache pressure
• #121: The distribution of term frequencies is similar for different texts of significant large size.
• #122: Heaps’ law gives the vocabulary size in collections.
• #133: Positional indexes are helpful, but we’ll ignore them for now
• #147: (Salton & McGill 1983)
• #153: The classifier that assigns a vector x to the class with the highest posterior is called the Bayes classifier. The error associated with this classifier is called the Bayes error. This is the lowest possible error rate for any classifier over the distribution of all examples and for a chosen hypothesis space
• #154: A complete probability distribution over documents −  defines likelihood for any possible document d (observation) −  P(relevant) via P(document): PR∣d∝Pd∣RPR −  can “generate” synthetic documents willsharesomepropertiesoftheoriginalcollection Not all IR Models do this – possible to estimate p(R|d) directly – log regression Assumptions: one relevance value for every word w Words are conditionally independent given R – false, but allows to lower down the number of parameters All words absent are equally likely to be observed in relevant and not relevant classes
• #156: One relevance status value per word empty document (all words absent) is equally likely to be observed in relevant and non-relevant classes (provides a natural zero) - practical reason, only score terms that appear in the query (TAT)
• #159: Doesn’t model word dependence. Doesn’t account for document length. Doesn’t model word frequencies
• #160: Now D_t = d_t account for the number of times we observe the term in the document (we have a vector of frequencies)
• #165: Can we seen as a probabilisitic automata They originate from probabilistic models of language gen- eration developed for automatic speech recognition systems in the early 1980's (see e.g. Rabiner 1990). Automatic speech recognition systems combine prob- abilities of two distinct models: the acoustic model, and the language model. The acoustic model might for instance produce the following candidate texts in decreasing order of probability: \food born thing", \good corn sing", \mood morning", and \good morning". Now, the language model would determine that the phrase \good morning" is much more probable, i.e., it occurs more frequently in English than the other phrases. When combined with the acoustic model, the system is able to decide that \good morning" was the most likely utterance, thereby increasing the system's performance. For information retrieval, language models are built for each document. By following this approach, the language model of the book you are reading now would assign an exceptionally high probability to the word \retrieval" indicating that this book would be a good candidate for retrieval if the query contains this word.
• #166: For some applications we want all this highly probable P3 In IR P1=P2
• #169: Veto terms Original multiple bernoulli, multinomial widely used now accountsformultiplewordoccurrencesinthequery(primitive)– wellunderstood:lotsofresearchinrelatedfields(andnowinIR) – possibilityforintegrationwithASR/MT/NLP(sameeventspace)
• #171: Discounting methods Problem with all discounting methods: – discounting treats unseen words equally (add or subtract ε) – somewordsaremorefrequentthanothers Essentially, the data model and retrieval function are one and the same
• #173: Different ways of smoothing, dirichlet priors smoothing particularly popular