Semantic Search overview at SSSW 2012

Editor's Notes

• #8: In fact, some of these searches are so hard that the users don’t even try them anymore
• #9: With ads, the situation is even worse due to the sparsity problem. Note how poor the ads are…
• #11: Search is a form of content aggregation
• #19: Facebook invited, but continues to pursue OGP
• #24: Semantic search can be seen as a retrieval paradigm Centered on the use of semantics Incorporates the semantics entailed by the query and (or) the resources into the matching process, it essentially performs semantic search.
• #25: Bar celona
• #32: Close to the topic of keyword-search in databases, except knowledge-bases have a schema-oblivious design Different papers assume vastly different query needs even on the same type of data
• #41: Miss structural information in texts Hyperlinks Linguistic structure Positional information
• #42: - Real world objects
• #46: SELECT Returns all, or a subset of, the variables bound in a query pattern match. CONSTRUCT Returns an RDF graph constructed by substituting variables in a set of triple templates. ASK Returns a boolean indicating whether a query pattern matches or not. DESCRIBE Returns an RDF graph that describes the resources found. Graph patterns are defined recursively. A graph pattern may have zero or more optional graph patterns, and any part of a query pattern may have an optional part. In this example, there are two optional graph patterns. Section 6 introduces the ability to make portions of a query optional; Section 7 introduces the ability to express the disjunction of alternative graph patterns; and Section 8 introduces the ability to constrain portions of a query to particular source graphs. Section 8 also presents SPARQL's mechanism for defining the source graphs for a query. Basic graph patterns allow applications to make queries where the entire query pattern must match for there to be a solution. For every solution of a query containing only group graph patterns with at least one basic graph pattern, every variable is bound to an RDF Term in a solution. However, regular, complete structures cannot be assumed in all RDF graphs. It is useful to be able to have queries that allow information to be added to the solution where the information is available, but do not reject the solution because some part of the query pattern does not match. Optional matching provides this facility: if the optional part does not match, it creates no bindings but does not eliminate the solution. The UNION pattern combines graph patterns; each alternative possibility can contain more than one triple pattern: SPARQL provides a means of combining graph patterns so that one of several alternative graph patterns may match. If more than one of the alternatives matches, all the possible pattern solutions are found.
• #49: Web data: Text + Linked Data + Semi-structured RDF + Hybrid data that can be conceived as forming data graphs Hear abour bob and alice all the time (in computer science literatures), want to find out more… build Semantic Web search engine. To address complex information needs by exploiting Web data: - Query as a set of constrains Match structured data Match text
• #50: - Less than 5 percent of IR papers deal with query processing and the aspect of efficiency
• #51: Partitioning has impact on performance!) Blocking: iterator-based approaches Non-blocking: good for streaming, good we cannot wait for some parts of the results to be completely worked-off Link data: cannot wait for sources, (some are slower then other) thus better to push data into query processing as the they come instead of pulling data and wait (busy waiting) Top-k:
• #53: -phrase patterns (e.g., “X is the capital of Y”) for large scale extraction. Such simple patterns, when coupled with the richness and redundancy of theWeb, can be very useful in scraping millions or even billions of facts from the Web. - patterns: Matched when keywords or data types Xi appear in sequence. Matched if all keywords/data types/patterns appear within an m-words window. For extraction: relation patterns For text search: entity patterns -When not assuming that all relevant data can be extracted such matching against text still needed: Hybrid search
• #54: -phrase patterns (e.g., “X is the capital of Y”) for large scale extraction. Such simple patterns, when coupled with the richness and redundancy of theWeb, can be very useful in scraping millions or even billions of facts from the Web. - patterns: Matched when keywords or data types Xi appear in sequence. Matched if all keywords/data types/patterns appear within an m-words window. For extraction: relation patterns For text search: entity patterns -When not assuming that all relevant data can be extracted such matching against text still needed: Hybrid search
• #55: Given some materialized indexes  no joins at all Given sorted inputs  sorted merge join
• #56: Given some materialized indexes  no joins at all Given sorted inputs  sorted merge join join
• #57: Every task is a challenge of itself, some more some less well elaborated There are separate challenges for every problems
• #59: Approximate  many results  need ranking Ranking also needed in the case where qp is complete and sound, but queries and data representation so imprecise such that we have to deal with too many results
• #60: Web data: Text + Linked Data + Semi-structured RDF + Hybrid data that can be conceived as forming data graphs Hear abour bob and alice all the time (in computer science literatures), want to find out more… build Semantic Web search engine. To address complex information needs by exploiting Web data: - Query as a set of constrains Match structured data Match text
• #61: Syntactic works vs. works at Semantic works vs. employ
• #62: Aboutness (Semantics employ at matching not ranking!)
• #63: text which contains a large mentions of “Berlin” is likely to be about “Berlin” i is more likely to be the correct interpretation of K when terms in K co-occur in a large number of context (bag of words) associated with i “ Berlin and apartment” more often co-occur in the geographic location context/topic than in the context of people “ Berlin and apartment” more often co-occur in the geographic location context/topic than in the context of people
• #64: - Underlying most research on probabilistic models of Information Retrieval is PRP - are ranked by the posterior probability that they belong to the relevant class . . Robertson [19] also shows that it is equivalent to rank the documents by the odds of their being observed in the relevant class: - Here represents the class of documents relevant to user’s query, and is the class of non-relevant documents. - The P(D|R)/ estimation of differs in various models The Binary Independence Model [17, 23] treats each document as a binary vector over the vocabulary space, ignoring word frequencies. - 2-Poisson Model [18] goes a step further in modeling term frequencies in documents according to a mixture of two Poisson distributions. - Here are the probabilities of the word being present document sampled from the relevant class. These probabilities estimated using heuristic techniques in the absence of relevance information. - Let’s assume that the query words and the words in relevant documents are sampled identically and independently from a unigram distribution ( finite universe of unigram distributions from which we could sample)
• #65: In the multinomial unigram model, each document D is represented as a multinomial probability distribution PðtjhDÞ over all the terms t in the vocabulary. a more general and flexible retrieval model can be obtained by using a comparison of two language models as the basis for ranking. Several authors proposed the use of the Kullback–Leibler (KL)-divergence for ranking, since it is a well established measure for the comparison of probability distributions with some intuitive properties—it always has a non-negative value and equal distributions receive a zero divergence value (Lafferty et al., 2001; Ng, 2001; Xu & Croft, 1999). Using KL-divergence, documents are scored by measuring the divergence between a query model hQ and each document model hD. Since we want to assign a high score for high similarity and a low score for low similarity, the KL-divergence is negated for ranking purposes. where V denotes the set of all terms used in all documents in the collection. KL-divergence is also known as the relative entropy, which is defined as the cross-entropy of the observed distribution (in this case the query) as if it was generated by a reference distribution (in this case the document) minus the entropy of the observed distribution.
• #66: Here, n(v;D) is the count of the word v in the document, j d j is the document length, and P(v j C) is the background probability of v.
• #67: Probabiligy the term generated by the overall colleciton
• #68: Here P(w|Ojk) is the probability of generating w by the th j field of record k. P(w|Ojk) can be computed by treating each Ojk as a document, The generative story is that the document needs to generate the f1 field models, and then each f1 model will generate the query term q1 and the structural term #combine[f2]( q2 ). Next, each f2 field model in each f1 model will generate the query term q2
• #69: From the schema graph G(VG, EG), we create the authority transfer schema graph GA (VG, EA ) to reflect the authority flow through the edges of the graph. This may be either a trial and error process, until we are satisfied with the quality of the results, or a domain expert’s task. In particular, for each edge eG = (u → v) of EG, two authority transfer edges, e f G = (u → v) and e b G = (v → u) are created. The two edges carry the label of the schema graph edge and, in addition, each one is annotated with a (potentially different) authority transfer rate - α(e f G) and α(e b G) Correspondingly The motivation for defining two edges for each edge of the schema graph is that authority potentially flows in both directions and not only in the direction that appears in the schema. For example, a paper passes its authority to its authors and vice versa. Notice however, that the authority flow in each direction (defined by the authority transfer rate) may not be the same. For example, a paper that is cited by important papers is clearly important but citing important papers does not make a paper important. Given a data graph D(VD, ED) that conforms to an authority transfer schema graph GA (VG, EA ), ObjectRank derives an authority transfer data graph DA (VD, EA D) (Figure 5) as follows