Semantic Search using RDF Metadata (SemTech 2005)

Semantic Search Using RDF Metadata Semantic Technology Conference 2005 8 March 2005 Bradley P. Allen Siderean Software, Inc.

Overview Semantic search Motivation Enterprise adoption Semantic search using RDF Examples Lessons Directions

Problem “ We have to understand what information we have and organize it,’ says [Santa Clara Co. CIO] Ajmani, who estimates that saving each employee an hour a month spent looking for information would save millions of dollars.” [Information Week, 1/19/04] “… typical enterprise floundering in a sea of information … too many repositories, each with its own set of applications.” [IDC, 2004] “ The search capabilities on most company and content-oriented Web sites are as bad now as they were several years ago.” [eWeek, 1/26/04]

Portal-driven demand for a better solution “ A portal provides an integrated information source for our internal process users or external customers” “ Now we have to architect the information related to business processes differently to search across multiple repositories” But they lack tools and applications that support this

Current solutions Enterprise search, portals, knowledge management and content management systems lashed up in ad hoc architectures Doesn’t unify data and content Doesn’t provide context or scope Too many results (requires searching the answer to the original search)

Why semantic search? Explicitly represented knowledge can Unify access to both content and data Create context and frames of reference Intellectual contributions that inform the search process must be captured The answer should include the question

Semantic search – some definitions Search: the process of retrieving objects matching a given query Semantic search: Search that uses an explicit representation of knowledge to retrieve, organize or display objects matching a query Search that transparently renders human insight into the nature of matches

Benefits in the enterprise Addresses pervasive frustration with enterprise search Let users Find high-value information quickly Add more value to it, and Share it with others Aligns information to business needs

Roots Parametric search Query by example Retrieval by reformulation Rabbit, Argon Work in existing enterprise search and knowledge management Autonomy, Semio

Semantic search requires metadata Ontologies Specifications of how to represent classes, instances and their properties Sometimes called “vocabularies” Controlled vocabularies Terms for saying what something is about Also called “taxonomies” and “thesauri” Instances Descriptions of resources Application profiles Specifications of which classes and properties are useful and how they are to be used in an application

Current metadata solutions are costly Much custom development done Not easy to tag or incorporate content into the desired structures No easy way for groups creating the vocabularies to deliver them to production environments Perceived lack of tools Point solutions not well integrated Existing platform solutions closed

Metadata in today’s enterprises From thirty interviews conducted with Fortune 1000 organizations during Fall 2004 Use of metadata not yet widespread but emerging Understanding varies widely across enterprises Three basic approaches Top down, bottom up, and give up

Approach: top down CEO says “We must be an information-driven company” “ Corporate controlled vocabulary that all divisions will use” Typically based on Dublin Core Used for subject tagging The effort is multi-year, ROI hard to track, and may not be implemented or adopted widely

Approach: bottom up Groups determine their vocabulary while describing their process Often in a collaboration environment Light tagging of content when it is created or when the content is published to a portal Again, based on Dublin Core and their own controlled vocabularies

Approach: give up Assumption: too difficult to create metadata from existing content “ We can’t ever hope to organize this morass of content, so let’s put in a search appliance like Google” “ Our internal needs are like the public internet and users are familiar with Google searches” But still feel that metadata would improve matters, particularly within business units

Don’t give up! RDF can make metadata use easier and less costly An open standard for metadata reduces cost and avoids technology and vendor lock-in A “universal solvent” for data and content A platform for reuse and sharing

Building semantic search systems with RDF Define/reuse ontologies expressed in RDF(S) Classes for defining instances and controlled vocabularies Properties for facets and additional attributes Import/transform instances into an RDF representation Resources referred to via URIs Content and controlled vocabularies Write application profiles in terms of RDF

Types of semantic search in RDF Searching for RDF Swoogle Adding value to search using RDF TAP, FOAFNaut Searching resources using RDF Edutella, Seamark

Swoogle: Searching for RDF Crawling for SW documents Leverages Google indexing And structure of key document types Searching for ontologies and instance data Mostly relevant to people bulding semantic applications rather than general users

TAP: Adding value to search using RDF Layering “related items” on top of traditional Web search Arm’s length integration and value-add for traditional Web search

FOAFNaut: Adding value to search using RDF Specialized search and visualization over FOAF networks Introducing the notion of social aspects of finding information

Edutella: Searching resources using RDF P2P architecture federating collections of learning objects Work on distributing RDF queries using schema information RDF as a more natural representation for learning objects than IEEE LOM

Seamark: Searching resources using RDF Using ontologies and taxonomies to define navigation over specific collections First implementation of faceted navigation using RDF

Faceted navigation as a type of semantic search Metadata may be faceted, i.e., includes properties whose ranges form a near-orthogonal set of controlled vocabularies Creator: Dickens, Charles Subject: Arsenic, Antimony Location: World > U.S. > California > Venice Facets form a frame of reference for information overview, access and discovery Other properties serve as landmarks and cues

Case study: DC 2003 Online Proceedings Further the goals of the Dublin Core Metadata Initiative (DCMI) by providing DC-centric faceted navigation of online proceedings

Project timeline July 2003 Initial experiment using DC 2002 site August 2003 Initial proposal to DCMI Iterative prototyping involving Selection and development of ontologies Generation of instance metadata Specification of application profile Conversion of DC2003 dataset into navigable RDF Elapsed time to implement: 1 day September 2003 Design and editing of controlled vocabulary Final iterations on site pages Launch at conference

Ontology Reused ontologies and metadata vocabularies Papers and posters: Dublin Core Creators: Friend Of A Friend (FOAF) Subjects: Thesaurus Interchange Format (TIF) Added relatively few properties and classes in a conference ontology Events Tracks

Ontology for conferences <s:Class rdf:about="&dcconf;Event"> <s:label>Presentation</s:label> </s:Class> <s:Class rdf:about="&dcconf;Paper"> <s:label>Paper</s:label> <s:subClassOf rdf:resource="&dcconf;Event"/> </s:Class> <s:Class rdf:about="&dcconf;Track"> <s:label>Conference Track</s:label> </s:Class> <rdf:Property rdf:about="&dcconf;track"> <s:label>Track</s:label> <s:comment>The track that the given paper is in.</s:comment> <s:domain rdf:resource="&dcconf;Event" /> <s:range rdf:resource="&dcconf;Track" /> </rdf:Property>

Controlled vocabulary Author-assigned keywords used as source materials Combined author-assigned with editorial judgment about the CV terms and structure

Wrapping author-assigned keywords <tif:Term rdf:about="&dcconf2003;Relational_Database"> <tif:value>Relational Database</tif:value> <tifs:USE rdf:resource="&dcconf2003;Relational_Databases" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;relationship_metadata"> <tif:value>Relationship metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;requirements"> <tif:value>Requirements</tif:value> </tif:Term> <tif:Term rdf:about="&dcconf2003;resource_discovery"> <tif:value>Resource discovery</tif:value> <tifs:BT rdf:resource="&dcconf2003;Discovery" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;resource-level_metadata"> <tif:value>Resource-level metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;SCORM"> <tif:value>SCORM</tif:value> <tifs:USE rdf:resource="&dcconf2003;Sharable_Content_Object_Reference_Model_SCORM" /> </tif:Term>

Adding editorial control <tif:Term rdf:about="&dcconf2003;Domain_Metadata"> <tif:value>Domain Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Applications" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;Governments"> <tif:value>Governments</tif:value> <tifs:BT rdf:resource="&dcconf2003;Organizations_and_Domains" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;Federal_Geographic_Data_Committee_Metadata"> <tif:value>Federal Geographic Data Committee Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Governments" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;Geospatial_Metadata"> <tif:value>Geospatial Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Organizations_and_Domains" /> </tif:Term> <tif:Term rdf:about="&dcconf2003;Government_Agency_Metadata"> <tif:value>Government Agency Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Governments" /> </tif:Term>

Instance metadata Paper and poster metadata automatically extracted from author submissions Ad hoc Perl script Manual review and cleanup of generated RDF Mostly Dublin Core with some application-specific properties Creator and organization metadata manually collated from paper and poster metadata Represented in FOAF (but not in the manner in which FOAF is typically used)

Papers and posters <dcconf:Paper rdf:about="http://www.siderean.com/dc2003/103_paper-22.pdf"> <seamark:texturl>http://www.siderean.com/dc2003/103_paper-22.pdf</seamark:texturl> <rdf:type rdf:resource="&dcconf;Event"/> <dcconf:track rdf:resource="&dcconf;Interoperability" /> <dc:title>Two Paths to Interoperable Metadata</dc:title> <dc:creator rdf:resource="&dcconf;Godby_Carol" /> <dc:creator rdf:resource="&dcconf;Smith_Devon" /> <dc:creator rdf:resource="&dcconf;Childress_Eric" /> <dc:description> This paper describes a prototype for a Web service that translates between pairs of metadata schemas. Despite a current trend toward encoding in XML and XSLT, we present arguments for a design that features a more distinct separation of syntax from semantics. The result is a system that auomates routine processes, has a well-defined place for human input, and achieves a clean separation of the document data model, the document translations, and the machinery of the application. </dc:description> <dc:subject rdf:resource="&dcconf2003;metadata_schema_translation" /> <dcconf:authorKeyword rdf:resource="&dcconf2003;metadata_schema_translation" /> <dc:subject rdf:resource="&dcconf2003;Web_services" /> <dcconf:authorKeyword rdf:resource="&dcconf2003;Web_services" /> <dc:subject rdf:resource="&dcconf2003;communities_of_practice" /> <dcconf:authorKeyword rdf:resource="&dcconf2003;communities_of_practice" /> </dcconf:Paper>

Creators and organizations <foaf:Person rdf:about="&dcconf;Greenberg_Jane"> <foaf:name>Greenberg, Jane</foaf:name> <foaf:mbox rdf:resource="mailto:janeg@ils.unc.edu" /> <foaf:memberOf rdf:resource="&dcconf;University_of_North_Carolina_at_Chapel_Hill" /> <foaf:publication rdf:resource="http://www.siderean.com/dc2003/202_Paper82-color-NEW.pdf" /> </foaf:Person> <foaf:Organization rdf:about="&dcconf;University_of_North_Carolina_at_Chapel_Hill"> <foaf:name>University of North Carolina at Chapel Hill, USA</foaf:name> <foaf:member rdf:resource="&dcconf;Greenberg_Jane" /> <foaf:member rdf:resource="&dcconf;Crystal_Abe" /> </foaf:Organization>

Application profile Expressed in XRBR (XML For Retrieval By Reformulation) Specifies a view over (possibly heterogeneous) RDF schemas with hints as to its interpretation and use for faceted navigation Provides a language for query reformulation and refinement in the context of navigation Query: “give me all resources where…” + advice Response: result set + suggested query refinements + original query

Application profile: specifying facets <xrbr:query xmlns:xrbr="http://www.siderean.com/2001/10/xrbr/" item-type="http://www.dcmi.org/dcconf/objects#Event" sort-dimension="title" > <xrbr:hint flattenresults="yes" startpagecolumns="4"/> <xrbr:dimensions> <xrbr:dimension name="title" predicate="http://purl.org/dc/elements/1.1/title"> <xrbr:hint textsearch="yes" label="Title" function="itemlabel"/> <xrbr:return /> </xrbr:dimension> <xrbr:dimension name="description" predicate="http://purl.org/dc/elements/1.1/description"> <xrbr:hint textsearch="yes" label="Description" function="itemdescription"/> <xrbr:return /> </xrbr:dimension> … </xrbr:dimensions> </xrbr:query>

Application profile: specifying hierarchical facets … <xrbr:dimension name="BT1" predicate="http://purl.org/dc/elements/1.1/subject" display-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif#value" root-resource="http://www.dcmi.org/dcconf/2003#Organizations_and_Domains" ancestor-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif-simple#BT" > <xrbr:hint label="Organizations and Domains" facet="yes“ scopenote="Sectors, languages, special literatures or communities that use metadata" /> <xrbr:suggestions count="7" /> </xrbr:dimension> …

Application profile: flattening graphs … <xrbr:structure name="creator" predicate="http://purl.org/dc/elements/1.1/creator"> <xrbr:dimension name="creatorname" predicate="http://xmlns.com/foaf/0.1/#name"> <xrbr:hint label="Author" textsearch="yes"/> <xrbr:suggestions count="7" /> <xrbr:return /> </xrbr:dimension> <xrbr:dimension name="creatororg“ predicate="http://xmlns.com/foaf/0.1/#memberOf" display-predicate="http://xmlns.com/foaf/0.1/#name"> <xrbr:hint label="Author Affiliation" /> <xrbr:suggestions count="7" /> <xrbr:return /> </xrbr:dimension> </xrbr:structure> …

Automatically generated interface

Case study: Environmental Health News Aggregating news stories from the Web Semi-automated metadata creation by a team of subject matter experts and editors Semantic search to design custom feeds

Case study: Gateway to Educational Materials Aggregating learning objects from members of the GEM Consortium Embedding semantic search into a portal

Case study: NASA JPL Project information aggregated from content and data repositories Using and extending taxonomies Exploiting document type/genre

Related work in RDF OCLC Metadata Switch MIT Simile Longwell Haystack Aduna Sesame Ontoprise OntoSeek Nature Publishing Group Urchin

Issues Scale: must be commensurate with expectations and requirements from traditional web and enterprise search Number of objects, feeds: 10 6 to 10 9 Ingest rates: ~ 10 3 – 10 4 triples/sec, how many per resource? Tagging: where and when? Latency: < 0.5 sec user time regardless of application Retrieval algorithms: many alternatives still being explored Federated services vs. centralized servers Relationship to relevance ranking Support for aggregate and text search operators in RDF query Usability: lots of work to be done to validate benefits Navigation Precision and recall Visualization Security, trust and provenance: just beginning to understand

Lessons Balanced incremental approach Leverage metadata and indices at hand Exploit statistics where desirable But layer a framework on top to structure the statistics Significant mileage from very simple frameworks

Lessons: ontologies Don’t do: assume you have to build elaborate OWL ontologies Don’t have to boil the ocean to get the benefits OWL DL, are OWL Full are overkill for this class of application Do: Tiny Ontologies Stitched Together RDF Schema with a smattering of RDF/OWL properties (e.g., owl:inverse) Start with DC + SKOS + FOAF

Lessons: controlled vocabularies Don’t do: huge monolithic taxonomies Unless they are ready at hand and can be reused largely without modification Do: bite-sized controlled vocabularies that exploit faceted approaches 4 facets x 10 terms per facet versus 10 4 terms in a single taxonomy Start with flat term lists Add BT/NT/RT relationships over time

Lessons: instances Manual creation Don’t do: exhaustive author creation of metadata Do: community annotation and tagging (Semi-)automated creation Don’t do: assume elaborate information extraction based on NLP, subject tagging and categorization Do: quick and dirty NEE or better yet, stick to readily available asset and relational metadata (date, creator, document type/genre) Much of the benefit at a fraction of the effort

Application profiles Metadata is increasingly pervasive The way to leverage existing information infrastructure Exploit “on-demand” information integration feature of RDF DB + XML -> XLST - > RDF(S)

The big question: statistics vs. knowledge Statistics can’t deliver everything Alan Kay’s puppy analogy Vitanyi work on “Google learning” On the other hand, knowledge is dearly won CYC Need a balance that enables adoption without losing the benefits Lessons from Statistics vs. knowledge in NLP Expert systems

Future directions User tagging + RDF: the killer SW application? The rehabilitation of metadata in the social software community The re-emergence of RSS/RDF “ Folksonomy”-driven collaborative search Del.icio.us, Flickr, CiteULike Growth of the SW compared to historical growth of the Web: it’s 1994 all over again

Summary Semantic search has a role in today’s enterprises RDF provides a framework that can ease adoption and encourage innovation in semantic search The future for enterprise and consumer use looks bright

Semantic Search using RDF Metadata (SemTech 2005)

Recommended

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Semantic Search using RDF Metadata (SemTech 2005) (20)

More from Bradley Allen (6)

Recently uploaded (20)

Semantic Search using RDF Metadata (SemTech 2005)