Cross-domain Document Retrieval: Matching between Conversational and Formal Writings
0 likes208 views
This paper challenges a cross-genre document retrieval task, where the queries are in formal writing and the target documents are in conversational writing. In this task, a query, is a sentence extracted from either a summary or a plot of an episode in a TV show, and the target document consists of transcripts from the corresponding episode. To establish a strong baseline, we employ the current state-of-the-art search engine to perform document retrieval on the dataset collected for this work. We then introduce a structure reranking approach to improve the initial ranking by utilizing syntactic and semantic structures generated by NLP tools. Our evaluation shows an improvement of more than 4% when the structure reranking is applied, which is very promising.
Ad
More Related Content
What's hot (20)
Similar to Cross-domain Document Retrieval: Matching between Conversational and Formal Writings (20)
Ad
More from Jinho Choi (20)
Ad
Recently uploaded (20)
Cross-domain Document Retrieval: Matching between Conversational and Formal Writings
- 1. Cross-genre Document Retrieval: Matching between Conversational and Formal Writings Tomasz Jurczyk and Jinho D. Choi Department of Mathematics and Computer Science, Emory University STRUCTURE MATCHING AND RERANKING • Document retrieval has been a central task in natural language processing and information retrieval. • Most of previous work has focused on document retrieval in the same text domains. • Dedicated approach is needed to provide robust and meaningful models in cross-genre text domains. Examples of source text (left) and target text (right) domains • Transcripts from 8 seasons of the show Friends are provided by The Character Mining project. • Multiple fan sites have been crawled to collect summaries and plots. • Generally, summaries give broad descriptions of the episodes, whereas plots describe facts within individual scenes. • All the summaries and plots are sentence segmented and parsed using NLP4J. • The dataset consists of queries (segmented sentences) associated with their relevant episodes. Dialogue, summary and plot data • Structure reranking approach is presented that addresses the challenge that comes from the cross-genre aspect of our data. • Relations used in structure matching are extracted from both types of texts. • Currently, heuristic methods are used to extract meaningful contextual words from semantic roles labels and dependency structures. • Relations are attached to the speaker names to compensate the lack of entity information. • First, the hypothesis that our structure ranking can retrieve documents correctly for queries that Elasticsearch could not is tested. The overview of our structure reranking • Extracted structures from scripts and queries are then matched, where the algorithms tries to maximize the similarity score between relations. • Structure matching scores are used as features in the neural network model. • If the Binary (BIN) classifier disqualifies the top-ranked document from Elasticsearch, the top-k documents are reranked by the weighted sums of different scores. • A new method for a cross-genre document retrieval task between conversational and format writing is presented. • Our structure reranking approach gives an improvement of more than 4% of P@1. • In the future, we want to explore how the dialogue structure (speaker/utterance) could be better used to improve the structure extraction. • Also, we want to combine other NLP tasks such as coreference resolution and disfluency detection to provide further improvements. CONCLUSIONS • Document retrieval for cross-genre aspect is first analyzed using a baseline that consists of Elasticsearch search engine. • Each episode is indexed as a document using the Okapi25 BM25 setting. • The top-k most relevant documents are retrieved for each query. OVERVIEW EXPERIMENTS Elasticsearch results on (summary + plot) Two sets of relations, from dialogue and plot. The upper presents a slice of script where the answer is located, the lower presents the query to be matched. Results on queries failed by Elasticsearch for k=10 • Our structure matching can correctly associate a significant portion of queries that Elasticsearch could not. • The highest score achieved by our structure matching is 39.53%, which is significantly lower than Elasticsearch. • The neural network model trained on the combination of Elasticsearch and our structure matching gives an over 4% improvement on the evaluation set. • The data is split into training, development, and evaluation sets, where queries from each episode are randomly assigned. Data split (# of queries) ACKNOWLEDGMENT We gratefully acknowledge the Department of Mathematics and Computer Science at Emory University for supporting this work. Any content presented here is solely the responsibility of the authors and does not necessarily represent the official view of the organization.