![Knowledge Graph Construction
• We constructed KGs based on the contents of 8 of Sherlock
Holmes’s short mystery stories.
• The Speckled Band, The Dancing Men, A Case Of Identity, The Devil’s Foot,
The Crooked Man, The Abbey Grange, The Resident Patient, Silver Blaze
• Procedure[kawamura et al. JIST2019]
1. Extract sentences necessary for deduction from mystery stories (in
Japanese) whose copyrights have expired.
2. Rewriting the original text into sentences with clear a subject and object
(i.e., short sentences).
3. Assign semantic roles (e.g., 5W1H) to phrases using natural language
processing tools (Japanese semantic role labeling).
4. Control vocabulary (e.g., predicates, names of characters, and places)
5. Add relationships between scenes (e.g., temporal relationships).
6. Translate the source text into English and convert the entire scenes into a
knowledge graph.
7](https://image.slidesharecdn.com/ijckg2022-221027113956-fd5fa76e/85/Contextualized-Scene-Knowledge-Graphs-for-XAI-Benchmarking-6-320.jpg)
![Guidelines for KG refinement
7. Giving a type at nesting
• In order to express appropriately nested
structures caused by hearsay, each
utterance is decomposed as a scene and
given an appropriate type and source of
information.
• For example, the scene (ID-a)
“Holmes said, “Mr. A said that Mr. B said
something ([ID-y])”([ID-x]).”
is decomposed as follows:
Unification of
triple structure
18
# Holmes said kd:ID-x
kd:ID-a rdf:type kgc:Situation ;
kgc:subject kd:Holmes ;
kgc:hasPredicate kdp:say ;
kgc:what kd:ID-x .
# Mr. A said kd:ID-y
kd:ID-x rdf:type kgc:Statement ;
kgc:InfoSource kd:Holmes ;
kgc:subject kd:A ;
kgc:hasPredicate kdp:say ;
kgc:what kd:ID-y .
# Mr. B said something
kd:ID-y rdf:type kgc:Statement ;
kgc:InfoSource kd:A ;
kgc:subject kd:B ;
kgc:hasPredicate kdp:say ;
kgc:what sometihng .](https://image.slidesharecdn.com/ijckg2022-221027113956-fd5fa76e/85/Contextualized-Scene-Knowledge-Graphs-for-XAI-Benchmarking-16-320.jpg)
![Guidelines for KG refinement
8. Mapping verbs to hasPredicate values
• Verb forms are unified in the active voice
• “Mr. A was shot by Mr. B” is rephrased as “Mr. B shot Mr. A.”
• Verb tenses are unified in the present tense
• The verb (the value of hasPredicate) in the scene “Mr. B shot Mr. A” is “shoot” in the
present tense, not the past tense.
• Emotional expressions are unified into states, not verbs
• The scene “John Straker was excited,” “excited” is not treated as a verb, but is taken as a
state and the value of hasProperty.
• Scenes involving verbs followed by infinitives are broken down
• The scene “John Straker tried to go check the stable”, instead of creating a verb like
tryTo, we break the scene down as follows: “John Straker try [ID-x].” and “John Straker go
to check the table.” ([ID-x])
• Auxiliary verbs and verbs concatenated into one verb
• In the scene “Percy Trevelyan had to prepare the money,” mustPrepare is created as a
verb.
• In addition, that verb is defined that it consists of “must” and “prepare.”
Unification of
the vocabulary
20](https://image.slidesharecdn.com/ijckg2022-221027113956-fd5fa76e/85/Contextualized-Scene-Knowledge-Graphs-for-XAI-Benchmarking-18-320.jpg)
Ad
More Related Content
Similar to Contextualized Scene Knowledge Graphs for XAI Benchmarking (20)
More from KnowledgeGraph (20)
![[part 1]ナレッジグラフ推論チャレンジ・Tech Live！](https://cdn.slidesharecdn.com/ss_thumbnails/kgrctechliive2020-1021-201023101135-thumbnail.jpg?width=560&fit=bounds)
![[part 2]ナレッジグラフ推論チャレンジ・Tech Live！](https://cdn.slidesharecdn.com/ss_thumbnails/kgrctechlive-201023091455-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Ad
Contextualized Scene Knowledge Graphs for XAI Benchmarking
- 1. Contextualized Scene Knowledge Graphs for XAI Benchmarking Takahiro Kawamura1,2, Shusaku Egami2, Kyoumoto Matsushita3, Takanori Ugai3,2, Ken Fukuda2, Kouji Kozaki4,2 1National Agriculture and Food Research Organization (NARO), Japan 2National Institute of Advanced Industrial Science and Technology (AIST), Japan 3Fujitsu Ltd., Japan 4Osaka Electro-Communication University, Japan 1 IJCKG2022
- 2. Outline 1. Summary 2. Knowledge Graph Construction and KGRC 3. Guidelines for Knowledge Graph Refinement 4. Application of Guidelines 5. Conclusion 2
- 3. • Construct scene KG dataset and hold AI competitions to gather methods related to inference and estimation from wide range of engineers and researchers. • Guideline for KG refinement based on lessons learned from held competition (2018 – present) Summary 3 • AI technologies that have explainability (i.e., XAI) or interpretability are attracting attention. • AI technologies that combine inductive machine learning and deductive knowledge utilization are expected to become necessary in the future. Background • A suitable dataset for the evaluation of XAI tasks should include not only relatively simple relationships, but also more complex relationships that reflect the real world. • e.g., spatial, temporal, causal, and contextual relationships Problem • Design appropriate indicators and then evaluate, classify, and systematize AI technologies with explanatory and interpretability, especially those that combine inductive machine learning and deductive knowledge processing. Objective Proposal
- 4. Knowledge Graph Construction and KGRC 4
- 5. Knowledge Graph Reasoning Challenge Investigation strategy Criminal motive …. A Contest to develop AI systems which have abilities for “Reasoning” and “Explanation” such like Sherlock Holmes. Sherlock Holmes mystery story Knowledge Graph (LOD) AI system that estimate criminals with reasonable explanations using the KG and other knowledge The motive is … Trick is … The criminal is XX Because … 2018 – 2021 by the Special Interest Group on Semantic Web and Ontology (SWO) of the Japanese Society for AI (JSAI) 5
- 6. Knowledge Graph Construction • We constructed KGs based on the contents of 8 of Sherlock Holmes’s short mystery stories. • The Speckled Band, The Dancing Men, A Case Of Identity, The Devil’s Foot, The Crooked Man, The Abbey Grange, The Resident Patient, Silver Blaze • Procedure[kawamura et al. JIST2019] 1. Extract sentences necessary for deduction from mystery stories (in Japanese) whose copyrights have expired. 2. Rewriting the original text into sentences with clear a subject and object (i.e., short sentences). 3. Assign semantic roles (e.g., 5W1H) to phrases using natural language processing tools (Japanese semantic role labeling). 4. Control vocabulary (e.g., predicates, names of characters, and places) 5. Add relationships between scenes (e.g., temporal relationships). 6. Translate the source text into English and convert the entire scenes into a knowledge graph. 7
- 7. Knowledge Graph Schema • Basic policy • Focus on scenes in a novel and the relationship of those scenes, including the characters, objects, places, etc., with related scenes • Only scenes that are judged to be necessary for the deduction are converted to KGs • A scene ID (IRI) has subjects, verbs, objects, etc. • Edges mainly represent five Ws (When, Where, Who, What, and Why). Scene1 Scene2 Scene3 Scene4 Scene5 Resource Literal type subject source source subject subject hasPredicate source hasPredicate subject hasPredicate then therefore because then 8
- 8. Original Sentence (EN|JA) Absolute Time Property values are defined as resources to be referred in the other scene Predicate Subject Relationship to other Scene ID Scene Type - Situation︓ Fact - Statement︓Remark by A - Talk︓ Remark by A to B - Thought︓ Idea of A Schema (Scene): Example Unique ID (IRI) of Each scene 10
- 9. Submission Categories for KGRC 1. Main Category: development of a system that can accomplish the task of one or more subject stories 2. Tool Category (From 2019): development of a tool with which a task can be partially solved 3. Idea Category: development of idea to realize category 1. or 2. (no actual implementation is necessary) Numbers of Submissions 11 Main 5 Idea 3 Main 4 Tool 3 Idea 2 Main 2 Tool 2 Idea 3 Main 13 Idea 8 Tool 8 2018 2019 2020 Total 29 Main 2 Tool 3 2021* *Student only
- 10. （１）Approach for reasoning/explanations 12 2018 2019 2020 2021 (for student) Category (1)Aproaches (2)External Knowledge Category (1)Aproaches (2)External Knowledge Category (1)Aproaches (2)External Knowledge Category (1)Aproaches (2)External Knowledge Main.1 knowledge processing original rules Main.1 knowledge processing ＋Machine Learning text of novels original rules other external information Main.1 Machine Learning none Main.1 Machine Learning ConceptNet, original ontology Main.2 knowledge processing original ontologies reasoning rules Main.2 Machine Learning none Main.2 Machine Learning ConceptNet Main.2 knowledge processing Original ontology, original rules Main.3 Machine Learning text of other Holmes novels Main.3 Machine Learning none Tool.１ knowledge processing none Tool.1 knowledge processing - Main.4 knowledge processing original rules for Presumption of culprit ontology of motivation Main.4 knowledge processing (＋Machine Learning) original ontologies WordNet Wikipedia Tool.２ knowledge processing Wikidata Tool.2 Visualization - Main.5 knowledge processing original rules Tool.１ Machine Learning none Idea.１ Machine Learning WordNet Tool.3 Knowledge processong Wikidata, JIWC Idea.１ Multi agent none Tool.２ NLP none Idea.２ knowledge processing Wikidata，ICD-10 Idea.２ knowledge processing original knowledge Tool.３ knowledge processing NRC Emotion / Affect Intensity Lexicon Idea.３ Machine Learning none Idea.３ knowledge processing original ontologies Idea.１ Machine Learning Wikipedia Idea.２ Machine Learning none
- 11. International Knowledge Graph Reasoning Challenge (IKGRC) • The 1st International Knowledge Graph Reasoning Challenge (IKGRC2023) • co-located with 17th IEEE International Conference on Semantic Computing (ICSC) • The Hills Hotel, Laguna Hills, California, USA (format: Hybrid), February 1-3, 2023 • Important date • Abstract submission (2 pages, IEEE format) : November 30, 2022 • Accepted papers will be published in the Proceedings of the ICSC. • Acceptance notification: December 7, 2022 • Submission of application materials: January 15, 2023 • Workshop day: February 2-3, 2023 13 https://ikgrc.org/2023/
- 12. Guidelines for Knowledge Graph Refinement 14
- 13. Guidelines for KG refinement • We present a guideline consisting of 10 items/steps for refining the KGs based on lessons learned from our KGs of 8 mystery stories. 1 Short sentences in English are converted to a syntax that is easy to change to an RDF Unification of triple structure 2 Adding implicit scenes Addition of implicit information that is not explicitly described 3 Add time information 4 Screening of sentences to be treated as scenes Unification of triple structure 5 Unification of triplication from typical sentence patterns 6 Division when there is more than one subject or object 7 Giving a type at nesting 8 Mapping verbs to hasPredicate values Unification of the vocabulary 9 Unification of words such as object and complement 10 Uniform treatment of modifiers 15
- 14. Guidelines for KG refinement 1. Short sentences in English are converted to a syntax that is easy to change to an RDF • Clarify the division between subject and predicate. • For example, in the sentence “There is no place for Percy Trevelyan,” it is difficult to understand whether the subject is “Percy Trevelyan” or “Percy Trevelyanʼs place,” and also whether the predicate is “does not have place” or “does not exist”. • In such sentences, the subject is decided to be “Percy Trevelyan” because the target to which information is to be added is “Percy Trevelyan.” • Complements omitted objects, complements, places, and so on. • ex 1) Although there is no location information in “Helen lives with Roylott,” it is clear from the context that she lives in “Roylott’s house,” so the place should be added. • ex 2) In the phrase “Roylott is a father-in-law,” it is not clear from whose perspective he is a father-in-law, so the additional information should be provided. Unification of triple structure 16
- 15. Guidelines for KG refinement 4. Screening of sentences to be treated as scenes • For example, the sentence “The money is 1000 pounds a year” cannot be understood as a stand-alone scene. Therefore, a triple is added to supplement the scene such as “Helen and Julia receive their inheritance.” Unification of triple structure 17 5. Unification of triplication from typical sentence patterns • For example, “there is” and “exists” are unified into “exists” to standardize the symbols used in the inference process. • Also, information (adjectives) that describe properties are unified with the value hasProperty; 6. Division when there is more than one subject or object • For example, the scene “Holmes and Watson got out of the carriage” splits the subject (value of the subject) into two parts, “Holmes” and “Watson”. • Also, the scene “Holmes placed a box of matches and a burnt candle near a long, thin walking stick” splits the object (value of kgc:what) into “a box of matches” and “a candle”.
- 16. Guidelines for KG refinement 7. Giving a type at nesting • In order to express appropriately nested structures caused by hearsay, each utterance is decomposed as a scene and given an appropriate type and source of information. • For example, the scene (ID-a) “Holmes said, “Mr. A said that Mr. B said something ([ID-y])”([ID-x]).” is decomposed as follows: Unification of triple structure 18 # Holmes said kd:ID-x kd:ID-a rdf:type kgc:Situation ; kgc:subject kd:Holmes ; kgc:hasPredicate kdp:say ; kgc:what kd:ID-x . # Mr. A said kd:ID-y kd:ID-x rdf:type kgc:Statement ; kgc:InfoSource kd:Holmes ; kgc:subject kd:A ; kgc:hasPredicate kdp:say ; kgc:what kd:ID-y . # Mr. B said something kd:ID-y rdf:type kgc:Statement ; kgc:InfoSource kd:A ; kgc:subject kd:B ; kgc:hasPredicate kdp:say ; kgc:what sometihng .
- 17. Guidelines for KG refinement 2. Adding implicit scenes • For example, “the day Helen’s mother died” can be expressed as a single literal as “death_day_of_helen’s_mother.” However, it cannot be used for inference because it does not logically express the information that this is the day that Helen’s mother died. • Therefore, we introduce a new scene “Helen’s mother died.” 19 3. Add time information • If there is no description of time in the text, absolute time is given to each scene to the extent that it does not affect the narrative. • Qualitative temporal relationships, such as “then,” “before,” “after,” are added as connections between scenes to clarify the time-series information. Addition of implicit information that is not explicitly described
- 18. Guidelines for KG refinement 8. Mapping verbs to hasPredicate values • Verb forms are unified in the active voice • “Mr. A was shot by Mr. B” is rephrased as “Mr. B shot Mr. A.” • Verb tenses are unified in the present tense • The verb (the value of hasPredicate) in the scene “Mr. B shot Mr. A” is “shoot” in the present tense, not the past tense. • Emotional expressions are unified into states, not verbs • The scene “John Straker was excited,” “excited” is not treated as a verb, but is taken as a state and the value of hasProperty. • Scenes involving verbs followed by infinitives are broken down • The scene “John Straker tried to go check the stable”, instead of creating a verb like tryTo, we break the scene down as follows: “John Straker try [ID-x].” and “John Straker go to check the table.” ([ID-x]) • Auxiliary verbs and verbs concatenated into one verb • In the scene “Percy Trevelyan had to prepare the money,” mustPrepare is created as a verb. • In addition, that verb is defined that it consists of “must” and “prepare.” Unification of the vocabulary 20
- 19. Guidelines for KG refinement 9. Unification of words such as object and complement • Assign unique names and IRIs to people and things • List the people and things that appear first, and assign unique names and IRIs to them. • Replace collation with named entities and scene IDs • To distinguish whether it is a concrete person or thing, replace directives, pronouns, and so forth, with proper nouns • Unified notation for labels and IRIs • Establish conventions for the use of camel notation, snake notation, space delimiters, and so on, to ensure consistency within a KG. Unification of the vocabulary 21 10. Uniform treatment of modifiers • We use are source as it is if it has a qualifier, such as “red carpet,” because a modifier may be used as a keyword in a story. • The type is then defined as “carpet” and the property (value of hasProperty) is defined as “red”.
- 21. Verification of the guideline application Conduct a trial application of the guideline with a third party • Worker • Software engineer who has knowledge of RDF and an outline of the KGRC, but was not involved in the creation of the guidelines • Task • Apply the guideline to 8 KGs by one worker • The data are RDF triples converted to spreadsheet format 23 Approximate working time: 30 man-days Work sheet Check sheet
- 22. Application policy of the guideline 1 Short sentences in English are converted to a syntax that is easy to change to an RDF (A) Extracts the points that need to be modified in advance because these tasks are high-cost 2 Adding implicit scenes 3 Add time information (B) Can be handled mechanically to some extent 4 Screening of sentences to be treated as scenes (A) 5 Unification of triplication from typical sentence patterns (B) 6 Division when there is more than one subject or object 7 Giving a type at nesting (A) 8 Mapping verbs to hasPredicate values (B) 9 Unification of words such as object and complement 10 Uniform treatment of modifiers Pending due to insufficient consideration. 24
- 23. Results • Can third party apply the guideline? • The results of the third party's work were generally appropriate, although some corrections were desirable. • For the guidelines that were limited to the identification of areas that should be modified due to the high cost, the guidelines contain items that should be considered for individual modification policies • Therefore, it is preferable to consider the detailing of the guidelines. • Applicability of the guideline to KGs in general • Next page 25
- 24. Applicability of the guideline to KGs in general 1 Short sentences in English are converted to a syntax that is easy to change to an RDF Refinement methods common to all KGs 2 Adding implicit scenes Refinement methods specific to Scene KGs 3 Add time information 4 Screening of sentences to be treated as scenes 5 Unification of triplication from typical sentence patterns 6 Division when there is more than one subject or object Refinement methods common to all KGs 7 Giving a type at nesting Refinement methods specific to Scene KGs 8 Mapping verbs to hasPredicate values Refinement methods common to all KGs 9 Unification of words such as object and complement 10 Uniform treatment of modifiers 26
- 25. International Knowledge Graph Reasoning Challenge (IKGRC) • The 1st International Knowledge Graph Reasoning Challenge (IKGRC2023) • co-located with 17th IEEE International Conference on Semantic Computing (ICSC) • The Hills Hotel, Laguna Hills, California, USA (format: Hybrid), February 1-3, 2023 • Important date • Abstract submission (2 pages, IEEE format) : November 30, 2022 • Accepted papers will be published in the Proceedings of the ICSC. • Acceptance notification: December 7, 2022 • Submission of application materials: January 15, 2023 • Workshop day: February 2-3, 2023 27 https://ikgrc.org/2023/
- 26. Conclusion • Guideline for KG refinement • We developed a guideline based on lessons learned from Knowledge Graph Reasoning Challenge in Japan • We applied the guideline to the eight KGs, and published them on our GitHub repository: https://github.com/KnowledgeGraphJapan/KGRC-RDF • We provide the refined KGs for International Knowledge Graph Reasoning Challenge: https://ikgrc.org/2023/ • Future works • Consideration of a policy for vocabulary unification • Consideration of new representation of scenes 28
- 27. Thank you for your attention! https://ikgrc.org/2023/ https://github.com/KnowledgeGraphJapan/KGRC-RDF This work was supported by JSPS KAKENHI Grant Number 19H04168. http://knowledge-graph.jp/sparql-ikgrc.html