Effect of Heuristics on Serendipity in Path-Based Storytelling with Linked Data

Editor's Notes

• #3: Algorithmic storytelling can be seen as a particular kind of querying data. Given a set of keywords or entities, which are typically, but not necessarily dissimilar, it aims at generating a story by explicitly relating the query context with a path that includes semantically related resources. Storytelling is utilized for example in entertaining applications and visualizations in order to enrich related Linked Data resources with data from multimedia archives and social media as well as in scientific research fields such as bio-informatics where biologists try to relate sets of genes arising from different experiments by investigating the implicated pathways or discovering stories through linked books.
• #5: Algorithmic storytelling can be seen as a particular kind of querying data. Given a set of keywords or entities, which are typically, but not necessarily dissimilar, it aims at generating a story by explicitly relating the query context with a path that includes semantically related resources. Storytelling is utilized for example in entertaining applications and visualizations in order to enrich related Linked Data resources with data from multimedia archives and social media as well as in scientific research fields such as bio-informatics where biologists try to relate sets of genes arising from different experiments by investigating the implicated pathways or discovering stories through linked books.
• #6: Research questions: Relate to the title of the paper, we want to get an indication of the effect of using different heuristics and find suitable measure to gain insight in them: 3 components: [pathfinding] [serendipity] and [user judgments]
• #7: The aspects that make a story a good story are captured in the term serendipity. The term depicts a mixture between casual, lucky, helpful and unforeseen facts, also in an information context. In fact, users want to be surprised and they want to discover, confirm, and extend knowledge - but not feel unsure while doing so. This means that users can always relate presented story facts to their background knowledge.
• #9: The most frequently encountered algorithm to determine a path between multiple resources is the A* algorithm. This algorithm, which is based on a graph representation of the underlying data (i.e., resources and links between them define nodes and edges, respectively) determines an optimal solution in form of a lowest-cost traversable path between two resources (determined using H and W). The optimality of a path, which is guaranteed by the A* algorithm, does not necessarily comply with the users’ expectations
• #10: Our algorithm reduces the arbitrariness of a path between these resources by increasing the relevance of the links between the nodes using a domain-delineation step. The path is refined by iteratively applying the A* algorithm and with each iteration attempting to improve the overall semantic relatedness between the resources until a fixed number of iterations or a certain similarity threshold is reached. H and W are not fixed as in the original core algorithm.
• #11: Jaccard statistical approach taking into account the relative number of common predicates of two nodes. The higher the number of common predicates, the more likely similar properties of the nodes and thus the semantically closer in terms of distance the corresponding nodes. NDD See normalized web distance but with f being a different function: f (n)∈N denotes thenumber of DBpedia nodes linking to node n∈G , f (n,m)∈N denotes the number of DBpedia nodes linking to both nodes n and m. Confidence asymmetrical statistical measure that can be thought of as the probability that node a occurs provided that node b has already occurred.
• #12: Jaccard (see prev) JCW looking at the classes of each of the nodes and determining the most common denominator of those classes in the ontology. Once this type is determined, the number of subjects that exist with this type is divided by the total number of subjects. The higher this number, the more generic the class, thus the more different two nodes. CND can be used to compute a weight that encourages rarity of items in a path. It ranks more rare resources higher, thereby guaranteeing that paths between resources prefer specific relations. The main idea is to avoid that paths go via generic nodes. It makes use of the node degree, the number of in and outgoing links.
• #15: f(x) and f(y) are the number of hits for search terms x and y, respectively; andf(x, y) is the number of web pages on which both x and y occur.
• #17: Semantic relatedness is improved (distance is decreased) in all cases except when entities were already closely related. NGD: normalized web search distance implemented on top of Bing Search API.
• #18: A set of paths is not a presentable story yet. We note that even if a path comprise just the start and destination (indicating they are linked via common hops or directly to each other), the story will contain interesting facts. This is because each step in the path is separated with at least one hop from the next node. For example, to present a story about Carl Linnaeus and Charles Darwin, the story could start from a path that goes via J.W. von Goethe. The resulting statements serve as basic facts, which are relation-object statements, that make up the story. It is up to the application or visualization engine to present it to end-users and enrich it with descriptions, media or further facts.
• #19: We applied SemRank to evaluate the paths, in particular to capture the serendipity of each path. The serendipity is measured by using a factor µ to indicate the so called ‘refraction’ how different each new step in a path is compared to the previous averaged over the entire path. Furthermore the information gain is modulated using the same factor µ . The information gain is computed from the weakest point along the path and an average of the rest. * The information gain indicates how much new information is added over all the possibilities that were in fact available. It is averaged over all the steps (facts). If a fact is more rare it will contribute relatively more than others at a certain point. This involves looking at the frequency of the predicates and the resources in the path and how likely it was that a certain resource was included a certain point in a path. In information theory, the amount of information contained in an event is measured by the negative logarithm of the probability of occurrence of the event. More details -> refer to SemRank paper: Kemafor Anyanwu, Angela Maduko, and Amit Sheth. 2005. SemRank: ranking complex relationship search results on the semantic web. In Proceedings of the 14th international conference on World Wide Web (WWW '05). ACM, New York, NY, USA, 117-127. DOI=http://dx.doi.org/10.1145/1060745.1060766 * Average re- fraction R(p): path -> how different is the item from the previous, i.e. thus how much does it contribute to the path so far. There are three special cases [4]: conventional with µ = 0 leading to 1 I(p) SemRank(0, p) = , serendipity plays no role and so no emphasis is put one newly gained or unexpected information; mixed with µ = 0.5 leading toSemRank(0.5, p)= R(p) I(p) 1 2I(p) + ]×−[1+ [ ], a balance between unexpected and newly gained information; 2 2 and discovery with µ = 1 leading to SemRank(1, p) = I(p)×−[1+R(p)], emphasizing unexpected and newly gained information.
• #21: On the one hand the conventional and mixed mode for SemRank put less emphasis on novelty and focuses mainly on semantic association and information content. The jaccard distance combination used as weight and heuristic is not entirely surprisingly the best choice for this scenario. On the other hand the results of the original algorithm making use of the common node degree as weight together with the jaccard distance is confirmed by the results of the improved algorithm with the common node degree however with a slightly lower rank in the new algorithm. Using the JCW however leads to even higher ranks. In terms of discovery, the original algorithm outperforms the JaccardJaccard combination. The CNDJaccard improved algorithm is able to slightly outperform all the other combinations.
• #22: The effect of the heuristics according to user judgments compared to the overall median. The JCWJaccard confirms already good results with SemRank. The CNDJaccard scores relatively well too. The scores for relevancy, consistency and discovery as unexpected - but relevant - facts are highly dependent on the user who judges. Some users might be interested in the more trivial path as well in some cases. Nevertheless, we used the overall judgment as a baseline to compare the judgments with the same combinations of heuristics and weights as before.
• #23: The suggested stories that center around a certain via-fact are not always considered relevant by some users even though the algorithms might consider them so.
• #24: Algorithmic storytelling can be seen as a particular kind of querying data. Given a set of keywords or entities, which are typically, but not necessarily dissimilar, it aims at generating a story by explicitly relating the query context with a path that includes semantically related resources. Storytelling is utilized for example in entertaining applications and visualizations in order to enrich related Linked Data resources with data from multimedia archives and social media as well as in scientific research fields such as bio-informatics where biologists try to relate sets of genes arising from different experiments by investigating the implicated pathways or discovering stories through linked books.
• #25: We proposed an optimized pathfinding algorithm for storytelling that reduces the number of arbitrary resources revealed in paths contained in the story. Preliminary evaluation results using the DBpedia dataset indicate that our proposal succeeds in telling a story featuring better link estimation, especially in cases where the previous algorithm did not make seemingly optimal choices of links. By defining stories as chains of links in Linked Data, we optimized the storytelling algorithm and tested with several heuristics and weights. The most consistent output was generated with the Jaccard distance used both as weight and heuristic; or as heuristic in combination with the Jiang-Conrath distance as weight. The most arbitrary facts occur in a story when using the combined node degree as weight with the Jaccard distance as heuristic, both in the optimized and the original algorithm. User judgments confirm the findings for the Jiang-Conrath weight and the original algorithm and for the Jaccard distance used as weight and heuristic in terms of discovery. There is no clear positive effect however according the users in terms of consistency and relevancy there.
• #26: Future work will focus on validating the correlation between the effect of the link estimation on the arbitrariness as perceived by users and computational semantic relatedness measures such as SemRank. Additionally, we will measure the scalability of our approach by implementing the algorithms (i) solely on the client, (ii) completely on the sever, and (iii) in a distributed client/server architecture.