Small Is Beautiful: Summarizing Scientific Workflows Using Semantic Annotations. IEEE BigData Conference 2013

Editor's Notes

• #2: Producing attractive publications
• #4: In this age of data-intensive science we’re witnessing the unprecedented generation and sharing of large scientific datasets, where the pace of data generation has far surpassed the pace of conducting analysis over the data. Scientific Workflows [6] are a recent but very popular method for task automation and resource integration. Using workflows, scientists are able to systematically weave datasets and analytical tools into pipelines, represented as networks of data processing operations connected with dataflow links. (Figure 1 illustrates a workflow from genomics, which “from a given set of gene ids, retrieves corresponding enzyme ids and finds the biological pathways involving them, then for each pathway retrieves its diagram with a designated coloring scheme”). As well as being automation pipelines, workflows are of paramount importance for the provenance of data generated from their execution [6]. Provenance refers to data’s deriva- tion history starting from the original sources, namely its lineage. Producing attractive publications
• #6: The systematic workflow-based organization of analysis steps and the faithful collection of provenance is important for activities such as 1) Reporting: Scientists are often expected to report on the methods they followed and their resultant datasets and the base/origin data they used in academic papers describing findings. 2) Handling Change: By having an explicit specification of the analysis process and data dependencies, scientists could inquire on the impact that a change of a particular input or tool will have on results and which parts of the experiment would need to be re-run. These potential ways of exploitation, however, are hampered by the complexity of Scientific Workflows. Producing attractive publications
• #7: By explicitly outlining the analysis process, work- flow descriptions make-up a prospective part of provenance. The instrumented execution of the workflow generates a veracious and elaborate data lineage trail outlining the derivation relations among resultant, intermediary and initial data artifacts, which makes up the retrospective part of provenance. Producing attractive publications
• #8: Workflows are typically complex artifacts containing sev- eral processing steps and dataflow links. Complexity is characterized by the environment in which workflows op- erate. Obfuscation degrades reporta- bility of the scientific intent embedded in the workflow. Complex workflows combined with the faithful collection of execution provenance for each step results in even more complex provenance traces containing long data trails, that overwhelm users who explore or query them [3], [1]. Producing attractive publications
• #9: Workflows that collate datasets and tools from 3rd- party autonomous providers contain several adapter steps for handling format and protocol heterogeneities [9][8]. Meanwhile workflows that perform analytics over local(ized) data typically contain several steps dedicated to organization and pre-processing of data. We have shown in a previous study [8] that these mundane operations account for up to 60% of the steps that compose workflows. These steps dealing with the technical detail of resource access or data organization obfuscate the scientifically significant/critical operations of the workflow. Producing attractive publications
• #10: Lack of effective simplification solutions have led sci- entists to devise their own ways to cope with complexity. These manual abstraction solutions have the disadvantage of being hard-coded into the workflow design and may not simultaneously cater for different aspects of complexity, as we shall analyze in this paper. We observe that there are two dimensions along which workflows can be abstracted, 1) process-wise, where a simplified view of the process encoded in the workflow is given and 2) data-wise, where significant data products (in- termediary or final) that a workflow generate is to generate are pinpointed. Scientists perform workflow abstractions in the following ways: Grouping operations into subworkflows. Subworkflows are a design construct for developing modular workflows, which is an established best practice in workflow development [8]. The major driver for sub-workflows is to create modules of significant or re-usable function within or across workflows. In practice, however, subworkflows can also be used for purposes other than functional modularity. For example, the workflow illustrated in Figure 1 contains a subworkflow that simply concatenates two parameters. Such subworkflows are created for aesthetic purposes to simplify the design, as opposed to abstracting significant functionality.• Sinking ports expected to hold significant intermediary data artifacts to workflow outputs. The links tagged with stars in Figure 1 correspond to such sinking activity. We refer to these as “bookmarked data links” in the rest of the paper. Bookmarking is done when the user wants to report significant side-products alongside the intended outputs of the workflow, and encodes this as part of the workflow design. Hard-coded bookmarking is a poor practice for workflow design, as it clutters the output signature of the workflow. Moreover, it creates more complexity process- wise while trying to provide abstract data-wise.• Using libraries of components engineered to work to- gether. Several workflow systems provide libraries of com- ponents that have well-defined and interoperable-by-design invocation interfaces. By solely using modules from such libraries, compact workflows with little/no data adapter operations can be devised. This approach, however, is highly dependent on those curated libraries, and as soon as one tries to incorporate resources from other contexts or libraries, then one needs data adaptation and organization. Producing attractive publications
• #11: In this paper we propose the generation of workflow description summaries to tackle complexity in experiment reporting scenarios. Our approach has the following charac- teristics and contributions: We focus on data-driven work- flows, which is the paradigm supported by most scientific workflow systems. We provide a graph based model for representing data-flows. We use graph re-writing as the method of summarization. We re-write workflow graphs with rules built on 1) semantic annotations that depict the data- oriented function of operations within workflows, which we call motifs, and 2) a set of workflow reduction primitives that we introduce. Rules associate motifs, and combina- tions of thereof, with reduction actions that specify the abstraction operation to be performed. Re-write rules capture scientist-specified strategies on how summarization should occur. Based on observations on workflows from the largest public repository of workflows, we elaborate two sample strategies that aim to abstract out “mundane” data adapter and organizer steps in workflows. In order to highlight the characteristic of primitives and asses our approach, we conducted an experiment where we summarized 30 real-world scientific workflows. Summarization is done by using the two sample strategies and we compare the results against summaries that are manually generated by the workflow designers as the ground truth. Producing attractive publications
• #12: Scientific Workflow MotifsAt the heart of our approach lies the exploitation of information on the data processing nature of activities in workflows. We propose that this information is provided through semantic annotations, which we call motifs. We use a light-weight ontology, which we developed in previous work [8] based on an analysis of 200+ workflows (Ac- cessible: http://purl.org/net/wf-motifs). The motif ontology characterizes operations wrt their Data-Oriented functional nature and the Resource-Oriented implementation nature. We refer the reader to [8] for the details, here we briefly introduce some of the motifs in light of our running example. • Data-Oriented nature. Certain activities in workflows, such as DataRetrieval, Analysis or Visualizations, perform the scientific heavy lifting in a workflow. The gene annotation pipeline in Figure 1) collects data from various resources through several retrieval steps ( see “get enzymes by gene” , “get genes by enzyme” opera- tions). The data retrieval steps are pipelined to each other through use of adapter steps, which are categorized with the DataP reparation motif. Augmentation is a sub-category of preparation operations. Augmentation decorates data with resource/protocol specific padding or formatting. (The sub- workflow “Workflow89” in our example is an augmenter that builds up a well formed query request out of given parameters.) Extraction activities perform the inverse of augmentation, they extract data from raw results returned from analyses or retrievals (e.g. SOAP XML messages). Splitting and Merging are also a sub-category of data preparation, they consume and produce the same data but change its cardinality (see the “Flatten List” steps in our example). Another general category is DataOrganization activities, which perform querying and organization func- tionsoverdatasuchas,F iltering,J oiningandGrouping. The “Remove String Duplicates” method in our example is an instance of the filtering motif. Another frequent motif is DataMoving. In order to make data accessible to external analysis tools data may need to be moved in and out of the workflow execution environment such activities fall into this category (e.g. upload to web, write to file etc). The “Get Image From URL 2” operation in our genomics workflow is an instance of this motif.• Resource-Oriented nature, which outlines the implemen- tation aspects and reflects the characteristic of resources that underpin the operation. Classifications in this category are: Human − Interactions vs entirely Computational steps, Statefull vs Stateless Invocations and using Internal (e.g. local scripts, sub-workflows) vs External (e.g. web service, or external command line tools) computational artifacts. In our example, all data retrieval operations are realized by external, stateless resources (web services), whereas all data preparation operations are realized with Producing attractive publications
• #13: We address “pure” data flows, i.e. those, in which the order of execution is determined entirely by the availability of data rather than any explicit control-flow constructs (e.g. loops, conditionals).Definition 1. We define a data-driven workflow as a directed acyclic graph (DAG) using the following pair W = ⟨N,E⟩, where N is a set of nodes representing the operations that constitute the workflow and the input and output ports of such operations: N = (NOP ∪ NP ). NOP nodes represent analysis operations in the workflow.The set of ports NP is composed of four sets, depending on whether the port is an input or output and whether it belongs to an operation or is a source/sink port of the entire workflow:NP=(NI∪NO∪NIW∪NOW).Nodes are connected using three types of edges: E = (EP −>OP ∪ EOP −>P ∪ EP −>P ), where EP −>OPconnect input ports to the operations they belong to, EOP−>P connect operations to their output ports, and EP −>Pconnect the output port of an operation to the input port of a following operation. In Figure 2 we provide a mapping from the fragment of our genomics workflow to the graph representation, depicting operation nodes (white ellipses), port nodes (gray circles). (Not depicted in this figure, is an inferred influence relationship from each input of an operation to each output the operation. This relation is not utilized in graph re- writing but is used to query workflow descriptions and their summaries).Definition 2. Producing attractive publications
• #14: A data-flow decorated with motif annota- tions is 3-tuple: W = ⟨N,E,motifs⟩, where motifs is a function that maps each graph node n ∈ N, be it an operation or port, to its motifs. Operation nodes could have motif annotations from operation, resource perspectives, and port nodes can have annotations from the data perspective (omitted in this paper due to space limitations).As an example, we illustrate below two motif annotations of two operations in our running example. It is worth mentioning that a given node can be associated with multiple motifs.motifs(color pathway by objects) = {⟨m1 : Augmenter⟩} motifs(Get Image From URL 2) = {⟨m2 : DataRetrieval⟩} Producing attractive publications
• #16: 3) Collapse: The collapse primitive is a way to eliminate an operation from the workflow not by introducing indi- rect data links, but by introducing (sometimes redundant, therefore weak) composites. Collapse can be performed either upstream or downstream. We define the function collapse op downstream : ⟨W,op⟩ → ⟨W′,M⟩ as one, which takes an annotated workflow definition, and collapses the designated operation op into all operations that immedi- ately succeed it (a.k.a. collapse hosts) via a direct dataflow link. Collapse is realized by creating composites of op exclusively with each of its hosts (i.e. ignoring relations with other hosts). Upstream collapse is realized by creating composites of op with its immediate predecessors.Motif Behavior: The operation that is collapsed onto another does not carry forward its motifs to the newly generated composite. Whereas the motifs of the operation onto which op is collapsed, i.e. the host propagates its motifs to the composite. Also, the composite inherits the identifier label of the host operation.Constraint Checks: In order for an operation to be collapsed downstream it is required that it is succeeded by at least one operation. The reverse constraint is applicable when collapsing upstream. To prevent against creating cyclic summaries, the host operations on which an operation is to be collapsed shall not co-exist on a data-flow path.Dataflow Characteristic: In case there are more than one host operations, on which collapse will occur the resulting composites are “weak” due to the exclusivity of composi- tions. Collapsing onto multiple hosts creates a redundancy of ports on generated composites and causes the “spreading” of influence relations, where a weak-composite does not provide the full picture of influence relations. For example, in Figure 5, the operation Z is collapsed upstream onto X and Y . The newly created composites X′ and Y ′ are weak in the sense that they separately provide information on some but not all of the inputs that have influenced the generation of e.g. output O3′ and O3′′ which both map to O3. Producing attractive publications
• #18: Composition: We define the function compose ops : ⟨W, op1, op2⟩ → ⟨W ′ , M ⟩ as one, which takes an annotated workflow graph, and replaces operations op1 and op2 with their composite op3 and returns the updated workflow graph. The primitive also returns a set of mappings between the ports of the resulting workflow and the input workflow.Motif Behavior: The composite op3 inherits motifs of both of its constituents. Operation nodes are identified by their labels, in the composition primitive the label for the composite is a newly generated identifier.Constraint Checks: Operations that have a direct datalink among each other can be subject to composition. To prevent the creation of cyclic summaries, the two operations to be composed shall not co-exist on another indirect data flow path other than the direct link(s) connecting them.Dataflow Characteristic: The composition primitive may result in loss of information on negative influence relations among artifacts. For example, in Figure 3, in the original workflow the input i9 is influential in the creation of outputs o5 and o6 but not in the creation of o4. When operations Z and T are composed however, we will have the information stating that i9′ which maps to i9 is influential in the creations of not only o5′ o6′ but also of o4′. Producing attractive publications
• #19: 2) Elimination: We define the function eliminate op : ⟨W,op⟩ → ⟨W′,M⟩ as one, which takes an annotated workflow definition, and eliminates a designated operation node from it. Elimination results in a set of indirect data links connecting the predecessors of the deleted operation to its successors. The source and target ports of these new indirect links are the cartesian product of the to-be-deleted operation’s inlinks’ sources and its outlinks’ targets. As with other primitives elimination also returns a set of mappings between the ports of the resulting workflow and the input workflow. Motif Behavior: The designated operation is eliminatedtogether with its motifs and these are not propagated to any other node in the workflow graph.Constraint Checks: There are no particular constraint checks for the elimination primitive.Dataflow Characteristic: Within the conventional defini- tion of a workflow, ports are connected with direct datalinks, which correspond to data transfers i.e. the data artifact !-# appearing at the source port is transferred to the target port as-is. In the indirect link case, however, there is no direct data transfer as the source and target artifacts are dissimilar,%&# and some “eliminated” data processing occurs inbetween. Producing attractive publications
• #20: In this section, we present two sample strategies for sum- marizing workflow graphs. Such strategies are dependent on the domain in which workflows exist, the characteristics of workflows and the user preferences on summary. These open-world workflows mainly collate datasets and tools from 3rd party autonomous resource providers. Observations on Taverna workflows and their textual narratives supplied by users pointed that:• Scientists rarely report on data adapter steps within their brief textual summaries of workflows. They choose to mention steps that they deem significant, from a scientific perspective e.g. Data Retrieval, Analysis, Visualization etc. in their summaries.• Scientists also report on the intermediary Inputs and Out- puts of significant steps. This is done either by mentioning data artifacts in text summary or manually bookmarking them to workflow outputs. Bookmarking, however, is done at stages of data processing, where data is in its the mostreporting friendly form such as 1) Data with Reduced Cardinality and 2) Data that is stripped of Implementation Specific Payload. Consequently, we elaborate the following strategies: Producing attractive publications
• #21: Summary-By-Elimination: Requires minimal amount of annotation effort and operates with a single rule. We have only classified the scientifically significant operations to denote their motifs, and all remaining operations in the workflow are designated to be DataPreparation steps. The following reduction rule is then used to summarize the workflow graph W. Summary-By-Collapse: Requires annotation of work- flow with motif classes having more specificity and a more fine-grained set of rules. Consequently annotations on data preparation steps are more specific designating what kind of preparation activity is occurring (e.g. Merging, Splitting, Augmentation etc). We use these annotations to remove those data preparation steps from the work- flow by the collapse primitive and collapsing in the di- rection so as to retain the ports that carry more reporting friendly artifacts. We collapse Augmentation, Splitting andRef erenceGenerationoperationsdownstreamastheir outputs are less favored than their inputs, whereas we col- lapse Extraction, Merging, ReferenceResolution and all DataOrganization steps downstream as their outputs are preferred over inputs in a summary. Two rules in this approach are given below: Producing attractive publications
• #23: In our evaluation we annotated the workflow corpusmanually using the motif ontology. The effort that goes into the design of workflows can be significant (up to several days for a workflow). When compared to design, the cost of annotation is minor (as it amounts to single attribute setup per operation). Annotation can be (semi)automated through application of mining techniques to workflows and execution traces. It is also possible to pool annotations in service registries or module libraries and automatically propagate annotations to operations that rare re-used from the registry. Instead of the motif ontology it is also possible to use other ontologies [10] to characterize operations in workflows. Producing attractive publications
• #24: For example, In Figures 7 and 8 we provide the result of applying the two summarization rules to our sample genomics workflow. Both strategies result in a process-wise compact version of the workflow that contains only the Data Retrieval operations that have survived elimination after exhaustive execution of the rules. The summaries convey the causal ordering of these operations and their direct/indirect Producing attractive publications
• #29: A handful of researchers have proposed solutions to abstract workflows using user- defined views [3], or user-controlled manual redaction of provenance [4], [7]. These solutions expect the user to manually designate significant steps or directly control the editing process. Moreover, none of them take into account the nature of data-processing performed by the steps in the workflow, e.g., whether it is utility operation or whether it performs a scientifically significant function. Producing attractive publications
• #30: Motifs are used in conjunction with reduction primitives to build summarization rules. The objective of rules is to determine how to reduce the workflow when certain motifs or combinations of motifs are encountered. We propose graph transformation rules of the form L 􏰀 P, where L specifies a pattern to be matched in the host graph specified in terms of a workflow-specific graph model and motif decorations and P specifies a workflow reduction primitive to be enacted upon the node(s) identified by the left-hand- side L. Primitive P captures both the replacement graph pattern and how the replacement is to be embedded into the host graph. Our approach is a more controlled primitive based re-writing mechanism, rather than a fully declarative one (as in traditional graph re-writing). This allows us to: 1) Preserve data causality relations among operations together with the constraints that such relations have (e.g. acyclicity for Scientific Datalows) 2) We can allow the user to specify high-level reduction policies by creating simple <motif, primitive> pairs. It would be a big undertaking to expect the users, who are not computer scientists or software engineers, to specify re-writes in a fully declarative manner (i.e. a matching pattern, a replacement pattern and embedding info) Producing attractive publications
• #31: Producing attractive publications
• #32: Producing attractive publications