Why Workflows Break

Editor's Notes

• #3: So why do we bother about reproducibility. Because results that scientists reach are not only insights. They are used to ensure progress in practice. Moreover, their results is built upon by other scientists to reach new results. Therefore, testing claimed results is crucial for science to be self-correcting. ============ Important scientific results not only give insight but also lead to practical progress. The ability to test results is crucial for science to be self-correcting. A hallmark of the scientific method is that experiments should be described in enough detail that they can be repeated and perhaps generalized. The idea in natural science is that if a scientist claims an experimental result, then another scientist should be able to check it. Similarly, in a computational environment, it should be possible to repeat a computational experiment as the authors have run it or to change the experiment to see how robust the authors’ conclusions are to changes in parameters or data (a concept called workability).
• #4: As an example, In this article, a researcher has found that many basic studies on cancer are unreliable, with grim consequences for producing new medicines in the future. This supports the need for testing research results. http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 "landmark" publications -- papers in top journals, from reputable labs -- for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.
• #5: In natural sciences, when a scientists claims an experimental result, then another scientists should be able to check it. This should also be possible in a computational environment to test the experiment as the authors have run it, or even change the experiment to see how robust the conclusions reached by the authors.
• #6: In natural sciences, when a scientists claims an experimental result, then other scientists should be able to check it. This should also be possible in a computational environment to test the experiment as the authors have run it, or even change the experiment to see how robust the conclusions reached by the authors. To do so, we need information that documents the experiment and specify the computational environment in which it was ran.
• #8: I am quoting these two quotes to underline the fact that the experiment results are not enough and that we need information about the process whereby such results were produced.
• #9: This is partly witnessed by existing workflow repositories, notably myExperiment and crowdLab, which provide scientists with the means to store, share, publish and reuse workflows. One of the good features of scientific workflows vis a vis reproducibility is that they are repeatable and configurable. Well in principle. Unfortunately, our experience suggests that workflows are likely to suffer over time from decay hindering or reducing the ability to execute them and reproduce the same results.
• #12: Analyse a sample of real workflows to determine if they suffer from decay and the reasons that caused their decay
• #17: C. Missing execution environment The execution of a workflow may rely on a particular local execution environment, for example, a local R server or a specific version of workflow execution software. Some of our test workflows exhibit this type of decay. Taverna often provides sufficient information about missing libraries, and sometimes workflow descriptions provide a warning about the requirement for a specific library. This type of decay appears to be fixable by installing the missing software, albeit requiring some effort. D. Insufficient descriptions about workflows Sometimes a workflow workbench cannot provide sufficient information about what caused the failure of a workflow run. Additional descriptions in the workflow can play an important role in assisting users reusing the workflows to understand the purpose of the workflow and its expected outcomes.
• #19: C. Missing execution environment The execution of a workflow may rely on a particular local execution environment, for example, a local R server or a specific version of workflow execution software. Some of our test workflows exhibit this type of decay. Taverna often provides sufficient information about missing libraries, and sometimes workflow descriptions provide a warning about the requirement for a specific library. This type of decay appears to be fixable by installing the missing software, albeit requiring some effort. D. Insufficient descriptions about workflows Sometimes a workflow workbench cannot provide sufficient information about what caused the failure of a workflow run. Additional descriptions in the workflow can play an important role in assisting users reusing the workflows to understand the purpose of the workflow and its expected outcomes.
• #23: Based on our findings, we bundle workflow specifications together with auxiliary information for mitigate its decay, e.g., example data inputs, annotations describing the workflow, provenance traces. The resulting aggregation is term Research Object, which abstractions that are used for workflow preservation. These are designed and tooling is developed around in the context of the Wf4Ever project To verify that a given research object provides the information necessary for preserving a workflow against decay, we adopt a checklist-based approach. Checklists are a well- established tool for guiding practices to ensure safety, quality and consistency in the conduct of complex operations [11], More recently, they have been adopted by the biological research community to promote consistency across research datasets [33] Checklists are well established tool for guiding practices to ensure safety, quality and consistency to conduct complex operations. In our cases, we use them to specify the minimum information needed to prevent workflow decay.
• #24: The checklists are expressed using the Minim model
• #25: As a proof of concept