Results may vary: Collaborations Workshop, Oxford 2014

Editor's Notes

• #2: how, why and what matters benchmarks for codes plan to preserve repair on demand description persists use frameworks partial replication approximate reproduction verification
• #3: Multidimensional paper
• #5: hand-wringing, weeping, wailing, gnashing of teeth. Nature checklist. Science requirements for data and code availability. attacks on authors, editors, reviewers, publishers, funders, and just about everyone.
• #7: Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328
• #8: Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome Daniel Garijo, Sarah Kinnings, Li Xie, Lei Xie, Yinliang Zhang, Philip E. Bourne mail, Yolanda Gil mail Published: November 27, 2013 DOI: 10.1371/journal.pone.0080278
• #11: Added afterwards. 1. Required as condition of publication, certain exceptions permitted (e.g. preserving confidentiality of human subjects) 2. Required but may not affect editorial/publication decisions 3. Explicitly encouraged/addressed; may be reviewed and/or hosted 4. Implied 5. No mention 59% of papers in the 50 highest-IF journals comply with (often weak) data sharing rules. Alsheikh-Ali et al Public Availability of Published Research Data in High-Impact Journals. PLoS ONE 6(9) 2011
• #14: Pressure from top, pressure from below Squeeze http://pantonprinciples.org/
• #15: anyone anything anytime publication access, data, models, source codes, resources, transparent methods, standards, formats, identifiers, apis, licenses, education, policies “accessible, intelligible, assessable, reusable”
• #18: The letter or the spirit of the experiment indirect and direct reproducibility Reproduce the same affect? Or same result? Concept drift towards bottom. As an old ontologist I wanted an ontology or a framework or some sort of property based classification.
• #19: / Minute Taking It examines the debate between Robert Boyle and Thomas Hobbes over Boyle's air-pump experiments in the 1660s. In 2005, Shapin and Schaffer were awarded the Erasmus Prize for this work.
• #23: Preservation - Lots of copies keeps stuff safe Stability dimension Add two more dimensions to our classification of themes A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System Overlap of course Static vs dynamic. GRANULARITY This model for audit and target of your systems overcoming data type silos public integrative data sets transparency matters cloud Recomputation.org Reproducibility by ExecutionRun It Reproducibility by InspectionRead It Availability – coverage Gathered: scattered across resources, across the paper and supplementary materials Availability of dependencies: Know and have all necessary elements Change management: Data? Services? Methods? Prevent, Detect, Repair. Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding Documentation vs Bits (VMs) reproducibility Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
• #24: Example of an extreme of the software issue Multi-code experiments platform libraries, plugins Infrastructure components, services infrastructure
• #25: Simplify Track Versions and retractions Error propagation Contributions and credits Fix Workflow repair, alternate component discovery, Black box annotation Rerun and Replay Partial reproducibility: Replay some of the workflow A verifiable, reviewable trace in people terms Analyse Calculate data quality & trust, Decide what data to keep or release Compare to find differences and discrepancies S. Woodman, H. Hiden, P. Watson,  P. Missier Achieving Reproducibility by Combining Provenance with Service and Workflow Versioning. In: The 6th Workshop on Workflows in Support of Large-Scale Science. 2011, Seattle
• #26: Client package (currently under development, will be available via Python Package Index (PyPI) for installation for all major platforms (Linux, Mac, Windows) Allows for calling Taverna Workflows available via Taverna Player List of available workflows can be retrieved from the BioVel Portal (Taverna Player) Users can enter the input values using Ipython Notebook (these values can be then results of the code previously run in the Notebook The outputs from running the workflow (the results) are returned to the Notebook and processed further The full workflow run and the overall process (provenance) can be saved in the Ipython Notebook format For an example (static for now), see http://nbviewer.ipython.org/urls/raw.githubusercontent.com/myGrid/DataHackLeiden/alan/Player_example.ipynb?create=1
• #29: Used by gigscience
• #30: Preservation - Lots of copies keeps stuff safe Stability dimension Add two more dimensions to our classification of themes A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System Overlap of course Static vs dynamic. GRANULARITY This model for audit and target of your systems overcoming data type silos public integrative data sets transparency matters cloud Recomputation.org Reproducibility by ExecutionRun It Reproducibility by InspectionRead It Availability – coverage Gathered: scattered across resources, across the paper and supplementary materials Availability of dependencies: Know and have all necessary elements Change management: Data? Services? Methods? Prevent, Detect, Repair. Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding Documentation vs Bits (VMs) reproducibility Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
• #31: Instrumented desktop or server tools
• #32: Variety: common metadata models rich metadata collection ecosystem Validity: auto record of experiment set-up, citable and shareable descriptions curation, publication, mixed stewardship third part availability model executability citability, QC/QA. trust. Social issues of understanding the culture of risk, reward, sharing and reporting.
• #33: This article by Phil Bourne et al doesn’t have any data sets deposited in repositories, but does include data in tables in the PDF, which are also available in the XML provided by PLoS. Here, Utopia has spotted that there’s a table of data (notice the little blue table icon to the left of the table). Clicking on the icon opens a window with a simple ‘spreadsheet’ of the data extracted from the paper, which you can then export in CSV to a proper spreadsheet of your choice. You can also scatter-plot the data to get a quick-and-dirty overview of what’s in the table.
• #35: Preservation - Lots of copies keeps stuff safe Stability dimension Add two more dimensions to our classification of themes A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System Overlap of course Static vs dynamic. GRANULARITY This model for audit and target of your systems overcoming data type silos public integrative data sets transparency matters cloud Recomputation.org Reproducibility by ExecutionRun It Reproducibility by InspectionRead It Availability – coverage Gathered: scattered across resources, across the paper and supplementary materials Availability of dependencies: Know and have all necessary elements Change management: Data? Services? Methods? Prevent, Detect, Repair. Execution and Making Environments: Skills/Infrastructure to run it: Portability and the Execution Platform (which can be people…), Skills/Infrastructure for authoring and reading Description: Explicit: How, Why, What, Where, Who, When, Comprehensive: Just Enough, Comprehensible: Independent understanding Documentation vs Bits (VMs) reproducibility Learn/understand (reproduce and validate, reproduce using different codes) vs Run (reuse, validate, repeat, reproduce under different configs/settings)
• #36: Recomputation not reproducibility ID it to Cite It: ORCID (people), DOI (data, models, tools ...) Tracking: local helper systems to instrument and track provenance Science as a Service: Virtual Machines, Cloud Appliances, Hosted platforms deploys on your behalf, no installations, common platforms Libraries and Repositories: with rich documentation Publish: executable papers, companion web sites, embedded electronic lab notebooks, active publications Explication of experimental mechanics: pipelines, workflows, script systems with documentation, common tools
• #37: the reproducibility ecosystem For peer and author complicated and scattered - super fragmentation – supplementary materials, multi-hosted, multi-stewarded. we must use the right platforms for the right tools The trials and tribulations of review Its Complicated www.biostars.org/ Apache Service based ScienceScience as a Service
• #39: A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Virtual machines are separated into two major classifications, based on their use and degree of correspondence to any real machine: System Zhao, Gomez-Perez, Belhajjame, Klyne, Garcia-Cuesta, Garrido, Hettne, Roos, De Roure and Goble. Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012 Reproducibility success is proportional to the number of dependent components and your control over them” Many reasons why. Change / Availability Updates to public datasets, changes to services / codes Availability/Access to components / execution environment Platform differences on simulations, code ports Volatile third-party resources (50%): Not available, available but inaccessible, changed Prevent, Detect, Repair
• #45: The only equation I have in the talk.
• #46: Added after LISC
• #48: ENCODE threads exchange between tools and researchers bundles and relates digital resources of a scientific experiment or investigation using standard mechanisms
• #52: http://www.youtube.com/watch?v=p-W4iLjLTrQ&list=PLC44A300051D052E5 Our collaboration with ISA/GigaScience/nanopublication is finally being written up and will be submitted to ECCB this Friday. We will upload a copy to Arxiv after the deadline. - We will continue our workshop at ISMB, with BioMED Central. And Kaitlin will also join us on the Panel. You can find more details about agenda and panel planning in other emails. Posted on December 11, 2013 by Kaitlin Thaney Part of the Science Lab’s mission is to work with other community members to build technical prototypes that move science on the web forward. In particular, we want to show that many problems can be solved by making existing tools and technology work together, rather than by starting from scratch.The reason behind that is two-fold: (1) most of the stuff needed to change behaviors already exists in some form and (2) the smartest minds are usually outside of your organization. Our newest project extends our existing work around “code as a research object”, exploring how we can better integrate code and scientific software into the scholarly workflow. The project will test building a bridge that will allow users to push code from their GitHub repository to figshare, providing a Digital Object Identifier for the code (a gold standard of sorts in science, allowing persistent reference linking). We will also be working on a “best practice” standard (think a MIAME standard for code), so that each research object has sufficient documentation to make it possible to meaningfully use. The project will be a collaboration of the Science Lab with Arfon Smith (Github; co-founder Zooniverse) and Mark Hahnel and his team at figshare. Why code? Scientific research is becoming increasingly reliant on software. But despite there being an ever-increasing amount of the academic process described in code, research communities do not yet treat these products as a fundamental component or  “first-class research object” (see our background post here for more). Up until recent years, the sole “research object” in discussion was the published paper, the main means of packaging together the data, methods and research to communicate findings. The web is changing that, making it easier to unpack the components such as data and code for the community to remix, reuse, and build upon. A number of scientists are pushing the envelope, testing out new ways of bundling their code, data and methods together. But outside of copy and pasting lines of code into a paper or, if we’re lucky, having it included in a supplementary information file alongside a paper, the code is still often separated from the documentation needed for others to meaningfully use it to validate and reproduce experiments. And that’s if it’s shared openly at all. Code can go a long way in helping academia move toward the holy grail that is reproducibility. Unfortunately, academics whose main research output is the code they produce, often cannot get the recognition they deserve for creating it. There is also a problem with versioning:  citing a paper written about software (as is common practice), gives no indication of which version, or release in GitHub terms, was used to generate the results. What we’re testing figshare and GitHub are two of the leading repositories for data and code (figshare for data; GitHub for code). Open data repositories like figshare have led the way in recent years in changing our practices in relation to data, championing the idea of data as a first-class research object. figshare and others such as Harvard’s Dataverse and Dryad have helped change how we think of data as part of the research process, providing citable endpoints for the data itself that the community trusts (DOIs), as well as clear licensing and making it easy to download, remix, and reuse information. One of the main objectives here is that the exact code used in particular investigations, can be accessed by anyone and persists in the form it was in when cited. This project will test whether having a means of linking code repositories to those commonly used for data will allow for software and code to be better incorporated into existing credit systems (by having persistent identifiers for code snapshots) and how seamless we can make these workflows for academics using tools they are already familiar with. We’ve seen this tested with data over recent years, with sharing of detailed research data associated with increased citation rates (Piwowar, 2007). This culture shift of publishing more of the product of research is an ongoing process and we’re keen to see software and code elevated to the same status as the academic manuscript. We believe that by having the code closely associated with the data it executes on (or generates) will help reduce the barriers when trying to reproduce and build upon the work of others. This is already being tested in the reverse, with computational scientists nesting their data with the code in GitHub (Carl and Ethan, for example). We want to find out if formally linking the two to help ease that pain will change behavior. We are also looking to foster a culture of reuse with academic code. While we know there are lots of variables in this space, we are actively soliciting feedback from the community to help determine best practices for licensing and workflows. How to get involved (UPDATE: Want to help us test? Instead of sending us an email, how about adding yourself to this issue in our GitHub repository? More about that here.) Mark and Arfon will be joining us for our next Mozilla Science Lab community call on December 12, 2013. Join us to hear more about the project. Have a question you’d like to ask? Add it to the etherpad! We’re also looking for computational researchers and publishers to help us test out the implementation. Shoot us an email if you’d like to participate. Posted in Uncategorized.
• #57: Its people!!!
• #59: from make reproducible to born reproducible tools/repositories needed, maintained and incorporated into working practices researchers will need to adapt their practices, be trained to reproduce, cost and responsibility should be transparent, planned for, accounted and borne collectively we all should start small, be imperfect but take action. Today. spreading the cost cradle to grave reproducibility tools, processes, standards combine making & reporting just enough, imperfect cost in train up and support planning We cannot sacrifice the youth Protect them….a new generation Ecosystem of support tools navigation