Towards an Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Systems and Applications
0 likes115 views
The document discusses advancements and challenges in scientific workflow systems, emphasizing the need for improved workflow management capabilities to handle complex and large-scale datasets. It outlines various research initiatives and community summits aimed at fostering collaboration, addressing interoperability, and ensuring fairness in workflow practices. The manuscript is a collaborative effort by UT-Battelle LLC for the U.S. Department of Energy, and it highlights the ongoing evolution of scientific workflows in the context of high-performance computing and emerging technologies.
1 of 15
Download to read offline
Ad
Recommended
A Semantic-Based Approach to Attain Reproducibility of Computational Environm...
A Semantic-Based Approach to Attain Reproducibility of Computational Environm...Idafen Santana Pérez This document presents a semantic-based approach aimed at ensuring reproducibility in scientific workflows, highlighting tools and methods such as the WICUS ontology network and the PEGASUS system. It discusses the importance of capturing, preserving, and reusing computational environments and provides a use case in astronomical imaging. The conclusion emphasizes the potential for extending this model to various workflows and improving annotation processes in the future.
Thrombus Training Dec. 2013
Thrombus Training Dec. 2013CREATIS This document provides an overview of various tools for code storage, data storage, data visualization, mesh editing, and medical image analysis available on the CREATIS computing cluster. It describes version control systems like Git and SVN for code storage, data storage options like TUX-FTP and iRODS, the ParaView application for scientific visualization, mesh editing tools like MeshLab, FreeCAD and Blender, and medical image analysis tools including MeVisLab, itkSNAP, GDCM and MITK. Tutorials and support contacts are provided for many of the applications.
Simulagora (Euroscipy2014 - Logilab)
Simulagora (Euroscipy2014 - Logilab)Logilab The document discusses reproducibility in e-science and presents the Simulagora platform, which aids researchers in producing and reproducing studies. It highlights key challenges in replication, such as hardware compatibility and software environment, emphasizing the need for tools that ensure traceability of data and software. Future developments of Simulagora are also mentioned, including enhanced collaboration features and remote access capabilities for efficient e-scientific research.
What's New in Cytoscape
What's New in CytoscapeKeiichiro Ono Cytoscape is an open-source platform for biological network data integration, analysis, and visualization, recently updated to version 3.4 with new features such as a network view manager, app collections, and performance improvements. The platform supports web applications via Cytoscape.js, which allows users to share and visualize network data interactively. The document discusses enhancements aimed at improving user experience and acknowledges the importance of a strong community for continuous development and support.
Scientific
Scientific marpierc Apache Airavata is an open source science gateway software framework that allows users to compose, manage, execute, and monitor distributed computational workflows. It provides tools and services to register applications, schedule jobs on various resources, and manage workflows and generated data. Airavata is used across several domains to support scientific workflows and is largely derived from academic research funded by the NSF.
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsKeiichiro Ono The document discusses building reproducible network data analysis and visualization workflows using REST APIs and containerization. It aims to solve problems with complex software stacks that are difficult to set up and not reproducible. The goal is to create reproducible and scalable "dry experiments" using Docker containers, GitHub for source code sharing, Jupyter notebooks as electronic lab notebooks, and the cyREST module for the Cytoscape network analysis software. Examples of scenarios using local workstations and cloud computing are presented, as well as a demo and future plans.
The Exascale Computing Project and the future of HPC
The Exascale Computing Project and the future of HPCinside-BigData.com The U.S. Department of Energy's Exascale Computing Project (ECP) aims to develop high-performance computing capabilities to address national challenges in science, healthcare, and security. With a budget of $1.7 billion and involvement from six core national laboratories, ECP focuses on creating an advanced software stack and fostering collaborations with industry. The initiative seeks to enhance U.S. economic competitiveness and improve the quality of life by delivering innovative simulation and data science solutions.
seanresume15-a
seanresume15-aSean Lynch Sean Lynch has over 15 years of experience leading teams that deliver software projects on time and under budget. He currently works as a Senior System Analyst and Solutions Architect at Blue Cross Blue Shield, where he has helped design and implement their large national data warehouse system. Previously he has worked as a consultant for several insurance and financial companies, managing projects and teams. He has a focus on quality results and extensive experience across the software development lifecycle.
A Query Model for Ad Hoc Queries using a Scanning Architecture
A Query Model for Ad Hoc Queries using a Scanning ArchitectureFlurry, Inc. The document presents the Burst system developed by Flurry, designed to support low-latency, single-pass ad hoc queries over large datasets of mobile application analytics. It focuses on a scan-only architecture, simplifying query execution to allow developers to interactively explore metrics with sub-second response times, contrasting traditional systems that necessitate lengthy waits for results. The Burst system, which is integrated with Flurry's Explorer product, effectively handles complex causal queries and data models for hundreds of thousands of applications, significantly enhancing query performance over legacy systems like Hadoop.
Next.ml Boston: Data Science Dev Ops
Next.ml Boston: Data Science Dev OpsEric Chiang This document discusses challenges in deploying machine learning models into production and potential solutions. It covers:
1. Issues with reproducibility due to dependencies and environment configurations when models are trained and deployed.
2. Problems with serializing models and transferring them between different versions of libraries and software stacks.
3. How containers can help address these issues by encapsulating the full runtime environment and dependencies of a model.
4. Managing both models and Docker containers is still required when using this approach.
A General Purpose Extensible Scanning Query Architecture for Ad Hoc Analytics
A General Purpose Extensible Scanning Query Architecture for Ad Hoc AnalyticsFlurry, Inc. The document presents Burst, an analytic query system designed for low-latency ad hoc analysis over large datasets, utilizing hardware-efficient scanning techniques and a flexible query language. It details the architecture's components, including an incremental ingestion pipeline for data processing, and explains how the system transforms ad hoc declarative queries into efficient scanning plans executed across distributed nodes. Moreover, it highlights the performance optimizations, schema independence, and future enhancements under development for improved efficiency and scalability.
Pegasus-Poster-2016-final-v2
Pegasus-Poster-2016-final-v2Samrat Jha Pegasus is a workflow management system that maps abstract workflows to executable workflows that can run on different execution environments. It consists of three main components - the Pegasus Mapper, HTCondor DAGMan, and HTCondor Schedd. Pegasus provides APIs in multiple languages including R to generate abstract workflows and can easily scale to workflows with millions of tasks. It handles mapping workflows to resources, data movement between sites, monitoring and debugging of workflows. Pegasus was used to run EPA and Bioconductor genomic analysis workflows across multiple sites by partitioning tasks.
SDCSB Advanced Tutorial: Reproducible Data Visualization Workflow with Cytosc...
SDCSB Advanced Tutorial: Reproducible Data Visualization Workflow with Cytosc...Keiichiro Ono This document provides an overview of a tutorial on building reproducible network data visualization workflows using Cytoscape and IPython Notebook. The tutorial will cover integrating data, analyzing networks, visualizing results, and preparing outputs for publication. It will demonstrate setting up a portable data analysis environment using Docker and sharing work through GitHub. The bulk of the tutorial will focus on using IPython Notebook as an electronic lab notebook for interactive and reproducible experiments with Cytoscape.
Converting scripts into reproducible workflow research objects
Converting scripts into reproducible workflow research objectsKhalid Belhajjame 1) The document presents a methodology to convert script-based experiments into reproducible workflow research objects (WROs). This addresses issues of understanding, reusing, and reproducing experiments conducted through scripts.
2) The methodology involves 5 steps: generate an abstract workflow, create an executable workflow, refine the workflow, record provenance data, and annotate and check the quality of the conversion.
3) Applying the methodology to a molecular dynamics simulation case study, the authors demonstrate how scripts can be transformed into WROs containing workflows, annotations, provenance data, and other resources needed for reproducibility.
r1501e
r1501eGeorge Vamos George Vamos has over 30 years of experience as a lead software engineer developing mission critical systems. He has expertise in medical data management systems, aerospace command and control systems, and business databases. Some of his accomplishments include developing medical data warehouses that can adapt to changing data formats, and databases shared by spacecraft tracking systems where much of the schema is auto-generated from requirements. He holds patents related to neural implant management technology.
A Sightseeing Tour of Prov and Some of its Extensions
A Sightseeing Tour of Prov and Some of its ExtensionsKhalid Belhajjame This document provides an overview of the PROV provenance model and some of its extensions. It discusses the motivation for provenance, the history and development of the PROV model, its key concepts of entities, activities, and agents. It also describes extensions like ProvONE and PAV that build upon PROV to model workflow and scientific provenance.
Keyur_Joshi_resume - Copy
Keyur_Joshi_resume - CopyKeyur Joshi Keyur Joshi is seeking a position as a Software Developer/Engineer. He has a Master's degree in Computer Science from SUNY Buffalo and a Bachelor's degree in Electronics and Communications Engineering. He has over 2 years of experience as a Software Developer at Kuberre Systems, where he develops web applications using technologies like Smart GWT, RESTful services, and DynamoDB. His skills include programming languages like C/C++, Java, Python, and databases like MySQL, Oracle, and Hadoop. He has completed projects in areas like digit recognition, sentiment analysis, click prediction, data warehousing, and distributed systems.
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeAlexander Pico This document provides an overview and agenda for an introductory workshop on network visualization and analysis using Cytoscape. The agenda includes introductions, an overview of Cytoscape concepts and user interface, six tutorials, breaks, and a presentation on pathway analysis. The document discusses loading and visualizing networks and attributes in Cytoscape, different types of biological networks, visualization techniques like layouts and data mapping, and tips for using Cytoscape effectively.
BioThings API: Promoting Best-practices via a Biomedical API Development Ecos...
BioThings API: Promoting Best-practices via a Biomedical API Development Ecos...Chunlei Wu The document outlines the Biothings API framework, which enables modular access to various biological data such as genes, variants, and chemicals through a streamlined API interface. It includes services for annotation and querying data, ensuring high performance, usability, and sustainability. The document also emphasizes developer support, data sources, and functionalities such as batch queries and the ability to create custom data sources.
RELIANCE ROHub hackathon
RELIANCE ROHub hackathonRaul Palma The project focuses on research lifecycle management technologies for Earth science communities, funded by the European Union's Horizon 2020 program. It emphasizes the creation, sharing, and preservation of research objects (ROs) through a holistic management system, enhancing collaboration and reproducibility in scientific research. Key features include integrating external data services, managing RO evolution, and providing semantic enrichment to improve findability and accessibility.
NextGenML
NextGenML Moldovan Radu Adrian The document outlines the NextGenML platform, a container-based hybrid cloud solution designed to simplify and scale machine learning operations through a robust, enterprise-level pipeline integrated into Kubernetes. The platform enables data science teams to manage the entire modeling process, providing tools for performance metrics, real-time progress tracking, and data traceability, while promoting collaboration between data engineers and scientists. Additionally, it features automation for CI/CD, flexible integration with numerous cloud services, and a comprehensive asset catalog for machine learning management.
KFServing, Model Monitoring with Apache Spark and a Feature Store
KFServing, Model Monitoring with Apache Spark and a Feature StoreDatabricks The document discusses the integration of kfserving with a feature store for model monitoring and enhancement in machine learning applications. It emphasizes the importance of enriching input data with context and history to improve prediction accuracy and outlines workflows for model serving and data validation. The implementation details include real-time feature updates, data drift detection, and the use of Kafka and Spark for scalable monitoring solutions.
Sparkr sigmod
Sparkr sigmodwaqasm86 This summary provides an overview of the SparkR package, which provides an R frontend for the Apache Spark distributed computing framework:
- SparkR enables large-scale data analysis from the R shell by using Spark's distributed computation engine to parallelize and optimize R programs. It allows R users to leverage Spark's libraries, data sources, and optimizations while programming in R.
- The central component of SparkR is the distributed DataFrame, which provides a familiar data frame interface to R users but can handle large datasets using Spark. DataFrame operations are optimized using Spark's query optimizer.
- SparkR's architecture includes an R to JVM binding that allows R programs to submit jobs to Spark, and support for running R execut
Weave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any KubernetesWeaveworks Weaveworks specializes in GitOps, providing a developer-centric operating model for cloud-native technologies, and offers both open-source and enterprise solutions for managing Kubernetes applications. The enterprise version includes features like application management UI, fleet management, and profiles for organizing Kubernetes resources securely and efficiently. The document introduces key personnel, outlines the company's history and contributions to open source, and details the capabilities of their GitOps products.
FAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble Le professeur Carole Goble de l'Université de Manchester présente les flux de travail computationnels FAIR, qui facilitent l'analyse et le partage de grandes quantités de données complexes dans les biosciences. Ces systèmes interopérables et modulables permettent la composition et l'exécution de workflows variés pour la gestion des données, favorisant la reproductibilité et l'accessibilité ouverte. L'initiative met en avant l'importance d'un écosystème collaboratif pour garantir des données et des logiciels équitables et durables.
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsCarole Goble The document discusses advances in scientific workflow environments, emphasizing automation and management of computational tasks across various infrastructures. It highlights the importance of provenance, interoperability, and the reuse of workflows in scientific research, with examples from bioinformatics and molecular simulations. The text also mentions specific tools and platforms used for developing and executing workflows, along with the importance of adopting common standards for effective collaboration.
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...Rafael Ferreira da Silva The document details the WorkflowHub community framework, which supports scientific workflow research and development by offering tools for analyzing execution traces and generating realistic synthetic workflows. It emphasizes the increasing complexity of workflows in scientific projects and the necessity for sophisticated workflow management systems. The framework also encourages contributions of workflow traces in a common format and provides open source packages for trace analysis and simulation of large-scale workflows.
Overview of Scientific Workflows - Why Use Them?
Overview of Scientific Workflows - Why Use Them?inside-BigData.com This document provides an overview of scientific workflows, including what they are, common elements, and problems they help address. A workflow is a formal way to express a calculation as a series of tasks with dependencies. Workflow tools automate task execution, data management, scheduling, and more. They can help scale applications from a local system to large clusters. An example is provided of how the CyberShake project uses the Pegasus workflow system to automate probabilistic seismic hazard analysis calculations involving hundreds of thousands of tasks and petabytes of data.
An Overview of VIEW
An Overview of VIEWShiyong Lu The document describes scientific workflows for big data and the challenges they present. It discusses Prof. Shiyong Lu's work on developing the VIEW system for designing, executing, and analyzing scientific workflows. The VIEW system provides a runtime environment for workflows, supports their execution on servers or clouds, and enables efficient storage, querying and visualization of workflow provenance data.
Scientific workflow-overview-2012-01-rev-2
Scientific workflow-overview-2012-01-rev-2Terence Critchlow This document discusses scientific workflows and the author's experience working with them. It begins by describing how the author got involved with scientific workflows through the SciDAC SDM Center. It then discusses how workflows help scientists by automating tasks to allow them to focus on science. The author describes some early challenges with workflows not being portable across machines. Finally, it discusses the need to better integrate workflows into the broader scientific process and allow scientists to more easily collect and share information about experiments.
More Related Content
What's hot (16)
A Query Model for Ad Hoc Queries using a Scanning Architecture
A Query Model for Ad Hoc Queries using a Scanning ArchitectureFlurry, Inc. The document presents the Burst system developed by Flurry, designed to support low-latency, single-pass ad hoc queries over large datasets of mobile application analytics. It focuses on a scan-only architecture, simplifying query execution to allow developers to interactively explore metrics with sub-second response times, contrasting traditional systems that necessitate lengthy waits for results. The Burst system, which is integrated with Flurry's Explorer product, effectively handles complex causal queries and data models for hundreds of thousands of applications, significantly enhancing query performance over legacy systems like Hadoop.
Next.ml Boston: Data Science Dev Ops
Next.ml Boston: Data Science Dev OpsEric Chiang This document discusses challenges in deploying machine learning models into production and potential solutions. It covers:
1. Issues with reproducibility due to dependencies and environment configurations when models are trained and deployed.
2. Problems with serializing models and transferring them between different versions of libraries and software stacks.
3. How containers can help address these issues by encapsulating the full runtime environment and dependencies of a model.
4. Managing both models and Docker containers is still required when using this approach.
A General Purpose Extensible Scanning Query Architecture for Ad Hoc Analytics
A General Purpose Extensible Scanning Query Architecture for Ad Hoc AnalyticsFlurry, Inc. The document presents Burst, an analytic query system designed for low-latency ad hoc analysis over large datasets, utilizing hardware-efficient scanning techniques and a flexible query language. It details the architecture's components, including an incremental ingestion pipeline for data processing, and explains how the system transforms ad hoc declarative queries into efficient scanning plans executed across distributed nodes. Moreover, it highlights the performance optimizations, schema independence, and future enhancements under development for improved efficiency and scalability.
Pegasus-Poster-2016-final-v2
Pegasus-Poster-2016-final-v2Samrat Jha Pegasus is a workflow management system that maps abstract workflows to executable workflows that can run on different execution environments. It consists of three main components - the Pegasus Mapper, HTCondor DAGMan, and HTCondor Schedd. Pegasus provides APIs in multiple languages including R to generate abstract workflows and can easily scale to workflows with millions of tasks. It handles mapping workflows to resources, data movement between sites, monitoring and debugging of workflows. Pegasus was used to run EPA and Bioconductor genomic analysis workflows across multiple sites by partitioning tasks.
SDCSB Advanced Tutorial: Reproducible Data Visualization Workflow with Cytosc...
SDCSB Advanced Tutorial: Reproducible Data Visualization Workflow with Cytosc...Keiichiro Ono This document provides an overview of a tutorial on building reproducible network data visualization workflows using Cytoscape and IPython Notebook. The tutorial will cover integrating data, analyzing networks, visualizing results, and preparing outputs for publication. It will demonstrate setting up a portable data analysis environment using Docker and sharing work through GitHub. The bulk of the tutorial will focus on using IPython Notebook as an electronic lab notebook for interactive and reproducible experiments with Cytoscape.
Converting scripts into reproducible workflow research objects
Converting scripts into reproducible workflow research objectsKhalid Belhajjame 1) The document presents a methodology to convert script-based experiments into reproducible workflow research objects (WROs). This addresses issues of understanding, reusing, and reproducing experiments conducted through scripts.
2) The methodology involves 5 steps: generate an abstract workflow, create an executable workflow, refine the workflow, record provenance data, and annotate and check the quality of the conversion.
3) Applying the methodology to a molecular dynamics simulation case study, the authors demonstrate how scripts can be transformed into WROs containing workflows, annotations, provenance data, and other resources needed for reproducibility.
r1501e
r1501eGeorge Vamos George Vamos has over 30 years of experience as a lead software engineer developing mission critical systems. He has expertise in medical data management systems, aerospace command and control systems, and business databases. Some of his accomplishments include developing medical data warehouses that can adapt to changing data formats, and databases shared by spacecraft tracking systems where much of the schema is auto-generated from requirements. He holds patents related to neural implant management technology.
A Sightseeing Tour of Prov and Some of its Extensions
A Sightseeing Tour of Prov and Some of its ExtensionsKhalid Belhajjame This document provides an overview of the PROV provenance model and some of its extensions. It discusses the motivation for provenance, the history and development of the PROV model, its key concepts of entities, activities, and agents. It also describes extensions like ProvONE and PAV that build upon PROV to model workflow and scientific provenance.
Keyur_Joshi_resume - Copy
Keyur_Joshi_resume - CopyKeyur Joshi Keyur Joshi is seeking a position as a Software Developer/Engineer. He has a Master's degree in Computer Science from SUNY Buffalo and a Bachelor's degree in Electronics and Communications Engineering. He has over 2 years of experience as a Software Developer at Kuberre Systems, where he develops web applications using technologies like Smart GWT, RESTful services, and DynamoDB. His skills include programming languages like C/C++, Java, Python, and databases like MySQL, Oracle, and Hadoop. He has completed projects in areas like digit recognition, sentiment analysis, click prediction, data warehousing, and distributed systems.
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeAlexander Pico This document provides an overview and agenda for an introductory workshop on network visualization and analysis using Cytoscape. The agenda includes introductions, an overview of Cytoscape concepts and user interface, six tutorials, breaks, and a presentation on pathway analysis. The document discusses loading and visualizing networks and attributes in Cytoscape, different types of biological networks, visualization techniques like layouts and data mapping, and tips for using Cytoscape effectively.
BioThings API: Promoting Best-practices via a Biomedical API Development Ecos...
BioThings API: Promoting Best-practices via a Biomedical API Development Ecos...Chunlei Wu The document outlines the Biothings API framework, which enables modular access to various biological data such as genes, variants, and chemicals through a streamlined API interface. It includes services for annotation and querying data, ensuring high performance, usability, and sustainability. The document also emphasizes developer support, data sources, and functionalities such as batch queries and the ability to create custom data sources.
RELIANCE ROHub hackathon
RELIANCE ROHub hackathonRaul Palma The project focuses on research lifecycle management technologies for Earth science communities, funded by the European Union's Horizon 2020 program. It emphasizes the creation, sharing, and preservation of research objects (ROs) through a holistic management system, enhancing collaboration and reproducibility in scientific research. Key features include integrating external data services, managing RO evolution, and providing semantic enrichment to improve findability and accessibility.
NextGenML
NextGenML Moldovan Radu Adrian The document outlines the NextGenML platform, a container-based hybrid cloud solution designed to simplify and scale machine learning operations through a robust, enterprise-level pipeline integrated into Kubernetes. The platform enables data science teams to manage the entire modeling process, providing tools for performance metrics, real-time progress tracking, and data traceability, while promoting collaboration between data engineers and scientists. Additionally, it features automation for CI/CD, flexible integration with numerous cloud services, and a comprehensive asset catalog for machine learning management.
KFServing, Model Monitoring with Apache Spark and a Feature Store
KFServing, Model Monitoring with Apache Spark and a Feature StoreDatabricks The document discusses the integration of kfserving with a feature store for model monitoring and enhancement in machine learning applications. It emphasizes the importance of enriching input data with context and history to improve prediction accuracy and outlines workflows for model serving and data validation. The implementation details include real-time feature updates, data drift detection, and the use of Kafka and Spark for scalable monitoring solutions.
Sparkr sigmod
Sparkr sigmodwaqasm86 This summary provides an overview of the SparkR package, which provides an R frontend for the Apache Spark distributed computing framework:
- SparkR enables large-scale data analysis from the R shell by using Spark's distributed computation engine to parallelize and optimize R programs. It allows R users to leverage Spark's libraries, data sources, and optimizations while programming in R.
- The central component of SparkR is the distributed DataFrame, which provides a familiar data frame interface to R users but can handle large datasets using Spark. DataFrame operations are optimized using Spark's query optimizer.
- SparkR's architecture includes an R to JVM binding that allows R programs to submit jobs to Spark, and support for running R execut
Weave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any KubernetesWeaveworks Weaveworks specializes in GitOps, providing a developer-centric operating model for cloud-native technologies, and offers both open-source and enterprise solutions for managing Kubernetes applications. The enterprise version includes features like application management UI, fleet management, and profiles for organizing Kubernetes resources securely and efficiently. The document introduces key personnel, outlines the company's history and contributions to open source, and details the capabilities of their GitOps products.
Similar to Towards an Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Systems and Applications (20)
FAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble Le professeur Carole Goble de l'Université de Manchester présente les flux de travail computationnels FAIR, qui facilitent l'analyse et le partage de grandes quantités de données complexes dans les biosciences. Ces systèmes interopérables et modulables permettent la composition et l'exécution de workflows variés pour la gestion des données, favorisant la reproductibilité et l'accessibilité ouverte. L'initiative met en avant l'importance d'un écosystème collaboratif pour garantir des données et des logiciels équitables et durables.
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsCarole Goble The document discusses advances in scientific workflow environments, emphasizing automation and management of computational tasks across various infrastructures. It highlights the importance of provenance, interoperability, and the reuse of workflows in scientific research, with examples from bioinformatics and molecular simulations. The text also mentions specific tools and platforms used for developing and executing workflows, along with the importance of adopting common standards for effective collaboration.
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...Rafael Ferreira da Silva The document details the WorkflowHub community framework, which supports scientific workflow research and development by offering tools for analyzing execution traces and generating realistic synthetic workflows. It emphasizes the increasing complexity of workflows in scientific projects and the necessity for sophisticated workflow management systems. The framework also encourages contributions of workflow traces in a common format and provides open source packages for trace analysis and simulation of large-scale workflows.
Overview of Scientific Workflows - Why Use Them?
Overview of Scientific Workflows - Why Use Them?inside-BigData.com This document provides an overview of scientific workflows, including what they are, common elements, and problems they help address. A workflow is a formal way to express a calculation as a series of tasks with dependencies. Workflow tools automate task execution, data management, scheduling, and more. They can help scale applications from a local system to large clusters. An example is provided of how the CyberShake project uses the Pegasus workflow system to automate probabilistic seismic hazard analysis calculations involving hundreds of thousands of tasks and petabytes of data.
An Overview of VIEW
An Overview of VIEWShiyong Lu The document describes scientific workflows for big data and the challenges they present. It discusses Prof. Shiyong Lu's work on developing the VIEW system for designing, executing, and analyzing scientific workflows. The VIEW system provides a runtime environment for workflows, supports their execution on servers or clouds, and enables efficient storage, querying and visualization of workflow provenance data.
Scientific workflow-overview-2012-01-rev-2
Scientific workflow-overview-2012-01-rev-2Terence Critchlow This document discusses scientific workflows and the author's experience working with them. It begins by describing how the author got involved with scientific workflows through the SciDAC SDM Center. It then discusses how workflows help scientists by automating tasks to allow them to focus on science. The author describes some early challenges with workflows not being portable across machines. Finally, it discusses the need to better integrate workflows into the broader scientific process and allow scientists to more easily collect and share information about experiments.
FAIR Computational Workflows
FAIR Computational Workflows Carole Goble The document discusses the significance of fair computational workflows in bioscience, emphasizing their role in processing complex data and promoting transparency and reproducibility. It outlines various aspects such as tool interoperability, data management, and community collaboration, highlighting the need for standardized practices and services to enhance workflow accessibility. The text advocates for a collective responsibility to implement fair principles, ensuring that workflows and associated data are findable, accessible, interoperable, and reusable (FAIR).
EOSC-Life Workflow Collaboratory
EOSC-Life Workflow CollaboratoryCarole Goble The document outlines the EOSC-Life Workflow Collaboratory, which aims to enhance access to and reuse of research data and tools within the European Open Science Cloud. It describes the collaborative environment supporting data-intensive bioscience research through sophisticated workflows, ensuring FAIR (Findable, Accessible, Interoperable, and Reusable) data practices. The initiative promotes interoperability between various workflow management systems and encourages community engagement towards the development and maintenance of workflows.
Creating abstractions from scientific workflows: PhD symposium 2015
Creating abstractions from scientific workflows: PhD symposium 2015dgarijo This document discusses the creation of abstractions in scientific workflows. It hypothesizes that it is possible to automatically extract reusable patterns and abstractions from scientific workflow repositories that could be useful for developers. The document outlines challenges in workflow representation, abstraction, reuse, and annotation. It then describes an approach to define vocabularies and methodologies for publishing workflows as linked data. This includes defining a catalog of common workflow abstractions and techniques for finding and evaluating these abstractions across different workflow corpora. Evaluation shows the extracted patterns are similar to those defined by users and are considered useful.
Towards Workflow Ecosystems Through Semantic and Standard Representations
Towards Workflow Ecosystems Through Semantic and Standard Representationsdgarijo The document discusses the development of workflow ecosystems that integrate diverse workflow tools and functions, enhancing interoperability and flexibility. It outlines the core dimensions of workflow ecosystems, such as functional, usage, and abstraction heterogeneity and provides examples of existing projects and future work goals. Additionally, it highlights the benefits of these ecosystems while acknowledging current limitations and the need for further advancements.
From Scientific Workflows to Research Objects: Publication and Abstraction of...
From Scientific Workflows to Research Objects: Publication and Abstraction of...dgarijo This document discusses the transition from scientific workflows to research objects, focusing on the representation, publication, and abstraction of scientific experiments. It outlines the motivation for this research, which includes improving understandability and reuse of workflows, and presents methodologies for detecting common motifs and fragments within workflows. The concept of research objects is introduced as a means of aggregating various resources related to a research study.
Invited Talk for EUDAT Workshop in Barcelona
Invited Talk for EUDAT Workshop in Barcelona Ilkay Altintas, Ph.D. Scientific workflows help facilitate research by making complex processes easier to assemble, access diverse resources transparently, incorporate multiple tools, and ensure reproducibility. However, new challenges have emerged such as analyzing large amounts of sensor and genomic data. Workflows need to be more programmable, optimize resource usage across computing systems, and integrate with the full scientific process from data generation to publication. Next steps include specializing workflows for different domains and standards, treating workflows as publications, and catering to various hardware architectures.
From Scientific Workflows to Research Objects: Publication and Abstraction of...
From Scientific Workflows to Research Objects: Publication and Abstraction of...dgarijo The document discusses the transition from scientific workflows to research objects, focusing on the representation and publication of scientific experiments. It explores key concepts such as workflow motifs, provenance traces, and the use of linked data to improve interoperability and understandability of scientific workflows. The research emphasizes the need for abstraction definitions and algorithms to detect common fragments among workflows for better reuse and exploration.
Data legend dh_benelux_2017.key
Data legend dh_benelux_2017.keyRichard Zijdeman This document summarizes the results of an empirical analysis of 177 scientific workflows from Taverna and Wings systems. The analysis identified common motifs in data-oriented activities and workflow implementation styles. For data activities, motifs included data preparation, data transformation, data movement and data visualization. For workflows, motifs involved different ways activities were combined and implemented. The identified motifs could help inform workflow design practices and tools to generate workflow abstractions, improving understanding and reusability of workflows.
2016-10-20 BioExcel: Advances in Scientific Workflow Environments
2016-10-20 BioExcel: Advances in Scientific Workflow EnvironmentsStian Soiland-Reyes The document outlines the complexities of scientific workflows, emphasizing the orchestration, automation, and management of computational tasks within various environments. It discusses the need for provenance, reproducibility, and interoperability in workflows, along with examples of existing workflow systems and tools. Additionally, it highlights the future directions for scientific workflows and the potential of containerization and cloud computing in enhancing these processes.
The Interplay of Workflow Execution and Resource Provisioning
The Interplay of Workflow Execution and Resource ProvisioningRafael Ferreira da Silva The document discusses the complexities of workflow execution and resource provisioning in high-performance computing (HPC) and cloud environments. It highlights the importance of workflow optimizations, fault tolerance, and task clustering for efficient data management and execution in scientific applications. Additionally, it covers case studies demonstrating the successful application of the Pegasus workflow management system in scientific research and its impact on various domains.
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, RomeCarole Goble The document discusses the integration and management of scientific workflows, focusing on interoperability, data processing, and reproducibility within the context of diverse research disciplines, particularly in ecology and biology. It outlines the significance of provenance tracking for data and workflows, the importance of systematic assembly and automation of workflows, and presents various resources and platforms available for managing workflows. Additionally, it highlights the evolving nature of workflow management systems and their impact on scientific productivity and collaboration.
Scientific Workflows: what do we have, what do we miss?
Scientific Workflows: what do we have, what do we miss?Paolo Romano This document discusses scientific workflows and outlines some key points:
- Scientific workflows are used to automate data retrieval and analysis processes from multiple databases and tools. Workflow management systems help implement these processes.
- Issues with current workflow systems include lack of automatic composition capabilities, performance limitations especially with large data volumes, and ensuring reproducibility of results over time as databases and tools change.
- The document outlines approaches to address these issues such as using ontologies to support automatic composition, optimizing for performance through parallelization and alternative services, and capturing provenance data to improve reproducibility and reuse of analyses.
Opportunities and Challenges for Running Scientific Workflows on the Cloud
Opportunities and Challenges for Running Scientific Workflows on the Cloud lyingcom This document discusses opportunities and challenges for running scientific workflows on cloud computing platforms. It begins by introducing cloud computing and scientific workflows. It then describes three main opportunities: 1) increased scale to address larger problems, 2) on-demand resource allocation for improved efficiency, and 3) more flexibility to optimize performance and cost. Several challenges are also outlined, including architectural integration issues, data management difficulties, and the need for cloud-compatible workflow languages and services. The document concludes by proposing research directions to address these challenges and better support scientific workflows in cloud environments.
FAIR Computational Workflows
FAIR Computational WorkflowsCarole Goble This document discusses computational workflows and FAIR principles. It begins by providing background on computational workflows and their increasing importance. It then discusses challenges around finding, accessing, and sharing workflows. Next, it explores how applying FAIR principles to workflows could help address these challenges by making workflows and their associated objects findable, accessible, interoperable, and reusable. This includes discussing applying metadata standards, using persistent identifiers, and developing principles for FAIR workflows and FAIR software. The document concludes by examining the roles and responsibilities of different stakeholders in working towards FAIR workflows.
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...
WorkflowHub: Community Framework for Enabling Scientific Workflow Research a...Rafael Ferreira da Silva
Ad
More from Rafael Ferreira da Silva (20)
Modeling and Simulation of Parallel and Distributed Computing Systems with Si...
Modeling and Simulation of Parallel and Distributed Computing Systems with Si...Rafael Ferreira da Silva The document discusses the challenges in experimental research within parallel and distributed computing systems and proposes the use of simulation as a solution. It highlights the capabilities of simulation frameworks like SimGrid, Wrench, and WFCommons in providing accurate and scalable simulations for workflow management and CI systems. The frameworks facilitate a simulation-driven engineering lifecycle, allowing for effective testing and evaluation of complex workflows in scientific research.
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...Rafael Ferreira da Silva The document provides guidelines for best practices in developing scientific software frameworks. It discusses hosting open source projects on version control platforms and ensuring documentation, testing, continuous integration/delivery, and other development practices are followed. Specific examples mentioned include the WRENCH simulation framework, Pegasus workflow system, and scikit-learn machine learning library. The document emphasizes practices like writing tests, tracking issues, reviewing code quality, and releasing versions in a semantic and citable manner.
Bridging Concepts and Practice in eScience via Simulation-driven Engineering
Bridging Concepts and Practice in eScience via Simulation-driven EngineeringRafael Ferreira da Silva The document discusses the disconnect between theoretical and practical applications in eScience, highlighting the limitations of real-world experiments and the need for simulation as a solution. It introduces the WRENCH simulation framework, which simplifies the development of accurate and scalable simulators for complex computational infrastructures by building on the SimGrid platform. The framework aims to enhance research output by enabling researchers to effectively simulate and evaluate computational workflows and systems.
Accurately Simulating Energy Consumption of I/O-intensive Scientific Workflows
Accurately Simulating Energy Consumption of I/O-intensive Scientific WorkflowsRafael Ferreira da Silva The document discusses techniques for accurately simulating energy consumption in I/O-intensive scientific workflows, emphasizing the need to consider both CPU utilization and I/O operations. It uses the Pegasus Workflow Management System and Grid'5000 testbed to analyze power consumption correlations, revealing that I/O operations significantly impact energy efficiency. The authors propose a new model for energy consumption estimation that improves upon traditional methods, suggesting future work to validate this model across various workflows and platforms.
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...Rafael Ferreira da Silva The WRENCH project aims to provide a simulation workbench for scientific workflows, addressing the challenges of costly and time-consuming real-world experiments by offering accurate, scalable, and reproducible simulations. By building upon the SimGrid framework, WRENCH exposes high-level abstractions for developing custom simulators, facilitating research and development in complex systems. The project is open-source and supports a wide range of users, from developers to researchers, enabling innovative avenues in large-scale scientific computations and data analyses.
WRENCH: Workflow Management System Simulation Workbench
WRENCH: Workflow Management System Simulation WorkbenchRafael Ferreira da Silva The document presents the WRENCH project, which aims to facilitate the simulation of scientific workflows through a scalable and accurate simulation workbench. Funded by the NSF and CNRS, WRENCH provides a methodology for developing and analyzing workflows that benefits researchers, educators, and developers. Utilizing the SimGrid framework, WRENCH allows for high-level abstractions to be used in simulations, enhancing the study and optimization of complex workflow executions.
On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows
On the Use of Burst Buffers for Accelerating Data-Intensive Scientific WorkflowsRafael Ferreira da Silva The document discusses using burst buffers to accelerate I/O performance for data-intensive scientific workflows. It finds that burst buffers improved write performance by 9x and read performance by 15x for a cybersecurity workflow. However, performance decreased slightly with more than 64 nodes due to potential I/O bottlenecks. While burst buffers helped, other approaches like in-situ processing may also be needed to meet all application requirements. Future work includes investigating combined in-situ and in-transit analysis and developing a production workflow management system with burst buffer support.
Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific Wor...
Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific Wor...Rafael Ferreira da Silva This document explores the use of simple PID controllers to prevent and mitigate faults in scientific workflows, emphasizing their application in enhancing reliability and execution performance in distributed computing environments. The authors detail experimental evaluations that showcase how these controllers effectively detect and address potential failures, ultimately improving workflow completion times. Recommendations for future research include refining controller metrics and exploring the impact of multiple control loops on system consistency.
Automating Environmental Computing Applications with Scientific Workflows
Automating Environmental Computing Applications with Scientific WorkflowsRafael Ferreira da Silva The document discusses the use of scientific workflows to automate environmental computing applications, emphasizing their benefits such as enhanced reproducibility, automation of data transfers, and job management. It covers various aspects, including workflow composition, execution, and management systems, along with specific use cases like the Cybershake and ACME workflows. Future research directions suggest broader adoption among environmental scientists to increase efficiency in computations across diverse resources.
Analysis of User Submission Behavior on HPC and HTC
Analysis of User Submission Behavior on HPC and HTCRafael Ferreira da Silva The document analyzes user submission behavior on high-performance computing (HPC) and high-throughput computing (HTC) systems, detailing aspects such as workload characterization and think times. It discusses data-driven analysis of job submission interarrival times, including the impact of automated and batch submissions on user behavior. Conclusions indicate that user reaction dynamics and think times have remained stable over the past 20 years, with recommendations for future research to further investigate think time definitions and user responses.
Automating Real-time Seismic Analysis Through Streaming and High Throughput W...
Automating Real-time Seismic Analysis Through Streaming and High Throughput W...Rafael Ferreira da Silva The document discusses the automation of real-time seismic analysis using streaming and high-throughput workflows, emphasizing the importance of reliable risk assessment methods for seismic hazards. It outlines the steps and technologies involved in automating complex workflows, including the use of various computational models and tools like Pegasus for distributed computing. Additionally, it describes specific phases of seismic analysis, such as data preparation and cross-correlation, along with performance optimization strategies for large-scale workflows.
Performance Analysis of an I/O-Intensive Workflow executing on Google Cloud a...
Performance Analysis of an I/O-Intensive Workflow executing on Google Cloud a...Rafael Ferreira da Silva The document presents a performance analysis of an I/O-intensive scientific workflow executed on Google Cloud and Amazon Web Services, focusing on the montage application that processes large data volumes. It evaluates different storage configurations, revealing that VM storage outperforms cloud storage and submit host configurations in terms of makespan and efficiency. The findings emphasize the importance of optimizing data transfer mechanisms to enhance the execution of workflows in cloud environments.
Pegasus - automate, recover, and debug scientific computations
Pegasus - automate, recover, and debug scientific computationsRafael Ferreira da Silva Pegasus is a workflow management system that automates complex computational experiments. It handles tasks such as automating multi-step processing pipelines, enabling parallel distributed computations, automatically transferring data, and handling failures to provide reliability. Pegasus records provenance data to keep track of how results were produced and allows experimental workflows to be reproduced. It generates executable workflows from abstract descriptions to run experiments on distributed computing resources including clusters, grids, and clouds.
Task Resource Consumption Prediction for Scientific Applications and Workflows
Task Resource Consumption Prediction for Scientific Applications and WorkflowsRafael Ferreira da Silva The document discusses task resource consumption prediction for scientific workflows, focusing on scheduling and resource provisioning algorithms. It presents workload characterization from the Compact Muon Solenoid experiment, detailing job statistics and performance metrics, while introducing methods to predict resource requirements using regression trees. Additionally, it outlines an online estimation process for fine-grained task needs, highlighting accuracy improvements in runtime and resource consumption predictions based on historical data.
Characterizing a High Throughput Computing Workload: The Compact Muon Solenoi...
Characterizing a High Throughput Computing Workload: The Compact Muon Solenoi...Rafael Ferreira da Silva The document discusses the workload characterization of the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider, analyzing data from over 1.4 million jobs and 392 users. It details resource allocation and execution monitoring, as well as the use of statistical recursive partitioning to enhance the accuracy of workload estimations for job runtime, disk, and memory usage. The findings indicate that adequate and nearly optimal estimates can be achieved, particularly for job characteristics, with suggestions for improved data collection and application mechanisms.
Experiments with Complex Scientific Applications on Hybrid Cloud Infrastructures
Experiments with Complex Scientific Applications on Hybrid Cloud InfrastructuresRafael Ferreira da Silva The document discusses research challenges and solutions for executing complex scientific applications on hybrid cloud infrastructures. It highlights the development of workflow management systems like Pegasus and Hyperflow, cost optimization algorithms, and performance evaluation techniques for cloud services. The document further details various experiments aimed at improving the scalability and efficiency of scientific workflows across diverse cloud environments.
A Unified Approach for Modeling and Optimization of Energy, Makespan and Reli...
A Unified Approach for Modeling and Optimization of Energy, Makespan and Reli...Rafael Ferreira da Silva The document discusses a unified approach for optimizing energy consumption, makespan, and reliability in scientific workflows on large-scale computing infrastructures. It highlights the development of an energy consumption model that accounts for real-world infrastructure conditions and aims for multi-objective optimization while addressing the limitations of current research focused on single or dual objectives. The work emphasizes the importance of a fully instrumented platform for accurate monitoring and profiling to enhance workflow efficiency.
Leveraging Semantics to Improve Reproducibility in Scientific Workflows
Leveraging Semantics to Improve Reproducibility in Scientific WorkflowsRafael Ferreira da Silva The document discusses improving reproducibility in scientific workflows through semantic modeling and representation of computational environments. It outlines two conservation approaches: physical and logical, emphasizing the use of semantic annotations to describe resources and workflows. The proposed framework includes a reproducibility process involving setup, workflow management, and infrastructure specification to enhance the reliability of scientific computations.
A science-gateway for workflow executions: online and non-clairvoyant self-h...
A science-gateway for workflow executions: online and non-clairvoyant self-h...Rafael Ferreira da Silva The document discusses a science gateway for workflow executions that focuses on the online, non-clairvoyant self-healing of workflows in grid computing environments. It addresses challenges such as blocked activities, task granularity optimization, and fairness control among workflow executions. Additionally, it highlights various technical contributions and the effectiveness of these self-healing mechanisms in enhancing workflow completion rates and overall performance.
Toward Fine-Grained Online Task Characteristics Estimation in Scientific Work...
Toward Fine-Grained Online Task Characteristics Estimation in Scientific Work...Rafael Ferreira da Silva The document presents a method for online estimation of fine-grained task characteristics in scientific workflows, addressing challenges in accurate planning and execution. It details experiments conducted using various workflows, highlighting the effectiveness of online estimation compared to traditional offline methods in reducing estimation errors. Future work aims to further analyze the impact of re-planning and improve correlation sensitivity in workflow estimations.
Modeling and Simulation of Parallel and Distributed Computing Systems with Si...
Modeling and Simulation of Parallel and Distributed Computing Systems with Si...Rafael Ferreira da Silva
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...
Good Practices for Developing Scientific Software Frameworks: The WRENCH fram...Rafael Ferreira da Silva
Bridging Concepts and Practice in eScience via Simulation-driven Engineering
Bridging Concepts and Practice in eScience via Simulation-driven EngineeringRafael Ferreira da Silva
Accurately Simulating Energy Consumption of I/O-intensive Scientific Workflows
Accurately Simulating Energy Consumption of I/O-intensive Scientific WorkflowsRafael Ferreira da Silva
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...Rafael Ferreira da Silva
On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows
On the Use of Burst Buffers for Accelerating Data-Intensive Scientific WorkflowsRafael Ferreira da Silva
Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific Wor...
Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific Wor...Rafael Ferreira da Silva
Automating Real-time Seismic Analysis Through Streaming and High Throughput W...
Automating Real-time Seismic Analysis Through Streaming and High Throughput W...Rafael Ferreira da Silva
Performance Analysis of an I/O-Intensive Workflow executing on Google Cloud a...
Performance Analysis of an I/O-Intensive Workflow executing on Google Cloud a...Rafael Ferreira da Silva
Task Resource Consumption Prediction for Scientific Applications and Workflows
Task Resource Consumption Prediction for Scientific Applications and WorkflowsRafael Ferreira da Silva
Characterizing a High Throughput Computing Workload: The Compact Muon Solenoi...
Characterizing a High Throughput Computing Workload: The Compact Muon Solenoi...Rafael Ferreira da Silva
Experiments with Complex Scientific Applications on Hybrid Cloud Infrastructures
Experiments with Complex Scientific Applications on Hybrid Cloud InfrastructuresRafael Ferreira da Silva
A Unified Approach for Modeling and Optimization of Energy, Makespan and Reli...
A Unified Approach for Modeling and Optimization of Energy, Makespan and Reli...Rafael Ferreira da Silva
A science-gateway for workflow executions: online and non-clairvoyant self-h...
A science-gateway for workflow executions: online and non-clairvoyant self-h...Rafael Ferreira da Silva
Toward Fine-Grained Online Task Characteristics Estimation in Scientific Work...
Toward Fine-Grained Online Task Characteristics Estimation in Scientific Work...Rafael Ferreira da Silva
Ad
Recently uploaded (20)
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation Can We Use Rust to Develop Extensions for PostgreSQL?
(POSETTE: An Event for Postgres 2025)
June 11, 2025
Shinya Kato
NTT DATA Japan Corporation
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/from-enterprise-to-makers-driving-vision-ai-innovation-at-the-extreme-edge-a-presentation-from-sony-semiconductor-solutions/
Amir Servi, Edge Deep Learning Product Manager at Sony Semiconductor Solutions, presents the “From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge” tutorial at the May 2025 Embedded Vision Summit.
Sony’s unique integrated sensor-processor technology is enabling ultra-efficient intelligence directly at the image source, transforming vision AI for enterprises and developers alike. In this presentation, Servi showcases how the AITRIOS platform simplifies vision AI for enterprises with tools for large-scale deployments and model management.
Servi also highlights his company’s collaboration with Ultralytics and Raspberry Pi, which brings YOLO models to the developer community, empowering grassroots innovation. Whether you’re scaling vision AI for industry or experimenting with cutting-edge tools, this presentation will demonstrate how Sony is accelerating high-performance, energy-efficient vision AI for all.
June Patch Tuesday
June Patch TuesdayIvanti Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...
AudGram Review: Build Visually Appealing, AI-Enhanced Audiograms to Engage Yo...SOFTTECHHUB AudGram changes everything by bridging the gap between your audio content and the visual engagement your audience craves. This cloud-based platform transforms your existing audio into scroll-stopping visual content that performs across all social media platforms.
Artificial Intelligence in the Nonprofit Boardroom.pdf
Artificial Intelligence in the Nonprofit Boardroom.pdfOnBoard OnBoard recently partnered with Microsoft Tech for Social Impact on the AI in the Nonprofit Boardroom Survey, an initiative designed to uncover the current and future role of artificial intelligence in nonprofit governance.
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
Enabling BIM / GIS integrations with Other Systems with FME
Enabling BIM / GIS integrations with Other Systems with FMESafe Software Jacobs has successfully utilized FME to tackle the complexities of integrating diverse data sources in a confidential $1 billion campus improvement project. The project aimed to create a comprehensive digital twin by merging Building Information Modeling (BIM) data, Construction Operations Building Information Exchange (COBie) data, and various other data sources into a unified Geographic Information System (GIS) platform. The challenge lay in the disparate nature of these data sources, which were siloed and incompatible with each other, hindering efficient data management and decision-making processes.
To address this, Jacobs leveraged FME to automate the extraction, transformation, and loading (ETL) of data between ArcGIS Indoors and IBM Maximo. This process ensured accurate transfer of maintainable asset and work order data, creating a comprehensive 2D and 3D representation of the campus for Facility Management. FME's server capabilities enabled real-time updates and synchronization between ArcGIS Indoors and Maximo, facilitating automatic updates of asset information and work orders. Additionally, Survey123 forms allowed field personnel to capture and submit data directly from their mobile devices, triggering FME workflows via webhooks for real-time data updates. This seamless integration has significantly enhanced data management, improved decision-making processes, and ensured data consistency across the project lifecycle.
Edge-banding-machines-edgeteq-s-200-en-.pdf
Edge-banding-machines-edgeteq-s-200-en-.pdfAmirStern2 מכונת קנטים המתאימה לנגריות קטנות או גדולות (כמכונת גיבוי).
מדביקה קנטים מגליל או פסים, עד עובי קנט – 3 מ"מ ועובי חומר עד 40 מ"מ. בקר ממוחשב המתריע על תקלות, ומנועים מאסיביים תעשייתיים כמו במכונות הגדולות.
Creating Inclusive Digital Learning with AI: A Smarter, Fairer Future
Creating Inclusive Digital Learning with AI: A Smarter, Fairer FutureImpelsys Inc. Have you ever struggled to read a tiny label on a medicine box or tried to navigate a confusing website? Now imagine if every learning experience felt that way—every single day.
For millions of people living with disabilities, poorly designed content isn’t just frustrating. It’s a barrier to growth. Inclusive learning is about fixing that. And today, AI is helping us build digital learning that’s smarter, kinder, and accessible to everyone.
Accessible learning increases engagement, retention, performance, and inclusivity for everyone. Inclusive design is simply better design.
TrustArc Webinar - 2025 Global Privacy Survey
TrustArc Webinar - 2025 Global Privacy SurveyTrustArc How does your privacy program compare to your peers? What challenges are privacy teams tackling and prioritizing in 2025?
In the sixth annual Global Privacy Benchmarks Survey, we asked global privacy professionals and business executives to share their perspectives on privacy inside and outside their organizations. The annual report provides a 360-degree view of various industries' priorities, attitudes, and trends. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar features an expert panel discussion and data-driven insights to help you navigate the shifting privacy landscape. Whether you are a privacy officer, legal professional, compliance specialist, or security expert, this session will provide actionable takeaways to strengthen your privacy strategy.
This webinar will review:
- The emerging trends in data protection, compliance, and risk
- The top challenges for privacy leaders, practitioners, and organizations in 2025
- The impact of evolving regulations and the crossroads with new technology, like AI
Predictions for the future of privacy in 2025 and beyond
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/addressing-evolving-ai-model-challenges-through-memory-and-storage-a-presentation-from-micron/
Wil Florentino, Senior Segment Marketing Manager at Micron, presents the “Addressing Evolving AI Model Challenges Through Memory and Storage” tutorial at the May 2025 Embedded Vision Summit.
In the fast-changing world of artificial intelligence, the industry is deploying more AI compute at the edge. But the growing diversity and data footprint of transformers and models such as large language models and large multimodal models puts a spotlight on memory performance and data storage capacity as key bottlenecks. Enabling the full potential of AI in industries such as manufacturing, automotive, robotics and transportation will require us to find efficient ways to deploy this new generation of complex models.
In this presentation, Florentino explores how memory and storage are responding to this need and solving complex issues in the AI market. He examines the storage capacity and memory bandwidth requirements of edge AI use cases ranging from tiny devices with severe cost and power constraints to edge servers, and he explains how new memory technologies such as LPDDR5, LPCAMM2 and multi-port SSDs are helping system developers to meet these challenges.
Providing an OGC API Processes REST Interface for FME Flow
Providing an OGC API Processes REST Interface for FME FlowSafe Software This presentation will showcase an adapter for FME Flow that provides REST endpoints for FME Workspaces following the OGC API Processes specification. The implementation delivers robust, user-friendly API endpoints, including standardized methods for parameter provision. Additionally, it enhances security and user management by supporting OAuth2 authentication. Join us to discover how these advancements can elevate your enterprise integration workflows and ensure seamless, secure interactions with FME Flow.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
Data Validation and System Interoperability
Data Validation and System InteroperabilitySafe Software A non-profit human services agency with specialized health record and billing systems. Challenges solved include access control integrations from employee electronic HR records, multiple regulations compliance, data migrations, benefits enrollments, payroll processing, and automated reporting for business intelligence and analysis.
Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Safe Software Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration
No-Code Workflows for CAD & 3D Data: Scaling AI-Driven Infrastructure
No-Code Workflows for CAD & 3D Data: Scaling AI-Driven InfrastructureSafe Software When projects depend on fast, reliable spatial data, every minute counts.
AI Clearing needed a faster way to handle complex spatial data from drone surveys, CAD designs and 3D project models across construction sites. With FME Form, they built no-code workflows to clean, convert, integrate, and validate dozens of data formats – cutting analysis time from 5 hours to just 30 minutes.
Join us, our partner Globema, and customer AI Clearing to see how they:
-Automate processing of 2D, 3D, drone, spatial, and non-spatial data
-Analyze construction progress 10x faster and with fewer errors
-Handle diverse formats like DWG, KML, SHP, and PDF with ease
-Scale their workflows for international projects in solar, roads, and pipelines
If you work with complex data, join us to learn how to optimize your own processes and transform your results with FME.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
MuleSoft for AgentForce : Topic Center and API Catalog
MuleSoft for AgentForce : Topic Center and API Catalogshyamraj55 This presentation dives into how MuleSoft empowers AgentForce with organized API discovery and streamlined integration using Topic Center and the API Catalog. Learn how these tools help structure APIs around business needs, improve reusability, and simplify collaboration across teams. Ideal for developers, architects, and business stakeholders looking to build a connected and scalable API ecosystem within AgentForce.
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance
Towards an Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Systems and Applications
- 1. ORNL is managed by UT-Battelle LLC for the US Department of Energy Towards an Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Systems and Applications Rafael Ferreira da Silva, Ph.D. Senior Research Scientist Data Lifecycle and Scalable Workflows https://rafaelsilva.com – [email protected] This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the U.S. Department of Energy (DOE). The U.S. government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for U.S. government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).
- 2. 2 2 Scientific Workflows Workflows are becoming more complex and require more sophisticated workflow management capabilities Workflows can now analyze terabyte- scale datasets, be composed of millions of individual tasks that execute for milliseconds up to several hours on distributed heterogenous platforms Catering to these workflow features and demands requires WMS research and development at several levels, from algorithms and systems to user interfaces Thomas McCauley ©2018 CERN ©2019 NASA
- 3. 3 3 Overview of Scientific Workflows Research Challenges DataSpaces Parallelism Interoperability Cloud Sequence Language HPC HTC Structure Composition Design Architecture Provenance Monitoring NVRAM In Situ Burst Buffers HDFS In Transit Execution Clustering Pilot Jobs I/O Energy CO2 Budget Network Transfers Optimization Exception Anomaly Failure Scheduling / Resource Provisioning Modeling and Simulation Community Building Education and Training Workflow Management Systems Domain Science Applications
- 4. 4 4 There is a myriad of workflow systems… The workflow systems landscape is segmented and presents significant barriers to entry due to the hundreds of seemingly comparable, yet incompatible, systems that exist https://s.apache.org/existing-workflow-systems https://github.com/pditommaso/awesome-pipeline
- 5. 5 5 Characterization of Workflow Systems for Extreme-Scale Applications https://doi.org/10.1016/j.future.2017.02.026
- 6. 6 6 Framework for Enabling Workflow Research and Development https://wfcommons.org WfCommons is a framework that provides a collection of tools for analyzing workflow execution instances, producing realistic synthetic workflow instances, and simulating workflow executions Open source Python package to analyze instances, generating workflow recipes, and generating synthetic, yet realistic, workflow instances
- 7. 7 7 Modeling and Simulation https://wrench-project.org Cyberinfrastructure Simulation Workbench https://simgrid.org Accurate and scalable simulation models of hardware/software stacks Objective #1: Make it easy to develop simulators of complex CI application executions Done by providing high-level, reusable simulation abstractions Objective #2: Produce accurate and scalable simulations Done by building on SimGrid
- 8. 8 8 Workflows Community Summits ExaWorks Identify challenges and actionable directions for short- and long-term community efforts Jan 2021: Bringing the Scientific Workflows Community Together April 2021: Advancing the State-of-the-art of Scientific Workflows Management Systems Research and Development board of experts http://workflowsri.org https://exaworks.org
- 9. 9 9 Workflows Community Summits (Jan 2021) • Themes – FAIR computational workflows – Training and education for workflow users – AI workflows – Exascale challenges and beyond – APIs, interoperability, reuse, and standards – Building a workflows community https://doi.org/10.5281/zenodo.4606958 ExaWorks http://workflowsri.org https://exaworks.org
- 10. 10 10 Workflows Community Summits (April 2021) • Topics – Definition of common workflow patterns and benchmarks – Identifying paths toward interoperability of workflow systems – Improving workflow systems' interface with legacy and emerging HPC software and hardware stacks https://doi.org/10.5281/zenodo.4915801 ExaWorks http://workflowsri.org https://exaworks.org
- 11. 11 11 FAIR Computational Workflows The FAIR principles have laid a foundation for sharing and publishing digital assets and in particular data Scientific workflows should support the creation of FAIR data and themselves adhere to the FAIR principles Challenges Define FAIR principles for computational workflows considering complex lifecycle from specification to execution and data products Define metrics to measure the FAIRness of a workflow Potential Efforts Outline rules for FAIR workflows Define recommendations for FAIR workflow repositories Automate FAIRness in workflows
- 12. 12 12 AI/ML Workflows Scientific workflows empowered with ML techniques largely differ from traditional scientific workflows running on HPC machines Challenges Lack of support for fine-grained data management features (file level) and versioning features Lack of capabilities for enabling workflow steering and dynamic workflow execution Integration of ML frameworks into the current HPC landscape Potential Efforts Develop use cases for sample problems with representative workflow structures and data types Develop AI workflows that can benchmark HPC systems
- 13. 13 13 Heterogeneous Computing Challenges Lack of support for heterogeneity of compute resources (CPUs, GPUs, FPGAs, RDMA, etc.) Unfavorable design of resource descriptions and mechanisms for workflow users/systems Potential Efforts Specify community benchmarks for heterogeneous resources Design common interfaces for enabling interoperability among resources Can we allow users to describe their computational steps independently of the compute engine and/or hardware used for running it? Science applications tend to have a longer lifetime than individual workflows technologies This is partly due to the evolution of hardware capabilities that force changes in software architecture and application organization
- 14. 14 14 Co-Design of Workflow Systems and Distributed Computing Infrastructures Challenges Theoreticians produce results that are never used by practitioners, and conversely practitioners use approaches that may be vastly sub-optimal because they are not informed by any theory Potential Efforts Develop a co-design methodology for bridging theoretical results and practical implementations Simulation-driven development of simulated platforms to quantify potential improvements yielded by theoretical results The disconnect between theoretical and practical works is an impediment to the advancement of distributed computing
- 15. Towards an Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Systems and Applications Rafael Ferreira da Silva, Ph.D. Senior Research Scientist Data Lifecycle and Scalable Workflows https://rafaelsilva.com – [email protected] Thank you! Questions?