Bioinformatics Data Pipelines built by CSIRO on AWS
Download as pptx, pdf5 likes727 views
The document discusses cancer genomics data pipelines and CSIRO's solutions. CSIRO has developed variant-spark, an open-source Apache Spark library for scalable genomic analysis. Variant-spark allows analysis of large genomic datasets up to 80% faster than other tools. CSIRO recommends using cloud data pipelines with serverless architectures, Apache Spark on AWS, and SaaS tools like Databricks for scalable, fast cancer genomics analysis. Their solutions provide reusable patterns for ingesting, processing, analyzing and visualizing genomic data in the cloud.
1 of 37
Downloaded 18 times
Ad
Recommended
Genome-scale Big Data Pipelines
Genome-scale Big Data PipelinesLynn Langit This document discusses genomic-scale data pipelines. It introduces Dr. Denis Bauer and his transformational bioinformatics team. It describes how genomic data and research will grow exponentially to exabytes by 2025. It outlines genomic research workflows and challenges like processing, analyzing, and visualizing large variant call format (VCF) data. It presents two cloud data pipeline patterns used by the team: 1) A Spark server cluster pipeline for machine learning on large genomic datasets. 2) A serverless pipeline using AWS Lambda and Step Functions for scalable genomic searches.
VariantSpark - a Spark library for genomics
VariantSpark - a Spark library for genomicsLynn Langit Variantspark is a bioinformatics library designed for the analysis of genomic data, utilizing unsupervised and supervised machine learning methods such as k-means and wide random forests. It enables efficient processing of large genomic datasets, with capabilities to handle millions of variants and samples, and integrates with cloud resources for enhanced performance. The library aims to provide faster and more accurate analyses in transformational bioinformatics research.
Genomic Scale Big Data Pipelines
Genomic Scale Big Data PipelinesLynn Langit The document discusses the advancements in genomic-scale data pipelines and their implications for bioinformatics, focusing on cloud-based solutions like Apache Hadoop and Apache Spark for efficient data analysis and machine learning. It presents methodologies for genetic research, including identifying disease-related variants and genome editing technologies. Additionally, the document highlights the need for scalable and parallelized processes to handle increasing genomic data demands in research.
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationIan Foster The document discusses the concept of the Discovery Cloud, emphasizing the need for automation and outsourcing to enhance scientific discovery through improved data management and computational capabilities. It explores tools like computational grids and cloud computing, exemplifying their use in various scientific fields to facilitate data sharing, analysis, and workflow automation. Additionally, the paper highlights initiatives such as the Globus platform, which aids in managing and transferring large datasets while promoting collaboration among researchers.
Reusable Software and Open Data To Optimize Agriculture
Reusable Software and Open Data To Optimize AgricultureDavid LeBauer The document discusses the development of reusable software and open data initiatives to optimize agriculture, highlighting projects like Pecan and BETYdb that focus on modularity, data harmonization, and reproducibility in scientific workflows. It emphasizes the integration of modern technology, such as remote sensing and high-performance computing, to improve agricultural practices and ecosystem modeling. The document also outlines collaborative efforts among various experts and institutions to enhance data accessibility and analysis for better agricultural outcomes.
Accelerating Time to Science: Transforming Research in the Cloud
Accelerating Time to Science: Transforming Research in the CloudJamie Kinney The document outlines AWS's commitment to advancing scientific computing by providing tools and resources to accelerate research and innovation across various scientific disciplines. It discusses the services offered by AWS, such as high-performance computing and collaborative environments, and highlights notable case studies that demonstrate the impact of AWS in scientific research, including genomics and data-driven analytics. AWS also supports researchers through free usage credits and public data sets to enhance collaborative research efforts globally.
Big data at experimental facilities
Big data at experimental facilitiesIan Foster 1) Scientists at the Advanced Photon Source use the Argonne Leadership Computing Facility for data reconstruction and analysis from experimental facilities in real-time or near real-time. This provides feedback during experiments.
2) Using the Swift parallel scripting language and ALCF supercomputers like Mira, scientists can process terabytes of data from experiments in minutes rather than hours or days. This enables errors to be detected and addressed during experiments.
3) Key applications discussed include near-field high-energy X-ray diffraction microscopy, X-ray nano/microtomography, and determining crystal structures from diffuse scattering images through simulation and optimization. The workflows developed provide significant time savings and improved experimental outcomes.
Accelerating Science with Cloud Technologies in the ABoVE Science Cloud
Accelerating Science with Cloud Technologies in the ABoVE Science CloudGlobus This document summarizes the use of the ABoVE Science Cloud (ASC) to support research for the Arctic-Boreal Vulnerability Experiment (ABoVE). The ASC provides researchers with large datasets, computing resources, and tools to process and analyze remote sensing and model data related to Alaska and northern Canada. Several examples are given of projects using the ASC, including analyzing satellite imagery to map forest structure, tracking surface water changes over time, characterizing fire history, and modeling future forest composition under climate change. The ASC aims to facilitate collaboration by allowing scientists to access common datasets and run computationally-intensive processes in the cloud without having to directly transfer large amounts of data.
Big Data, Big Computing, AI, and Environmental Science
Big Data, Big Computing, AI, and Environmental ScienceIan Foster The document discusses three transformative technologies: exascale computing, big data, and artificial intelligence, which are significantly enhancing scientific research capabilities. It outlines the advancements in AI for science, particularly using deep learning for applications like automated cloud classification and climate modeling, while emphasizing the integration of large-scale data from satellites. Additionally, it describes the processing and classification of extensive MODIS satellite imagery to develop insights into cloud behavior and environmental phenomena.
Data Automation at Light Sources
Data Automation at Light SourcesIan Foster The document discusses data automation at light sources, highlighting the challenges posed by immense data rates and the limitations of current methods. It emphasizes the need for new experimental designs, data management and analysis techniques, proposing automation and outsourcing strategies to handle data effectively. Examples include high-speed storage services, cloud-based automation, and the Data and Learning Hub for managing machine learning models and data publishing.
Machine Learning in Healthcare Diagnostics
Machine Learning in Healthcare DiagnosticsLarry Smarr Machine learning and artificial intelligence are rapidly transforming healthcare and medicine. Advances in genetic sequencing have enabled the mapping of human and microbial genomes at low costs. Researchers are using machine learning to analyze genomic and microbiome data to better understand health and disease. Non-von Neumann brain-inspired computing architectures are being developed for machine learning applications and could accelerate medical research and diagnostics. These technologies may help create personalized health coaching and move medicine from reactive sickcare to proactive healthcare.
Foster CRA March 2022.pptx
Foster CRA March 2022.pptxIan Foster The document discusses the concept of a national discovery cloud designed to enhance human intellect through advanced computing and data integration. It highlights the importance of creating new tools, platforms, and educational initiatives to facilitate research and innovation. Key challenges include ensuring privacy, balancing resource needs, and integrating diverse capabilities into a cohesive framework for the R&D community.
Coding the Continuum
Coding the ContinuumIan Foster The document discusses the evolution of computing and networking, emphasizing the concept of the 'computing continuum' where computing resources are distributed across various types of appliances and locations. It highlights technological advancements, such as 5G and cloud computing, and presents models for optimizing performance based on speed, location, and resource utilization. The narrative ties together concepts of latency, computational efficiency, and cost-management in modern computing environments.
2014 moore-ddd
2014 moore-dddc.titus.brown This document discusses enabling data-intensive biology through superior software and algorithms. It proposes a distributed graph database server that would allow querying across multiple public and private data sets. This would help address the growing data challenge in biology by providing a way to explore, query and mine large datasets in an open and collaborative manner. The goal is to incentivize data sharing and enable new types of data-driven investigations.
Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014
Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014Microsoft Azure for Research Microsoft Research (MSR) was established from a 1990 memo, led by Rick Rashid and later Peter Lee, to advance research, integrate findings into products, and impact the computing field. It encompasses over 1,100 researchers across global locations, significantly contributing to Microsoft's intellectual property and technology development, exemplified by breakthroughs in areas such as speech recognition and cloud computing. The Azure for Research program provides free access to cloud resources for collaborative and data-intensive scientific projects, showcasing MSR's commitment to accelerating scientific discovery.
High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...
High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...Larry Smarr The document discusses advancements in high-performance cyberinfrastructure at UC San Diego, focusing on enabling data-driven science for stem cell research and other fields. It highlights the deployment of a 10 Gbps network and various computational resources, such as the Triton data analysis facility and shared storage solutions, to address the challenges of massive data processing. Additionally, it outlines collaborative efforts and ongoing projects that aim to enhance scientific research capabilities through optimized data storage and analysis techniques.
NERSC, AI and the Superfacility, Debbie Bard
NERSC, AI and the Superfacility, Debbie BardPacificResearchPlatform The document outlines a workshop focused on advanced AI workflows in astronomy and microscopy at NERSC, highlighting its role as a production HPC and data facility supporting various scientific disciplines. It discusses the integration of simulation and data analysis, evolving scientific workflows, and the development of superfacilities to enhance data processing and analysis efficiency. Examples include a supernova detection pipeline and the implementation of high-throughput systems for real-time data management in large-scale astrophysics projects.
Big data ecosystem
Big data ecosystemSlideCentral This document provides a summary of the BigData ecosystem. It lists various distributed filesystems, NoSQL databases, data models, distributed programming frameworks, data ingestion tools, scheduling tools, system development tools, service programming tools, and machine learning tools that are part of the BigData ecosystem. It also defines the size of bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, and exabytes. Some related links on open data, NoSQL databases, traditional databases vs NoSQL, and the role of SQL in big data are also included.
Foss4G 2009 Scenz Grid
Foss4G 2009 Scenz Gridnoho SCENZ-Grid proposes a science collaboration and computation environment to enable researchers to do science research online together by sharing data and computational resources without duplicating them. It would allow researchers to collaboratively develop and use shared models and workflows, and connect researchers directly to policymakers, managers, educators and the public. The environment would use spatial, computational and web services and technologies like Web Processing Services, 52North, GeoServer and OpenLayers to query and analyze data from multiple sources on distributed computational resources. It aims to establish relationships between different datasets from various organizations through similarity matrices and expertise to combine attributes from different classifications.
Cloud Accelerated Genomics
Cloud Accelerated GenomicsIdan Tohami The document discusses the integration of cloud computing with genomics, emphasizing the need to accelerate the translation of genomic research into practical applications by addressing bottlenecks in data processing. It highlights Google's Cloud Platform and its various tools, such as BigQuery and Dataflow, which facilitate the analysis of large genomic datasets, and showcases example use cases. Overall, the text illustrates the evolution of genomic analysis methodologies, including the shift towards containerized environments for reproducible science and streamlined workflows.
Computational Materials Design and Data Dissemination through the Materials P...
Computational Materials Design and Data Dissemination through the Materials P...Anubhav Jain The Materials Project is a free online database containing calculated properties of over 150,000 materials designed to help researchers discover new functional materials. It provides data on electronic, thermal, mechanical, magnetic and other material properties calculated using high-performance computing. The database can be accessed through a website, API, and various apps. The code powering the Materials Project is also open source. It has been heavily used by the research community, with over 180,000 registered users conducting data-driven materials design studies. The Materials Project team is working to expand the community through initiatives like allowing experimental data contributions and benchmarking machine learning algorithms.
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...Data Con LA The document presents an overview of Heron, Twitter's real-time stream processing engine designed to enhance scalability, performance, and debuggability compared to its predecessor, Storm. It details Heron's architecture, including topology design and resource management, along with strategies to handle challenges like stragglers and back pressure. Additionally, the paper discusses Heron's capabilities in processing vast amounts of data efficiently while supporting diverse real-time use cases at Twitter.
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneIan Foster The document discusses the challenges in data-intensive science and the potential of outsourcing mundane research tasks through Software-as-a-Service (SaaS) solutions. It highlights how automation can accelerate discovery by reducing the time researchers spend on administrative tasks and data handling. The Globus Online platform is presented as an example of leveraging cloud services to streamline data movement and enhance research efficiency.
DuraMat Data Management and Analytics
DuraMat Data Management and AnalyticsAnubhav Jain This document summarizes work on developing clear sky detection methods and photovoltaic data analytics tools. It describes collaborating with NREL and kWh Analytics to build a robust clear sky detection method for the RdTools software. The goal is to automatically learn the best parameters for the PVLib clear sky model by comparing its labels to known clear sky labels from satellite data. It also discusses developing open-source software to analyze string-level I-V curves collected by Sandia National Labs to detect mismatching and extract IV parameters. The work aims to help researchers by providing data management, analytics and predictive modeling through a DuraMat Data Hub.
Research workflow - 4 June 2018
Research workflow - 4 June 2018Zachary Labe The document discusses the evolving role of Python in climate science research, highlighting that modern scientists increasingly spend time debugging code rather than analyzing traditional data sources. It emphasizes the importance of transparent, reproducible coding practices and collaborative tools like GitHub for managing scientific workflows. The document calls for greater innovation and sharing within the scientific community to address the challenges posed by big data.
Butler - a framework for a large-scale scientific analysis on the cloud - EOS...
Butler - a framework for a large-scale scientific analysis on the cloud - EOS...ATMOSPHERE . Butler is a framework for analyzing thousands of human genomes from cloud infrastructure. It provides functionality for provisioning infrastructure, configuration management, defining and executing workflows, and operational management. The document describes how Butler has been used to analyze over 2800 human genomes from several cancer studies across multiple cloud providers. Key components that improve analysis time include tools for workflow management and operational monitoring.
Sgg crest-presentation-final
Sgg crest-presentation-finalmarpierc The document provides an overview of the Science Gateway Group at Indiana University. It introduces the group members and describes their focus areas as developing open source software for cyberinfrastructure like Apache Rave and Apache Airavata. It discusses the group's work on extending collaborations with application scientists in various domains. The document also outlines possibilities for collaboration with the PTI CREST Lab on topics like scientific workflows and generalized execution frameworks.
DCSF 19 Towards Reproducable Climate Research
DCSF 19 Towards Reproducable Climate ResearchDocker, Inc. The document discusses the importance of reproducible climate research, emphasizing the need for data analyses and scientific claims to be published alongside their supporting data and code for verification and collaboration. It presents a beginner's guide on using Docker containers to facilitate climate analysis and shares steps for creating, building, and sharing Docker images for Jupyter notebooks. The document also highlights use-cases for Docker in data publishing and collaborative software development, while providing relevant references and resources.
Spark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scaleAndy Petrella This document discusses the use of spark, adam, tachyon, and the spark notebook for analyzing genomic data at scale, highlighting the challenges and benefits of distributed machine learning. It emphasizes the importance of collaboration among distributed teams in managing vast amounts of genomic data and the need for effective sharing of processes and results. The discussion also touches on the lengthy nature of medical research and the critical role of reproducibility in advancing scientific knowledge.
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...Shadab Ali Khan This document outlines a proposal to conceptualize and prototype a scalable genomic data analysis pipeline using Project Glow and Apache Spark on Amazon Web Services' (AWS) Databricks platform. It discusses the growing amounts of human genomic data, limitations of existing bioinformatics tools, and how distributed computing frameworks like Apache Spark and cloud platforms can enable analysis of huge genomic datasets.
More Related Content
What's hot (20)
Big Data, Big Computing, AI, and Environmental Science
Big Data, Big Computing, AI, and Environmental ScienceIan Foster The document discusses three transformative technologies: exascale computing, big data, and artificial intelligence, which are significantly enhancing scientific research capabilities. It outlines the advancements in AI for science, particularly using deep learning for applications like automated cloud classification and climate modeling, while emphasizing the integration of large-scale data from satellites. Additionally, it describes the processing and classification of extensive MODIS satellite imagery to develop insights into cloud behavior and environmental phenomena.
Data Automation at Light Sources
Data Automation at Light SourcesIan Foster The document discusses data automation at light sources, highlighting the challenges posed by immense data rates and the limitations of current methods. It emphasizes the need for new experimental designs, data management and analysis techniques, proposing automation and outsourcing strategies to handle data effectively. Examples include high-speed storage services, cloud-based automation, and the Data and Learning Hub for managing machine learning models and data publishing.
Machine Learning in Healthcare Diagnostics
Machine Learning in Healthcare DiagnosticsLarry Smarr Machine learning and artificial intelligence are rapidly transforming healthcare and medicine. Advances in genetic sequencing have enabled the mapping of human and microbial genomes at low costs. Researchers are using machine learning to analyze genomic and microbiome data to better understand health and disease. Non-von Neumann brain-inspired computing architectures are being developed for machine learning applications and could accelerate medical research and diagnostics. These technologies may help create personalized health coaching and move medicine from reactive sickcare to proactive healthcare.
Foster CRA March 2022.pptx
Foster CRA March 2022.pptxIan Foster The document discusses the concept of a national discovery cloud designed to enhance human intellect through advanced computing and data integration. It highlights the importance of creating new tools, platforms, and educational initiatives to facilitate research and innovation. Key challenges include ensuring privacy, balancing resource needs, and integrating diverse capabilities into a cohesive framework for the R&D community.
Coding the Continuum
Coding the ContinuumIan Foster The document discusses the evolution of computing and networking, emphasizing the concept of the 'computing continuum' where computing resources are distributed across various types of appliances and locations. It highlights technological advancements, such as 5G and cloud computing, and presents models for optimizing performance based on speed, location, and resource utilization. The narrative ties together concepts of latency, computational efficiency, and cost-management in modern computing environments.
2014 moore-ddd
2014 moore-dddc.titus.brown This document discusses enabling data-intensive biology through superior software and algorithms. It proposes a distributed graph database server that would allow querying across multiple public and private data sets. This would help address the growing data challenge in biology by providing a way to explore, query and mine large datasets in an open and collaborative manner. The goal is to incentivize data sharing and enable new types of data-driven investigations.
Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014
Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014Microsoft Azure for Research Microsoft Research (MSR) was established from a 1990 memo, led by Rick Rashid and later Peter Lee, to advance research, integrate findings into products, and impact the computing field. It encompasses over 1,100 researchers across global locations, significantly contributing to Microsoft's intellectual property and technology development, exemplified by breakthroughs in areas such as speech recognition and cloud computing. The Azure for Research program provides free access to cloud resources for collaborative and data-intensive scientific projects, showcasing MSR's commitment to accelerating scientific discovery.
High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...
High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...Larry Smarr The document discusses advancements in high-performance cyberinfrastructure at UC San Diego, focusing on enabling data-driven science for stem cell research and other fields. It highlights the deployment of a 10 Gbps network and various computational resources, such as the Triton data analysis facility and shared storage solutions, to address the challenges of massive data processing. Additionally, it outlines collaborative efforts and ongoing projects that aim to enhance scientific research capabilities through optimized data storage and analysis techniques.
NERSC, AI and the Superfacility, Debbie Bard
NERSC, AI and the Superfacility, Debbie BardPacificResearchPlatform The document outlines a workshop focused on advanced AI workflows in astronomy and microscopy at NERSC, highlighting its role as a production HPC and data facility supporting various scientific disciplines. It discusses the integration of simulation and data analysis, evolving scientific workflows, and the development of superfacilities to enhance data processing and analysis efficiency. Examples include a supernova detection pipeline and the implementation of high-throughput systems for real-time data management in large-scale astrophysics projects.
Big data ecosystem
Big data ecosystemSlideCentral This document provides a summary of the BigData ecosystem. It lists various distributed filesystems, NoSQL databases, data models, distributed programming frameworks, data ingestion tools, scheduling tools, system development tools, service programming tools, and machine learning tools that are part of the BigData ecosystem. It also defines the size of bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, and exabytes. Some related links on open data, NoSQL databases, traditional databases vs NoSQL, and the role of SQL in big data are also included.
Foss4G 2009 Scenz Grid
Foss4G 2009 Scenz Gridnoho SCENZ-Grid proposes a science collaboration and computation environment to enable researchers to do science research online together by sharing data and computational resources without duplicating them. It would allow researchers to collaboratively develop and use shared models and workflows, and connect researchers directly to policymakers, managers, educators and the public. The environment would use spatial, computational and web services and technologies like Web Processing Services, 52North, GeoServer and OpenLayers to query and analyze data from multiple sources on distributed computational resources. It aims to establish relationships between different datasets from various organizations through similarity matrices and expertise to combine attributes from different classifications.
Cloud Accelerated Genomics
Cloud Accelerated GenomicsIdan Tohami The document discusses the integration of cloud computing with genomics, emphasizing the need to accelerate the translation of genomic research into practical applications by addressing bottlenecks in data processing. It highlights Google's Cloud Platform and its various tools, such as BigQuery and Dataflow, which facilitate the analysis of large genomic datasets, and showcases example use cases. Overall, the text illustrates the evolution of genomic analysis methodologies, including the shift towards containerized environments for reproducible science and streamlined workflows.
Computational Materials Design and Data Dissemination through the Materials P...
Computational Materials Design and Data Dissemination through the Materials P...Anubhav Jain The Materials Project is a free online database containing calculated properties of over 150,000 materials designed to help researchers discover new functional materials. It provides data on electronic, thermal, mechanical, magnetic and other material properties calculated using high-performance computing. The database can be accessed through a website, API, and various apps. The code powering the Materials Project is also open source. It has been heavily used by the research community, with over 180,000 registered users conducting data-driven materials design studies. The Materials Project team is working to expand the community through initiatives like allowing experimental data contributions and benchmarking machine learning algorithms.
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...Data Con LA The document presents an overview of Heron, Twitter's real-time stream processing engine designed to enhance scalability, performance, and debuggability compared to its predecessor, Storm. It details Heron's architecture, including topology design and resource management, along with strategies to handle challenges like stragglers and back pressure. Additionally, the paper discusses Heron's capabilities in processing vast amounts of data efficiently while supporting diverse real-time use cases at Twitter.
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneIan Foster The document discusses the challenges in data-intensive science and the potential of outsourcing mundane research tasks through Software-as-a-Service (SaaS) solutions. It highlights how automation can accelerate discovery by reducing the time researchers spend on administrative tasks and data handling. The Globus Online platform is presented as an example of leveraging cloud services to streamline data movement and enhance research efficiency.
DuraMat Data Management and Analytics
DuraMat Data Management and AnalyticsAnubhav Jain This document summarizes work on developing clear sky detection methods and photovoltaic data analytics tools. It describes collaborating with NREL and kWh Analytics to build a robust clear sky detection method for the RdTools software. The goal is to automatically learn the best parameters for the PVLib clear sky model by comparing its labels to known clear sky labels from satellite data. It also discusses developing open-source software to analyze string-level I-V curves collected by Sandia National Labs to detect mismatching and extract IV parameters. The work aims to help researchers by providing data management, analytics and predictive modeling through a DuraMat Data Hub.
Research workflow - 4 June 2018
Research workflow - 4 June 2018Zachary Labe The document discusses the evolving role of Python in climate science research, highlighting that modern scientists increasingly spend time debugging code rather than analyzing traditional data sources. It emphasizes the importance of transparent, reproducible coding practices and collaborative tools like GitHub for managing scientific workflows. The document calls for greater innovation and sharing within the scientific community to address the challenges posed by big data.
Butler - a framework for a large-scale scientific analysis on the cloud - EOS...
Butler - a framework for a large-scale scientific analysis on the cloud - EOS...ATMOSPHERE . Butler is a framework for analyzing thousands of human genomes from cloud infrastructure. It provides functionality for provisioning infrastructure, configuration management, defining and executing workflows, and operational management. The document describes how Butler has been used to analyze over 2800 human genomes from several cancer studies across multiple cloud providers. Key components that improve analysis time include tools for workflow management and operational monitoring.
Sgg crest-presentation-final
Sgg crest-presentation-finalmarpierc The document provides an overview of the Science Gateway Group at Indiana University. It introduces the group members and describes their focus areas as developing open source software for cyberinfrastructure like Apache Rave and Apache Airavata. It discusses the group's work on extending collaborations with application scientists in various domains. The document also outlines possibilities for collaboration with the PTI CREST Lab on topics like scientific workflows and generalized execution frameworks.
DCSF 19 Towards Reproducable Climate Research
DCSF 19 Towards Reproducable Climate ResearchDocker, Inc. The document discusses the importance of reproducible climate research, emphasizing the need for data analyses and scientific claims to be published alongside their supporting data and code for verification and collaboration. It presents a beginner's guide on using Docker containers to facilitate climate analysis and shares steps for creating, building, and sharing Docker images for Jupyter notebooks. The document also highlights use-cases for Docker in data publishing and collaborative software development, while providing relevant references and resources.
Similar to Bioinformatics Data Pipelines built by CSIRO on AWS (20)
Spark Summit Europe: Share and analyse genomic data at scale
Spark Summit Europe: Share and analyse genomic data at scaleAndy Petrella This document discusses the use of spark, adam, tachyon, and the spark notebook for analyzing genomic data at scale, highlighting the challenges and benefits of distributed machine learning. It emphasizes the importance of collaboration among distributed teams in managing vast amounts of genomic data and the need for effective sharing of processes and results. The discussion also touches on the lengthy nature of medical research and the critical role of reproducibility in advancing scientific knowledge.
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...Shadab Ali Khan This document outlines a proposal to conceptualize and prototype a scalable genomic data analysis pipeline using Project Glow and Apache Spark on Amazon Web Services' (AWS) Databricks platform. It discusses the growing amounts of human genomic data, limitations of existing bioinformatics tools, and how distributed computing frameworks like Apache Spark and cloud platforms can enable analysis of huge genomic datasets.
Nyc big datagenomics-pizarroa-sept2017
Nyc big datagenomics-pizarroa-sept2017delagoya This document summarizes a presentation about using AWS Batch and AWS Step Functions for genomic analysis workflows. It discusses:
- AWS Batch for running containerized jobs on EC2 instances in a managed way. Jobs are run based on definitions, queues, and compute environments.
- AWS Step Functions for visualizing and coordinating the components of distributed applications using state machines and workflows.
- An example architecture using AWS Batch for the job execution layer and AWS Step Functions to orchestrate the workflow, providing flexibility, ease of deployment, and integration with non-Batch applications.
- Potential considerations for data sharing, multitenancy, and volume reuse when using AWS Batch for genomic analysis jobs.
Hadoop ecosystem for health/life sciences
Hadoop ecosystem for health/life sciencesUri Laserson Uri Laserson gave a presentation on using the Hadoop ecosystem for life sciences applications. He discussed how Hadoop was developed based on Google's GFS and MapReduce systems to provide scalable, fault-tolerant data storage and processing. Laserson then provided several examples of using Hadoop for genomics applications such as scaling genome sequencing pipelines and enabling interactive querying of large variant call format datasets. He also outlined other potential use cases in clinical data, manufacturing, and agriculture.
Hadoop for Bioinformatics: Building a Scalable Variant Store
Hadoop for Bioinformatics: Building a Scalable Variant StoreUri Laserson The document discusses the application of the Hadoop ecosystem in genomics, highlighting its evolution from Google’s infrastructure to an open-source platform at Yahoo!, known as Apache Hadoop. It emphasizes the advantages of Hadoop, such as fault tolerance, scalability, and effective data processing for large genomic datasets, while addressing challenges in bioinformatics, like data access and computational efficiency. Recommendations for future development include adopting Hadoop for data centralization, improving consent models, and promoting open-source methodologies to enhance reproducibility.
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudOla Spjuth The document discusses data-intensive bioinformatics and challenges with analyzing large genomic datasets on high performance computing (HPC) resources. It summarizes that storage is the biggest challenge as sequencing projects generate very large amounts of data and users do not clean up data. The strategies discussed to address this include assessing costs of storage and analysis upfront, limiting project lifetimes, moving to tiered storage, and improving efficiency. It also discusses using cloud computing resources through virtual clusters and containers to enable flexible, on-demand access and pay-per-use pricing models. Scientific workflows and microservices approaches are presented as ways to automate and orchestrate large-scale genomic analyses on distributed computing resources.
11-Big Data Application in Biomedical Research and Health Care.pptx
11-Big Data Application in Biomedical Research and Health Care.pptxshikhamittal42 Big data technologies are increasingly being used in biomedical research and healthcare. Three key points:
1) Biomedical data is growing rapidly in terms of volume, variety, and velocity, including omics data, electronic health records, medical images, and sensor data.
2) Technologies like Hadoop, cloud computing, and MapReduce are being used to store, process, and analyze large biomedical datasets across distributed systems. Tools have been developed for tasks like genome mapping, variant calling, and protein identification.
3) Big data is being applied to problems such as disease research, drug discovery, and precision medicine through techniques including machine learning, data mining, and integration of diverse data types. Repositories also
What is the "Big Data" version of the Linpack Benchmark? ; What is “Big Data...
What is the "Big Data" version of the Linpack Benchmark? ; What is “Big Data...Geoffrey Fox The document discusses the need for a 'big data' version of established benchmarks like Linpack and Berkeley Dwarfs to further high-performance analytics in data-heavy applications. It proposes developing a spectrum of benchmarks that encompass various facets and requirements of big data analytics, emphasizing the importance of covering both the Apache big data stack and parallel computing experiences. The presentation raises the potential for a comprehensive benchmark set that addresses the unique challenges and features of big data applications, encouraging collaboration and discussion in the research community.
Spark meetup london share and analyse genomic data at scale with spark, adam...
Spark meetup london share and analyse genomic data at scale with spark, adam...Andy Petrella The document covers a meetup presentation discussing the application of Apache Spark in genomics, specifically focusing on handling large genomic datasets like the 1,000 Genomes Project. It details the use of various tools and standards, such as ADAM and GA4GH, to process, analyze, and share genomic data efficiently. The presentation also highlights challenges and solutions in genomic data processing and the importance of standardized frameworks for data sharing in the field of genomics.
Big data analytics_7_giants_public_24_sep_2013
Big data analytics_7_giants_public_24_sep_2013Vijay Srinivas Agneeswaran, Ph.D 1. The document discusses the limitations of Hadoop for advanced analytics tasks beyond basic statistics like mean and variance.
2. It introduces several distributed data analytics platforms like Spark, Storm, and GraphLab that can perform tasks like linear algebra, graph processing, and iterative machine learning algorithms more efficiently than Hadoop.
3. Specific use cases from companies that moved from Hadoop to these platforms are discussed, where they saw significantly faster performance for tasks like logistic regression, collaborative filtering, and k-means clustering.
The Future of Healthcare with Big Data and AI with Ion Stoica and Frank Nothaft
The Future of Healthcare with Big Data and AI with Ion Stoica and Frank NothaftDatabricks The document discusses how big data and AI can transform healthcare through genomic analysis. It describes 3 major challenges with current approaches: 1) complex genomic pipelines are costly and time-consuming, 2) analytics are rigid and limit research scope, and 3) teams working on genomics are siloed, reducing productivity. The proposed solution is a unified analytics platform for genomics that features pre-built pipelines, powerful analytics like machine learning, and collaborative workspaces. This would help accelerate discovery by processing massive genomic datasets faster, enabling new research questions to be answered, and improving collaboration across teams.
Design and evaluation of a genomics variant analysis pipeline using GATK Spar...
Design and evaluation of a genomics variant analysis pipeline using GATK Spar...Paolo Missier The document discusses the design and evaluation of a genomics variant analysis pipeline utilizing GATK Spark tools, focusing on processing time, cost efficiency, and quality of analysis. It highlights a multi-sample whole exome sequencing (WES) pipeline and evaluates its performance across different deployment configurations, revealing insights into speed and resource utilization. Additionally, it addresses the adaptability of the pipeline to software version changes and the implications for ongoing data analysis in the genomic field.
What’s New in the Berkeley Data Analytics Stack
What’s New in the Berkeley Data Analytics StackTuri, Inc. The document discusses the Berkeley Data Analytics Stack (BDAS) developed by UC Berkeley's AMPLab. It summarizes the key components of the BDAS including Spark, Mesos, Tachyon, MLlib, and Velox. It describes how the BDAS provides a unified platform for batch, iterative, and streaming analytics using in-memory techniques. It also discusses recent developments like KeystoneML/ML Pipelines for scalable machine learning and SampleClean for human-in-the-loop analytics. The goal is to make it easier to build and deploy advanced analytics applications on large datasets.
All In - Migrating a Genomics Pipeline from BASH/Hive to Spark (Azure Databri...
All In - Migrating a Genomics Pipeline from BASH/Hive to Spark (Azure Databri...Databricks The document details the migration of a genomics pipeline from legacy systems to Azure Databricks, focusing on the challenges faced with the previous Hive implementation and the need for an efficient, stable, and cost-effective solution. It highlights the objectives for interoperability in clinical trials and pharmacogenomics, emphasizing the importance of real-time data for treatment decisions. Key insights from the migration experience include improved infrastructure stability, reduced costs, and the ability to handle increased data volume more effectively.
Open Source Networking Solving Molecular Analysis of Cancer
Open Source Networking Solving Molecular Analysis of CancerOpen Networking Summit The document discusses advancements in open-source networking and precision medicine, particularly in the molecular analysis of cancer, enabling tailored diagnostics and improved patient outcomes. It emphasizes the integration of various technologies such as real-time data processing tools (Kafka, NiFi), advanced databases (Cassandra), and machine learning platforms (Singa) to handle large datasets and facilitate drug development. The work highlights the challenges of synthesizing biometric data and the importance of open-source solutions to overcome technical obstacles while addressing the complexities of personalized medicine.
Production Bioinformatics, emphasis on Production
Production Bioinformatics, emphasis on ProductionChris Dwan Chris Dwan emphasizes the importance of production bioinformatics and the integration of data-driven insights in healthcare at Sema4. Sema4 leverages AI and machine learning to transform clinical and genomic data into personalized health trajectories, offering various genomic tests and data platforms. The document details operational methodologies, data management techniques, and a focus on maintaining high quality and accountability in bioinformatics workflows.
Big data analysing genomics and the bdg project
Big data analysing genomics and the bdg projectsree navya This document discusses a project analyzing genomics using big data techniques. It introduces Spark, a framework for large-scale data analysis, and ADAM, a Spark-based framework for genomic data. Key points covered include an overview of Spark and Hadoop, data types and formats in genomics, and using ADAM and Spark to perform genomic analysis and queries on large datasets like the 1000 Genomes Project.
Next generation genomics: Petascale data in the life sciences
Next generation genomics: Petascale data in the life sciencesGuy Coates The Sanger Institute, a major genomic research center funded by the Wellcome Trust, has significantly advanced DNA sequencing technologies, reducing the cost and time required to sequence human genomes. It is focused on managing vast amounts of data generated from these sequencing efforts and aims to enhance data sharing and collaboration through systems like iRODS. The institute's initiatives include analyzing cancer genomes to identify driver mutations for targeted therapies as part of the International Cancer Genome Project.
Big data week London Big data pipelining 0.2
Big data week London Big data pipelining 0.2Simon Ambridge This document details the steps to build and run a human genome big data pipeline using Docker, targeted for a presentation during Big Data Week London 2015. It covers prerequisites, installation procedures, running the Devoxx demo, and accessing data from Cassandra, aiming to enhance participants' understanding of modern data pipeline techniques. The document also references additional resources such as Spark Notebook, which facilitates reactive data science methodologies.
Cloud Computing and Innovations for Optimizing Life Sciences Research
Cloud Computing and Innovations for Optimizing Life Sciences ResearchInterpretOmics This document discusses how cloud computing and big data analytics can optimize life sciences research. It outlines some key benefits of cloud computing like scalability, flexibility, and pay-as-you-go models. The document also discusses challenges in healthcare big data like volume, velocity, variety and veracity of data. It provides examples of data analysis pipelines and tools for tasks like quality control, variant calling, clustering and enrichment analysis that can help researchers. Finally, it argues that cloud computing has the potential to address challenges and transform biology and healthcare research.
Ad
More from Lynn Langit (20)
VariantSpark on AWS
VariantSpark on AWSLynn Langit The document discusses Variantspark, a bioinformatics tool developed for AWS that enables efficient genomic variant analysis using supervised machine learning techniques. It describes its capabilities, performance metrics, and cloud data pipeline configurations, as well as the CSIRO team's efforts to create reproducible environments for researchers. Future development plans include comparing Variantspark's performance on different cloud platforms and optimizing workflow tools.
Serverless Architectures
Serverless ArchitecturesLynn Langit This document discusses serverless computing and compares different cloud computing services. It outlines how serverless computing can be used for various workloads including web requests, queue messages, transactions, infrequent tasks, and scheduled jobs. It also compares common services between AWS and GCP for virtual machines, containers, functions, storage, databases, machine learning, IoT, and more. Serverless architectures are suggested to isolate and scale dynamic workloads without needing to manage servers.
10+ Years of Teaching Kids Programming
10+ Years of Teaching Kids ProgrammingLynn Langit The document discusses coding education initiatives, focusing on the teaching of coding to children through various methods and tools, notably using visual programming languages like Kodu and Small Basic. It highlights events held globally, teacher training programs, and the development of free, open-source courseware for coding. The efforts aim to engage both boys and girls in coding, fostering collaborative learning and a growth mindset through hands-on experiences and risk-taking.
Blastn plus jupyter on Docker
Blastn plus jupyter on DockerLynn Langit The document discusses the integration of Jupyter notebooks in bioinformatics, highlighting their capabilities for data collection, preparation, model training, and deployment. It emphasizes Jupyter's role in reproducible research and collaboration, as well as its evolution from academic use to a market-leading tool with cloud options and robust customization. Additionally, it mentions the Basic Local Alignment Search Tool (BLAST) for sequence comparison and various platforms supporting Jupyter, like Docker and cloud services.
Testing in Ballerina Language
Testing in Ballerina LanguageLynn Langit The document discusses different types of tests for microservices including unit tests, service tests, services composition tests, deployment tests, and modern approaches to testing microservices. It provides examples of testing functions, services, interactions, and deploying to containers using tools like Docker. It emphasizes the importance of testing at the unit, integration, deployment levels as well as testing documentation, full stack setups, and chaos engineering.
Teaching Kids to create Alexa Skills
Teaching Kids to create Alexa SkillsLynn Langit The document discusses various methods for interacting with voice interfaces, emphasizing that they can be significantly faster than typing. It provides resources for trying out simulators, custom applications, and published skills for Alexa, as well as links to development tools and educational materials. Key activities include using Alexa Blueprints, the developer console, and accessing the Alexa Skills Store.
Practical cloud
Practical cloudLynn Langit AWS dominates the US cloud market with significant market share, relying heavily on established services like EC2 and RDS. New technologies such as serverless options (e.g., Lambda) and container management are emerging, presenting challenges and opportunities for cloud project execution and integration. Organizations face decision-making intricacies regarding service types, cost control, and the involvement of different roles in cloud migration.
Understanding Jupyter notebooks using bioinformatics examples
Understanding Jupyter notebooks using bioinformatics examplesLynn Langit The document discusses the evolution and capabilities of Jupyter Notebooks in bioinformatics and machine learning. It highlights the various stages in the ML lifecycle supported by Jupyter, including data collection, preparation, model evaluation, and deployment. Additionally, it covers advancements in genomic analysis using tools like Variant Spark and cloud-based Jupyter services, emphasizing their role in facilitating reproducible research and collaboration in academic environments.
Teaching Kids Programming
Teaching Kids ProgrammingLynn Langit Lynn Langit was introduced to programming at age 16 through Teaching Kids Programming (TKP) events. She went on to teach TKP events while studying computer science and bioinformatics in college. After interning at Microsoft Research and working at Pivotal Labs, she created TKP Java courseware to teach programming to middle school students. The courseware includes 80 coding lessons and bridges students to AP Computer Science courses using a teacher-led, puzzle-based approach. Langit advocates for developers to create educational courseware, work with K-12 teachers, and be visible technical role models to help address the lack of gender diversity in computer science fields.
Practical Cloud
Practical CloudLynn Langit The document analyzes the dominance of AWS in the cloud market, highlighting its market share and service offerings. It discusses various cloud project outcomes, the evolution of application deployment technologies, and the cost-effectiveness of different computing models, including server-based and serverless solutions. Additionally, it mentions the importance of data management and integration tools in optimizing cloud services.
Serverless Reality
Serverless RealityLynn Langit The document discusses various aspects of cloud computing, emphasizing serverless architectures and the use of AWS Lambda, Azure Functions, and other services. It highlights the benefits of dynamic workloads, scheduling tasks, and cost efficiency compared to traditional server setups. Additionally, it touches on machine learning applications and the importance of educating young students in programming.
Serverless Reality
Serverless RealityLynn Langit This document discusses serverless computing and compares it to traditional server-based computing. It defines serverless computing and provides examples of serverless technologies like AWS Lambda. It also outlines common use cases for serverless computing like handling dynamic workloads and scheduled tasks. Finally, it compares different services between server-based and serverless models like compute, files, databases, data pipelines, machine learning, and IoT.
Beyond Relational
Beyond RelationalLynn Langit The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.
New AWS Services for Bioinformatics
New AWS Services for BioinformaticsLynn Langit New AWS services were announced at re:Invent 2016 including Athena, Step Functions, Batch, Glue, and QuickSight that could be useful for scaling bioinformatics pipelines. Athena allows SQL queries on data stored in S3, Step Functions allows creating serverless visual workflows using Lambda functions, and Batch provides fully managed batch processing at scale across AWS services. Glue provides serverless ETL capabilities, and QuickSight allows creating quick data dashboards. Examples were shown of using these services for genomics workflows, running jobs on unmanaged compute environments, and processing genomic data.
Google Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsLynn Langit 1. The document discusses various Google Cloud Platform products and patterns for data pipelines, including virtual machines, storage, data warehousing, streaming analytics, machine learning, internet of things, and bioinformatics.
2. Demos and examples are provided of storage, virtual machines, BigQuery, Cloud Spanner, and machine learning on the Google Cloud Platform.
3. The core Google Cloud Platform products discussed for various data and analytics use cases include Cloud Storage, BigQuery, Cloud Dataflow, Compute Engine, Cloud Pub/Sub, and Bigtable.
Scaling Galaxy on Google Cloud Platform
Scaling Galaxy on Google Cloud PlatformLynn Langit The document outlines the features and capabilities of scaling the Galaxy platform on Google Cloud Platform (GCP), including the use of virtual machines, Docker containers, and genomics data management. It introduces various key concepts such as cloud storage, the Google Genomics API, and BigQuery for data analysis. Additionally, it presents advanced pipeline configurations and resources for efficient genomic data processing on GCP.
SQL Server on Google Cloud Platform
SQL Server on Google Cloud PlatformLynn Langit SQL Server can run fast and well-priced on Google Cloud Platform infrastructure, with data centers opening locally in Australia in 2017. GCP services like Google Compute Engine offer on-demand virtual machines in various sizes running Linux, Windows, and more. A demo showed how to set up and use SQL Server 2016 with its new features on GCP, with step-by-step guides, best practices, and load testing tutorials available.
Redis Labs and SQL Server
Redis Labs and SQL ServerLynn Langit The document discusses Redis, an open-source and commercial in-memory database platform known for its high performance and low latency, especially in high-volume write scenarios. It details the services provided by Redis Labs, including Redis Cloud and Redis Labs Enterprise Cluster, as well as various use case examples across different industries. The document emphasizes the operational benefits of Redis, such as cost savings, simplified scaling, and enterprise-class management and support.
Building a data warehouse with AWS Redshift, Matillion and Yellowfin
Building a data warehouse with AWS Redshift, Matillion and YellowfinLynn Langit The document outlines the process of building a data warehouse on AWS using Amazon S3 and Amazon Redshift for data analytics. It explains data collection, processing, and analysis methods, specifically using Matillion ETL for Redshift to prepare datasets from US airport flight information and weather data. Additionally, it discusses loading data into Redshift, setting distribution styles, and utilizing sorting keys for efficient queries, integrating visualization with Yellowfin.
What is 'Teaching Kids Programming'
What is 'Teaching Kids Programming'Lynn Langit This document discusses Teaching Kids Programming (TKP), a global non-profit that provides free, open-source coding curriculum to teach kids ages 10-17 Java and other programming languages. TKP has taught over 4,000 students through 70 trained teachers using a 40-hour Java curriculum that allows students to start coding within 90 seconds. They partner with schools, non-profits, and organizations around the world and provide teaching resources, curriculum, and support to address the lack of computer science education for kids. The goal is to help more young people, especially girls, gain skills in programming and consider careers in technology.
Ad
Recently uploaded (20)
The scientific heritage No 162 (162) (2025)
The scientific heritage No 162 (162) (2025)The scientific heritage The scientific heritage No 162 (162) (2025)
Gas Exchange in Insects and structures 01
Gas Exchange in Insects and structures 01PhoebeAkinyi1 Description of structures involved in gaseous exchange
Science 8 Quarter 4 first quiz digestive system.docx
Science 8 Quarter 4 first quiz digestive system.docxjunefermunez Quiz for the topic Digestive System. 4th Qaurter
Science Holiday Homework (interesting slide )
Science Holiday Homework (interesting slide )aryanxkohli88 science holiday homework
good for children
Overview of Stem Cells and Immune Modulation.ppsx
Overview of Stem Cells and Immune Modulation.ppsxAhmedAtwa29 This presentation, "Stem Cells & Immune Modulation: The Future of Regenerative Medicine?" by Dr. Ahmed Atwa, explores the groundbreaking potential of stem cells in regenerative medicine and immune regulation. It covers stem cell types (Embryonic, Adult, iPSCs), their unique properties (self-renewal, differentiation), and therapeutic applications in tissue repair, disease modeling, and immunomodulation. Special focus is given to mesenchymal stem cells (MSCs) and their role in modulating immune responses through cytokines, extracellular vesicles (EVs), and preconditioning strategies. The slides also address clinical challenges, cancer stem cell evasion, and future directions in stem cell therapy.
Paired Sketching of Distributed User Interfaces:Workflow, Protocol, Software ...
Paired Sketching of Distributed User Interfaces:Workflow, Protocol, Software ...Jean Vanderdonckt Paper présented at ACM EICS '25:
The evolving landscape of distributed user interfaces requires the prototyping stage also be distributed between users, tasks, platforms, and environments. To create a cohesive distribution of the user interface elements in such ecosystems, paired sketching has emerged as a collaborative design method that leverages multiple stakeholders’ strengths, including designers, developers, and end users, working in pairs. In the context of developer experience applied to paired sketching for distributed
user interfaces, we decomposed a workflow into four disciplines according to the Software and Systems Process Engineering
Meta-Model (SPEM) notation. First, we defined a protocol to deploy paired sketching of distributed user interfaces, supported
by UbiSketch, a collaborative software environment tailored featuring sketch recognition and whiteboarding. Second, to
evaluate paired sketching for engineering interactive systems, we conducted an experiment involving five pairs of stakeholders
who sketched a distributed user interface for inside-the-vehicule interaction distributed on four platforms: smartphone, tablet, pen display, and tabletop. Empirical results from questionnaires, reactivity, intention, perceived satisfaction, and free comments, suggest a preference order in which the tabletop is ranked first, followed by the tablet, smartphone, and pen display. Based on these results, we discuss the potential of paired sketching for distributed user interfaces.
Study of Appropriate Information Combination in Image-based Obfuscated Malwar...
Study of Appropriate Information Combination in Image-based Obfuscated Malwar...takahashi34 This slide introduces our study titled "Study of Appropriate Information Combination in Image-based Obfuscated Malware Detection", which was presented at DSN2025.
To tackle the threat of obfuscated malware, our study proposes an image-based detection method that combines four types of information—byte values, entropy, semantic information, and bigrams frequency—from Windows PE executable files. Proposed method is evaluated in a realistic environment containing both obfuscated and non-obfuscated benign and malicious files. Evaluation results show that combining these information types achieves 88.66% accuracy, demonstrating the effectiveness of information combination for detecting obfuscated malware.
MEDICINAL CHEMISTRY PROSPECTIVES IN DESIGN OF EGFR INHIBITORS.pptx
MEDICINAL CHEMISTRY PROSPECTIVES IN DESIGN OF EGFR INHIBITORS.pptx40RevathiP Introduction , importance and role of EGFR and EGFR inhibitors
GBSN__Unit 2 - Control of Microorganisms
GBSN__Unit 2 - Control of MicroorganismsAreesha Ahmad Microbiology for Nursing students - According to New PNC course curriculum - 2025
What is Skeleton system.pptx by aahil sir
What is Skeleton system.pptx by aahil sirbhatbashir421 science, any system of knowledge that is concerned with the physical world and its phenomena and that entails unbiased observations and systematic experimentation. In general, a science involves a pursuit of knowledge covering general truths or the operations of fundamental laws.
Science can be divided into different branches based on the subject of study. The physical sciences study the inorganic world and comprise the fields of astronomy, physics, chemistry, and the Earth sciences. The biological sciences such as biology and medicine study the organic world of life and its processes. Social sciences like anthropology and economics study the social and cultural aspects of human behaviour.
Science is further treated in a number of articles. For the history of Western and Eastern science, see science, history of. For the conceptualization of science and its interrelationships with culture, see science, philosophy of. For the basic aspects of the scientific approach, see physical science, principles of; and scientific method.
Model of a molecule. Atom, Biology, Molecular Structure, Science, Science and Technology. Homepage 2010 arts and entertainment, history and society
Britannica Quiz
Science Quiz
The Editors of Encyclopaedia Britannica
This article was most recently revised and updated by Encyclopaedia Britannica.
physics
Table of Contents
Introduction & Top Questions
The scope of physics
The methodology of physics
Relations between physics and other disciplines and society
References & Edit History
Related Topics
Images & Videos
Bernoulli model of gas pressureillustration of Robert Hooke's law of elasticity of materialsLaser Interferometer Space Antenna (LISA)curved space-timetemperature scalesBrownian particleElectromagnetic radiationDiffraction gratingMillikan oil-drop experiment
Robert Millikan's method for measuring the charge of an electron
For Students
Diffraction grating
physics summary
Quizzes
Michael Faraday (L) English physicist and chemist (electromagnetism) and John Frederic Daniell (R) British chemist and meteorologist who invented the Daniell cell.
Faces of Science
Italian-born physicist Dr. Enrico Fermi draws a diagram at a blackboard with mathematical equations. circa 1950.
Physics and Natural Law
Italian physicist Guglielmo Marconi at work in the wireless room of his yacht Electra, c. 1920.
All About Physics Quiz
atom. Orange and green illustration of protons and neutrons creating the nucleus of an atom.
Another Physics Quiz
Encyclopaedia Britannica thistle graphic to be used with a Mendel/Consumer quiz in place of a photograph.
How Much Do You Know About Physics?
Related Questions
Why does physics work in SI units?
Read Next
Highway Night Traffic Portland, drive, driving, car, automobile.
What's the Difference Between Speed and Velocity?
Galaxy clusters like Abell 2744 can act as a natural cosmic lens, magnifying light from more distant, background objects through gravity.
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...ShrayasiRoy2 In an era where users are bombarded with endless choices, capturing attention and driving meaningful engagement isn't just about building features — it's about understanding what drives human behavior at its core. This presentation offers a deep dive into how psychological principles can inform smarter, more intuitive product decisions. It’s not just theory — it’s a hands-on blueprint for applying human-centered thinking at every stage of product development.
Grounded in behavioral science, consumer psychology, and cognitive design, this deck unpacks the key psychological drivers behind user motivation, decision-making, emotional engagement, and habit formation. From attention economics and dopamine-driven interactions to trust cues, loss aversion, and the paradox of choice — you'll see how to harness what the mind naturally does to build digital products people don’t just use… they return to, talk about, and even advocate for.
We’ll explore:
Cognitive biases that shape user perception and choices — and how to use them to your advantage without crossing ethical lines.
Emotional design principles that build trust, trigger desire, and turn features into feelings.
User behavior loops that build stickiness, deepen retention, and create emotional investments.
Psychological friction — when to reduce it for conversion, and when to add it for intentionality.
Dark patterns vs. ethical persuasion — the thin line between influence and manipulation.
A/B testing with a psychological lens — learning not just what works, but why it works.
This is more than UX research or product marketing fluff. It’s a call to build with empathy, backed by science, and sharpened by real-world product thinking. Whether you’re shaping onboarding flows, gamifying engagement, designing pricing models, or rethinking retention strategies — this deck will arm you with a new lens: the human mind.
Perfect for product managers, growth strategists, UX designers, behavioral scientists, and anyone serious about building products that resonate, retain, and inspire action.
Because at the heart of every great product… is a human.
Let’s start building for the brain — not just the screen.
GBSN_Unit 3 - Medical and surgical Asepsis
GBSN_Unit 3 - Medical and surgical AsepsisAreesha Ahmad Microbiology for Nursing students - According to New PNC course curriculum - 2025
Bioinformatics Data Pipelines built by CSIRO on AWS
- 1. Cancer Genomics Data Pipelines Lynn & Samantha Langit CSIRO Bioinformatics / Australia June 2017 - Oslo
- 2. 3 Billion data points per patient DNA sample Up to 25% of the population could be sequenced by 2025
- 3. Two Perspectives Bioinformatics Research • Insight • Reproducibility Cloud Architecture • Speed • Low Cost • Simplicity
- 4. Cloud Data Pipeline Pattern Problem • Define business problem Data • Quality • Quantity Candidate Technologies • Ingest • ETL • Biz Analytics • ML • Visualization Build MVPs • Iterate • Learn Assemble Pipeline • Validate each section • Test at scale
- 6. Genomic Sequencing Results CRISPR-Cas9 for molecular engineering technology enables the accurate editing of genomes for researchers. It… Pattern-matching unique sequences of DNA Huge demand for large-scale computation Time-critical dimension to compute NIH-approved for human health Could revolutionize cancer treatments
- 7. Serverless Lambda Architecture Pattern Lambda function 1 Lambda function 2 Lambda function 3 buckets with objects DynamoDB API Gateway Users
- 8. CSIRO: Commonwealth Scientific & Industrial Research Organization
- 11. Scale Genomic Analysis GWAS = genome-wide sequencing data association studies Analysis on large cohort data or imputed SNP array data Clustering on genomic profiles to stratify large-cohort genomic data Viewing datasets with millions of features
- 12. Cloud Data Pipeline Pattern Problem • Define business problem Data • Quality • Quantity Candidate Technologies • Ingest • ETL • Biz Analytics • ML • Visualization Build MVPs • Iterate • Learn Assemble Pipeline • Validate each section • Test at scale
- 13. Genomics (ML) Pipeline Pattern
- 14. What is CSIRO’s solution? For Scale at reasonable cost Use Apache Hadoop For Scale at speed Use Apache Spark for Hadoop For Usability in bioinformatics Create a domain-specific API (OSS library) For global use Leverage Cloud Pipeline Patterns
- 15. GWAS Analysis with Variant-Spark On premise Hadoop Cluster with Apache Spark Genomics Analysts corporate data center
- 16. What is Apache Spark?
- 18. 80% faster than ADAM 90% faster than R 90% faster than Python
- 19. VariantSpark Uses Apache Spark to massively parallelize the generation of random forests to identify disease genes efficiently Analyzes 3,000 samples with 80 million features in < 30 minutes Enables real-time diagnosis by finding similar patients Contributes to motor neuron disease (ALS) research in Australia
- 21. Spark ML Classification Algorithms Wide Random Forest Ensemble of Decision Trees Logistic Regression variant-spark other libraries
- 22. OSS Library variant-spark for all
- 23. usable? performant? extendable? (clean code) using the best language (Scala)? using the ‘best version’ of Spark? using a version of wide random forests that is understandable? Is it…
- 24. How best to Deploy Cloud Hadoop? • IaaS EC2 instances with Apache Hadoop, Apache Spark, more… • PaaS Elastic Map Reduce (EMR) Hadoop cluster • SaaS Vendor-managed, i.e. DataBricks w/Jupyter Notebooks
- 30. DEMO: Who is a Hipster?
- 31. AWS EC2 Spot Instances
- 32. GWAS Analysis with Variant-Spark EC2 Hadoop Cluster with Apache Spark Genomics Analysts Availability Zone 1000 Genomes GWAS input Spot EC2 Hadoop worker instances EC2 Hadoop instances
- 33. Cloud Data Pipeline Pattern Problem Data Candidate Technologies Build MVPs Assemble Pipeline Analyze GWAS -> S3/Hadoop Ingest ETL Analyze Viz S3 -> Databricks DBFS Apache Spark Variant-Spark ML Notebook SQL, R or Python SaaS
- 35. Cloud Data Pipeline Pattern Problem Data Candidate Technologies Build MVPs Assemble Pipeline 1. Scan vcf -> S3/DynamoDB Ingest ETL Analyze Viz S3 Lambda Lambda Lambda/API Gateway Serverless 2. Analyze GWAS -> S3/Hadoop Ingest ETL Analyze Viz S3 -> Databricks DBFS Apache Spark Variant-Spark ML Notebook SQL, R or Python SaaS
- 36. Modern Big Data Pipelines • Problem #1 - Scan • Solution: Serverless Cloud Pipeline • Problem # 2 - Analyze • Solution: SaaS Cloud ML Pipeline
- 37. Cancer Genomics Data Pipelines Lynn & Samantha Langit CSIRO Bioinformatics & variant-spark June 2017 - Oslo
Editor's Notes
- #7: http://www.nature.com/news/first-crispr-clinical-trial-gets-green-light-from-us-panel-1.20137
- #9: http://bioinformatics.csiro.au/ and https://www.csiro.au/en/Locations/NSW/North-Ryde
- #10: https://www.gt-scan.net/ --AND- AMA with Dr, Bauer -- https://www.reddit.com/r/science/comments/5fiicm/science_ama_series_im_denis_bauer_a_team_leader/
- #11: https://aws.amazon.com/blogs/aws/genome-engineering-applications-early-adopters-of-the-cloud/
- #18: https://github.com/csirobigdata/variant-spark
- #22: https://en.wikipedia.org/wiki/Random_forest --and-- https://spark.apache.org/docs/1.6.2/ml-classification-regression.html
- #26: https://databricks.com/
- #35: https://aws.amazon.com/blogs/aws/genome-engineering-applications-early-adopters-of-the-cloud/
- #38: https://github.com/csirobigdata/variant-spark