Data Automation at Light Sources
Download as pptx, pdf1 like584 views
The document discusses data automation at light sources, highlighting the challenges posed by immense data rates and the limitations of current methods. It emphasizes the need for new experimental designs, data management and analysis techniques, proposing automation and outsourcing strategies to handle data effectively. Examples include high-speed storage services, cloud-based automation, and the Data and Learning Hub for managing machine learning models and data publishing.
1 of 35
Download to read offline
![von Laszewski et al., Real-time
analysis, visualization, and
steering of microtomography
experiments at photon
sources, SIAM Parallel
Processing, 1999
I have been working with light sources for some time!
“the data rates and
compute power
required ... are
prodigious, easily
reaching one gigabit per second
and a teraflop per second [respectively]”](https://image.slidesharecdn.com/foster-sri-june-2018-180612125408/85/Data-Automation-at-Light-Sources-3-320.jpg)
![Ptychography: Use GPU cluster for 360x speedup,
from 7 hours to 72 s
[Deng, Vine, Chen, Nashed, Philips, Jin,
Peterka, Ross, Jacobsen]
Enable online analysis and use of fly scans
Microtomography: Use 32K Mira BG/Q nodes to
reduce reconstruction time from days to 2 mins
[Bicer, Gursoy, Kettimuthu, De Carlo, Agrawal]
Identify and correct experimental
misconfiguration
High-energy diffraction microscopy: 10K BG/Q
nodes to reconstruct in 10 minutes
[Sharma, Almer, Wozniak, Wilde, Foster]
Zoom in on crack locations (switch far field near field)
Coherence
Brightness
High Energy
Micrometer porosity structure of shale samples
Microstructure of a copper wire, 0.2mm diameter
Work on high-speed analysis continues](https://image.slidesharecdn.com/foster-sri-june-2018-180612125408/85/Data-Automation-at-Light-Sources-4-320.jpg)
![DLHub
[“beam off image”, …]
model/xray/batch_predict
Automate and outsource:
(4) Data transformation and analysis
▪ Where are the model and trained weights?
▪ How do I run the model on my data?
▪ Should I run the model on my data?
▪ How can I retrain the model on new data?
https://doi.org/10.1109/NYSDS.2017.8085045](https://image.slidesharecdn.com/foster-sri-june-2018-180612125408/85/Data-Automation-at-Light-Sources-26-320.jpg)
![DLHub
[“beam off image”, …]
model/xray/batch_predict
Automate and outsource:
(4) Data transformation and analysis
▪ Where are the model and trained weights?
▪ How do I run the model on my data?
▪ Should I run the model on my data?
▪ How can I retrain the model on new data?
https://doi.org/10.1109/NYSDS.2017.8085045](https://image.slidesharecdn.com/foster-sri-june-2018-180612125408/85/Data-Automation-at-Light-Sources-27-320.jpg)
Ad
Recommended
Data automation 101
Data automation 101Yosua Michael Maranatha The document discusses data automation practices at Kumparan, focusing on the transition to data warehouses and ETL systems to improve data processing efficiency. Key tools like Airflow and BigQuery are utilized to automate data extraction, transformation, and loading, reducing human error and report delays. Additionally, derived automation systems are implemented to create and manage derived tables, enhancing data visualization and accessibility for analysts.
Learning Systems for Science
Learning Systems for ScienceIan Foster Deep learning is finding applications in science such as predicting material properties. DLHub is being developed to facilitate sharing of deep learning models, data, and code for science. It will collect, publish, serve, and enable retraining of models on new data. This will help address challenges of applying deep learning to science like accessing relevant resources and integrating models into workflows. The goal is to deliver deep learning capabilities to thousands of scientists through software for managing data, models and workflows.
Coding the Continuum
Coding the ContinuumIan Foster The document discusses the evolution of computing and networking, emphasizing the concept of the 'computing continuum' where computing resources are distributed across various types of appliances and locations. It highlights technological advancements, such as 5G and cloud computing, and presents models for optimizing performance based on speed, location, and resource utilization. The narrative ties together concepts of latency, computational efficiency, and cost-management in modern computing environments.
Research Automation for Data-Driven Discovery
Research Automation for Data-Driven DiscoveryGlobus This document discusses research automation and data-driven discovery. It notes that data volumes are growing much faster than computational power, creating a productivity crisis in research. However, most labs have limited resources to handle these large data volumes. The document proposes applying lessons from industry to create cloud-based science services with standardized APIs that can automate and outsource common tasks like data transfer, sharing, publishing, and searching. This would help scientists focus on their core research instead of computational infrastructure. Examples of existing services from Globus and the Materials Data Facility are presented. The goal is to establish robust, scalable, and persistent cloud platforms to help address the challenges of data-driven scientific discovery.
Materials Data Facility: Streamlined and automated data sharing, discovery, ...
Materials Data Facility: Streamlined and automated data sharing, discovery, ...Ian Foster The document details advancements in the Materials Data Facility (MDF) focused on streamlining and automating data sharing, discovery, access, and analysis for materials science. Key highlights include the ingestion of significant datasets, the deployment of APIs for community development, and the establishment of unified search across multiple data sources. Future steps involve enhancing data indexing and search capabilities while continuing to engage with the materials science community and educational institutions.
Data Tribology: Overcoming Data Friction with Cloud Automation
Data Tribology: Overcoming Data Friction with Cloud AutomationIan Foster The document discusses the concept of data tribology, which seeks to reduce 'data friction' through cloud automation and services. It highlights the various tasks that can be automated in research data handling, such as authentication, data transfer, sharing, and processing. The Globus platform is presented as a successful example of implementing these principles to enhance data accessibility and usability in research environments.
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationIan Foster The document discusses the concept of the Discovery Cloud, emphasizing the need for automation and outsourcing to enhance scientific discovery through improved data management and computational capabilities. It explores tools like computational grids and cloud computing, exemplifying their use in various scientific fields to facilitate data sharing, analysis, and workflow automation. Additionally, the paper highlights initiatives such as the Globus platform, which aids in managing and transferring large datasets while promoting collaboration among researchers.
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...Ian Foster The document discusses advances in online data analysis and reduction for extreme scale computing, highlighting the integration of automation to enhance the research data lifecycle. Emphasizing the importance of real-time analysis, it outlines the challenges faced in handling large datasets and proposes a co-optimized approach that integrates simulation, analysis, and data reduction. It also introduces the Codar initiative, which aims to develop infrastructure for effective online data services and addresses key research questions related to data management and error impact.
NIH Data Commons Architecture Ideas
NIH Data Commons Architecture IdeasIan Foster The document outlines the implementation of a commons platform for promoting continuous fairness in data sharing, emphasizing the need for data to be findable, accessible, interoperable, and reusable. It highlights essential components like Globus Auth for secure authentication, Globus Connect Server for data transfer, and the use of standardized identifiers and formats for data management and sharing. The infrastructure aims to support scalability, interoperability, and user-friendly experiences for diverse collaboration and virtual cohort creation.
Science cloud foster june 2013
Science cloud foster june 2013Kirill Osipov This document discusses building a "discovery cloud" to accelerate scientific discovery through on-demand computing. It proposes identifying time-consuming research activities that can be automated and outsourced as software-as-a-service. This would achieve economies of scale through leveraging infrastructure-as-a-service. The goal is to create a great user experience for scientific data management similar to consumer services like Dropbox. It also discusses integrating services like Globus for data management and Galaxy for analysis to provide flexible and scalable genomics analysis.
Science as a Service: How On-Demand Computing can Accelerate Discovery
Science as a Service: How On-Demand Computing can Accelerate DiscoveryIan Foster This document discusses the potential for "Science as a Service" by leveraging on-demand computing capabilities. It notes that most research labs have limited resources, so automation and outsourcing are needed to apply sophisticated methods to larger datasets. The author proposes building a "discovery cloud" by identifying time-consuming research activities that can be automated and delivered as software/platform/infrastructure as a service. This would help accelerate scientific discovery. Globus is highlighted as an example of a platform providing data management services using a software as a service model.
Cloud com foster december 2010
Cloud com foster december 2010Ian Foster The document discusses the significance of cloud computing for scientific processes, emphasizing the time-consuming nature of various tasks like data management and analysis. It outlines the potential of Software as a Service (SaaS) and other cloud services to streamline these processes in both science and business. Additionally, it introduces Globus Online, a tool for efficient file transfer and data management in grid computing environments.
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneIan Foster The document discusses the challenges in data-intensive science and the potential of outsourcing mundane research tasks through Software-as-a-Service (SaaS) solutions. It highlights how automation can accelerate discovery by reducing the time researchers spend on administrative tasks and data handling. The Globus Online platform is presented as an example of leveraging cloud services to streamline data movement and enhance research efficiency.
GlobusWorld 2015
GlobusWorld 2015Tanu Malik The document proposes a reproducible framework powered by Globus to help researchers share and reproduce scientific models and simulations. It describes challenges with current methods for sharing work, like dependencies or configuration issues, that prevent easy reproducibility. The framework aims to [1] capture scientific activities, code, data and environments; [2] preserve them as standardized packages called "SciUnits"; [3] share and distribute SciUnits so others can [4] re-execute and re-analyze the work without installation or configuration problems. Key components are outlined for establishing this framework to support reproducibility across different scientific domains.
Health & Status Monitoring (2010-v8)
Health & Status Monitoring (2010-v8)Robert Grossman This document describes two case studies of health and status monitoring systems used to monitor large, complex datasets and detect anomalies. In the first case study, a system monitored thousands of servers in a data center and detected dead or slow nodes that reduced overall performance. The second case study monitored billions of payment card transactions and developed over 15,500 statistical models to detect data quality issues and interoperability problems, improving approval rates and saving millions. Both cases highlighted the importance of executive support, dashboards, governance programs, and developing numerous statistical models tailored to different data segments.
Data Science at Scale by Sarah Guido
Data Science at Scale by Sarah GuidoSpark Summit This document summarizes Sarah Guido's talk on using Apache Spark for data science at Bitly. She discusses how Bitly uses Spark to extract, explore, and model subsets of their data including decoding Bitly links, performing topic modeling using LDA, and trend detection. While Spark provides performance benefits over MapReduce for these tasks, she notes issues with Hadoop servers, JVM, and lack of documentation that must be addressed for full production usage at Bitly.
Benchmarking Cloud-based Tagging Services
Benchmarking Cloud-based Tagging ServicesTanu Malik This document presents a framework for benchmarking cloud-based tagging services. It proposes a tagging data model and generates workloads using a Markov chain process. Experiments were conducted using OLTP-Bench to evaluate the performance of MySQL, Postgres and SQL Server under different tagging workloads and schemas on Amazon EC2. The results show that for sparse datasets, a vertically partitioned schema outperforms horizontal schemas, and the best database depends on the sparseness of the data. The benchmark aims to help determine the most efficient cloud platform for supporting dynamic, sparse tagging data.
re:Invent 2013-foster-madduri
re:Invent 2013-foster-madduriRavi Madduri This document summarizes a presentation about providing next-generation sequencing analysis capabilities using Globus Genomics. It outlines challenges with current manual approaches to sequencing data analysis, including difficulties moving large datasets between locations and maintaining complex analysis scripts. The presentation introduces Globus Genomics, which uses Globus data transfer services integrated with Galaxy to provide a workflow-based system for sequencing analysis without requiring local installation or configuration. Key benefits include on-demand access to scalable cloud resources, ability to easily modify and reuse analysis workflows, and integration with data sources. The system aims to accelerate genomic research by automating and simplifying analysis.
Big data visualization frameworks and applications at Kitware
Big data visualization frameworks and applications at Kitwarebigdataviz_bay The document discusses Kitware, Inc., a privately held company focused on big data visualization frameworks and applications, highlighting its inception, growth, and collaborations with numerous academic institutions, government agencies, and commercial companies. It details Kitware's core technologies, including the Visualization Toolkit (VTK) and ParaView, and addresses the challenges and strategies associated with handling big data across various domains. The presentation concludes with a focus on community-building efforts and the importance of shared frameworks for effective data management and analysis.
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
Share and analyze geonomic data at scale by Andy Petrella and Xavier TordoirSpark Summit This document discusses analyzing genomic data at scale using distributed machine learning tools like Spark, ADAM, and the Spark Notebook. It outlines challenges with genomic data like its large size and need for distributed teams in research projects. The document proposes sharing data, processes, and results more efficiently through tools like Shar3 that can streamline the data analysis lifecycle and allow distributed collaboration on genomic research projects and datasets.
Big Data Visualization
Big Data Visualizationbigdataviz_bay The document discusses big data visualization challenges, focusing on issues of volume, velocity, variety, and veracity. It presents advanced visual techniques for extreme-scale scientific simulations and dynamic network visualization, detailing methods for improving graph layouts and visualizing time-varying networks. Ongoing projects include applications in diverse fields such as biology, transportation, and cybersecurity, emphasizing the need for in situ visualization and data reduction methods.
So Long Computer Overlords
So Long Computer OverlordsIan Foster The document discusses the challenges and opportunities in research data management, emphasizing the need for scalable, cloud-based solutions to handle the increasing volume and complexity of data across various scientific fields. It highlights the inefficiencies of current practices and proposes a shift towards Research IT as a Service to enhance accessibility, streamline processes, and reduce costs. Through leveraging Software-as-a-Service, the goal is to democratize access to powerful research tools, ultimately accelerating discovery and innovation.
An Overview of Bionimbus (March 2010)
An Overview of Bionimbus (March 2010)Robert Grossman This document provides an overview of Bionimbus and the Open Cloud Consortium (OCC). Bionimbus is an open source cloud for biomedical research that provides services like elastic computing, databases, data transport and analysis pipelines. The OCC operates open clouds and develops standards to bridge private and public clouds. It runs an Open Cloud Testbed and is working to build an Open Science Data Cloud. The OCC aims to develop interoperable cloud architectures and operate infrastructure at data center scale to support open science.
Computing Outside The Box June 2009
Computing Outside The Box June 2009Ian Foster The document discusses the evolution and impact of outsourcing computing through various methods like cloud computing, grid computing, and software as a service. It highlights how these approaches enable more powerful, dynamic, and collaborative applications, revolutionizing both business and scientific computing. Additionally, it explores the architectures that support these new applications and the benefits of distributed computing in various fields such as biomedical informatics and weather modeling.
OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3Robert Grossman The document provides an overview of the Open Cloud Consortium (OCC), emphasizing its member-driven structure, focus on developing standards, benchmarks, and reference implementations in cloud computing. It discusses various types of clouds and use cases, particularly in managing large data applications, alongside ongoing working groups aimed at improving interoperability and performance measurement. The OCC also sponsors events and welcomes new members to participate in its initiatives.
Genomic Scale Big Data Pipelines
Genomic Scale Big Data PipelinesLynn Langit The document discusses the advancements in genomic-scale data pipelines and their implications for bioinformatics, focusing on cloud-based solutions like Apache Hadoop and Apache Spark for efficient data analysis and machine learning. It presents methodologies for genetic research, including identifying disease-related variants and genome editing technologies. Additionally, the document highlights the need for scalable and parallelized processes to handle increasing genomic data demands in research.
My Other Computer is a Data Center: The Sector Perspective on Big Data
My Other Computer is a Data Center: The Sector Perspective on Big DataRobert Grossman The document discusses the distinctions between small and large data, emphasizing the scale and computational resources of data center computing. It provides insights into big data frameworks like Google's MapReduce, Bigtable, and various NoSQL systems, while introducing the Sector architecture for simplifying data-intensive computing. Additionally, it highlights several applications of the Sector platform for handling large datasets in fields such as astronomy and genomics.
Open Science Data Cloud - CCA 11
Open Science Data Cloud - CCA 11Robert Grossman The Open Science Data Cloud is a hosted, managed, distributed facility that allows scientists to manage and archive medium and large datasets, provide computational resources to analyze the data, and share the data with colleagues and the public. It currently consists of 6 racks, 212 nodes, 1568 cores and 0.9 PB of storage across 4 locations with 10G networks. Projects using the Open Science Data Cloud include Bionimbus for hosting genomics data and Matsu 2 for providing flood data to disaster response teams. The goal is to build it out over the next 10 years into a small data center for science that can preserve data like libraries and museums preserve collections.
Gladier: The Globus Architecture for Data Intensive Experimental Research (AP...
Gladier: The Globus Architecture for Data Intensive Experimental Research (AP...Globus Gladier is a framework that combines instruments, storage, and compute using loosely coupled services to automate and simplify data-intensive experimental research workflows. It uses Globus services for remote data management and funcX for scalable remote execution. Gladier provides tools and a client to define workflows that span multiple facilities and orchestrate the execution of actions like data transfers, analysis, and publication. Example deployments at facilities like the Advanced Photon Source automate workflows for experiments like XPCS and crystallography.
Taming Big Data!
Taming Big Data!Ian Foster The document discusses the challenges and methodologies for managing and analyzing big data, highlighting the necessity for efficient data flow, automated management, and the development of discovery engines. It covers aspects such as file transfer protocols, network infrastructure, and the importance of collaboration in scientific research. The narrative emphasizes the iterative process of data discovery in the context of advancing technology and complex data environments.
More Related Content
What's hot (20)
NIH Data Commons Architecture Ideas
NIH Data Commons Architecture IdeasIan Foster The document outlines the implementation of a commons platform for promoting continuous fairness in data sharing, emphasizing the need for data to be findable, accessible, interoperable, and reusable. It highlights essential components like Globus Auth for secure authentication, Globus Connect Server for data transfer, and the use of standardized identifiers and formats for data management and sharing. The infrastructure aims to support scalability, interoperability, and user-friendly experiences for diverse collaboration and virtual cohort creation.
Science cloud foster june 2013
Science cloud foster june 2013Kirill Osipov This document discusses building a "discovery cloud" to accelerate scientific discovery through on-demand computing. It proposes identifying time-consuming research activities that can be automated and outsourced as software-as-a-service. This would achieve economies of scale through leveraging infrastructure-as-a-service. The goal is to create a great user experience for scientific data management similar to consumer services like Dropbox. It also discusses integrating services like Globus for data management and Galaxy for analysis to provide flexible and scalable genomics analysis.
Science as a Service: How On-Demand Computing can Accelerate Discovery
Science as a Service: How On-Demand Computing can Accelerate DiscoveryIan Foster This document discusses the potential for "Science as a Service" by leveraging on-demand computing capabilities. It notes that most research labs have limited resources, so automation and outsourcing are needed to apply sophisticated methods to larger datasets. The author proposes building a "discovery cloud" by identifying time-consuming research activities that can be automated and delivered as software/platform/infrastructure as a service. This would help accelerate scientific discovery. Globus is highlighted as an example of a platform providing data management services using a software as a service model.
Cloud com foster december 2010
Cloud com foster december 2010Ian Foster The document discusses the significance of cloud computing for scientific processes, emphasizing the time-consuming nature of various tasks like data management and analysis. It outlines the potential of Software as a Service (SaaS) and other cloud services to streamline these processes in both science and business. Additionally, it introduces Globus Online, a tool for efficient file transfer and data management in grid computing environments.
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneIan Foster The document discusses the challenges in data-intensive science and the potential of outsourcing mundane research tasks through Software-as-a-Service (SaaS) solutions. It highlights how automation can accelerate discovery by reducing the time researchers spend on administrative tasks and data handling. The Globus Online platform is presented as an example of leveraging cloud services to streamline data movement and enhance research efficiency.
GlobusWorld 2015
GlobusWorld 2015Tanu Malik The document proposes a reproducible framework powered by Globus to help researchers share and reproduce scientific models and simulations. It describes challenges with current methods for sharing work, like dependencies or configuration issues, that prevent easy reproducibility. The framework aims to [1] capture scientific activities, code, data and environments; [2] preserve them as standardized packages called "SciUnits"; [3] share and distribute SciUnits so others can [4] re-execute and re-analyze the work without installation or configuration problems. Key components are outlined for establishing this framework to support reproducibility across different scientific domains.
Health & Status Monitoring (2010-v8)
Health & Status Monitoring (2010-v8)Robert Grossman This document describes two case studies of health and status monitoring systems used to monitor large, complex datasets and detect anomalies. In the first case study, a system monitored thousands of servers in a data center and detected dead or slow nodes that reduced overall performance. The second case study monitored billions of payment card transactions and developed over 15,500 statistical models to detect data quality issues and interoperability problems, improving approval rates and saving millions. Both cases highlighted the importance of executive support, dashboards, governance programs, and developing numerous statistical models tailored to different data segments.
Data Science at Scale by Sarah Guido
Data Science at Scale by Sarah GuidoSpark Summit This document summarizes Sarah Guido's talk on using Apache Spark for data science at Bitly. She discusses how Bitly uses Spark to extract, explore, and model subsets of their data including decoding Bitly links, performing topic modeling using LDA, and trend detection. While Spark provides performance benefits over MapReduce for these tasks, she notes issues with Hadoop servers, JVM, and lack of documentation that must be addressed for full production usage at Bitly.
Benchmarking Cloud-based Tagging Services
Benchmarking Cloud-based Tagging ServicesTanu Malik This document presents a framework for benchmarking cloud-based tagging services. It proposes a tagging data model and generates workloads using a Markov chain process. Experiments were conducted using OLTP-Bench to evaluate the performance of MySQL, Postgres and SQL Server under different tagging workloads and schemas on Amazon EC2. The results show that for sparse datasets, a vertically partitioned schema outperforms horizontal schemas, and the best database depends on the sparseness of the data. The benchmark aims to help determine the most efficient cloud platform for supporting dynamic, sparse tagging data.
re:Invent 2013-foster-madduri
re:Invent 2013-foster-madduriRavi Madduri This document summarizes a presentation about providing next-generation sequencing analysis capabilities using Globus Genomics. It outlines challenges with current manual approaches to sequencing data analysis, including difficulties moving large datasets between locations and maintaining complex analysis scripts. The presentation introduces Globus Genomics, which uses Globus data transfer services integrated with Galaxy to provide a workflow-based system for sequencing analysis without requiring local installation or configuration. Key benefits include on-demand access to scalable cloud resources, ability to easily modify and reuse analysis workflows, and integration with data sources. The system aims to accelerate genomic research by automating and simplifying analysis.
Big data visualization frameworks and applications at Kitware
Big data visualization frameworks and applications at Kitwarebigdataviz_bay The document discusses Kitware, Inc., a privately held company focused on big data visualization frameworks and applications, highlighting its inception, growth, and collaborations with numerous academic institutions, government agencies, and commercial companies. It details Kitware's core technologies, including the Visualization Toolkit (VTK) and ParaView, and addresses the challenges and strategies associated with handling big data across various domains. The presentation concludes with a focus on community-building efforts and the importance of shared frameworks for effective data management and analysis.
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
Share and analyze geonomic data at scale by Andy Petrella and Xavier TordoirSpark Summit This document discusses analyzing genomic data at scale using distributed machine learning tools like Spark, ADAM, and the Spark Notebook. It outlines challenges with genomic data like its large size and need for distributed teams in research projects. The document proposes sharing data, processes, and results more efficiently through tools like Shar3 that can streamline the data analysis lifecycle and allow distributed collaboration on genomic research projects and datasets.
Big Data Visualization
Big Data Visualizationbigdataviz_bay The document discusses big data visualization challenges, focusing on issues of volume, velocity, variety, and veracity. It presents advanced visual techniques for extreme-scale scientific simulations and dynamic network visualization, detailing methods for improving graph layouts and visualizing time-varying networks. Ongoing projects include applications in diverse fields such as biology, transportation, and cybersecurity, emphasizing the need for in situ visualization and data reduction methods.
So Long Computer Overlords
So Long Computer OverlordsIan Foster The document discusses the challenges and opportunities in research data management, emphasizing the need for scalable, cloud-based solutions to handle the increasing volume and complexity of data across various scientific fields. It highlights the inefficiencies of current practices and proposes a shift towards Research IT as a Service to enhance accessibility, streamline processes, and reduce costs. Through leveraging Software-as-a-Service, the goal is to democratize access to powerful research tools, ultimately accelerating discovery and innovation.
An Overview of Bionimbus (March 2010)
An Overview of Bionimbus (March 2010)Robert Grossman This document provides an overview of Bionimbus and the Open Cloud Consortium (OCC). Bionimbus is an open source cloud for biomedical research that provides services like elastic computing, databases, data transport and analysis pipelines. The OCC operates open clouds and develops standards to bridge private and public clouds. It runs an Open Cloud Testbed and is working to build an Open Science Data Cloud. The OCC aims to develop interoperable cloud architectures and operate infrastructure at data center scale to support open science.
Computing Outside The Box June 2009
Computing Outside The Box June 2009Ian Foster The document discusses the evolution and impact of outsourcing computing through various methods like cloud computing, grid computing, and software as a service. It highlights how these approaches enable more powerful, dynamic, and collaborative applications, revolutionizing both business and scientific computing. Additionally, it explores the architectures that support these new applications and the benefits of distributed computing in various fields such as biomedical informatics and weather modeling.
OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3Robert Grossman The document provides an overview of the Open Cloud Consortium (OCC), emphasizing its member-driven structure, focus on developing standards, benchmarks, and reference implementations in cloud computing. It discusses various types of clouds and use cases, particularly in managing large data applications, alongside ongoing working groups aimed at improving interoperability and performance measurement. The OCC also sponsors events and welcomes new members to participate in its initiatives.
Genomic Scale Big Data Pipelines
Genomic Scale Big Data PipelinesLynn Langit The document discusses the advancements in genomic-scale data pipelines and their implications for bioinformatics, focusing on cloud-based solutions like Apache Hadoop and Apache Spark for efficient data analysis and machine learning. It presents methodologies for genetic research, including identifying disease-related variants and genome editing technologies. Additionally, the document highlights the need for scalable and parallelized processes to handle increasing genomic data demands in research.
My Other Computer is a Data Center: The Sector Perspective on Big Data
My Other Computer is a Data Center: The Sector Perspective on Big DataRobert Grossman The document discusses the distinctions between small and large data, emphasizing the scale and computational resources of data center computing. It provides insights into big data frameworks like Google's MapReduce, Bigtable, and various NoSQL systems, while introducing the Sector architecture for simplifying data-intensive computing. Additionally, it highlights several applications of the Sector platform for handling large datasets in fields such as astronomy and genomics.
Open Science Data Cloud - CCA 11
Open Science Data Cloud - CCA 11Robert Grossman The Open Science Data Cloud is a hosted, managed, distributed facility that allows scientists to manage and archive medium and large datasets, provide computational resources to analyze the data, and share the data with colleagues and the public. It currently consists of 6 racks, 212 nodes, 1568 cores and 0.9 PB of storage across 4 locations with 10G networks. Projects using the Open Science Data Cloud include Bionimbus for hosting genomics data and Matsu 2 for providing flood data to disaster response teams. The goal is to build it out over the next 10 years into a small data center for science that can preserve data like libraries and museums preserve collections.
Similar to Data Automation at Light Sources (20)
Gladier: The Globus Architecture for Data Intensive Experimental Research (AP...
Gladier: The Globus Architecture for Data Intensive Experimental Research (AP...Globus Gladier is a framework that combines instruments, storage, and compute using loosely coupled services to automate and simplify data-intensive experimental research workflows. It uses Globus services for remote data management and funcX for scalable remote execution. Gladier provides tools and a client to define workflows that span multiple facilities and orchestrate the execution of actions like data transfers, analysis, and publication. Example deployments at facilities like the Advanced Photon Source automate workflows for experiments like XPCS and crystallography.
Taming Big Data!
Taming Big Data!Ian Foster The document discusses the challenges and methodologies for managing and analyzing big data, highlighting the necessity for efficient data flow, automated management, and the development of discovery engines. It covers aspects such as file transfer protocols, network infrastructure, and the importance of collaboration in scientific research. The narrative emphasizes the iterative process of data discovery in the context of advancing technology and complex data environments.
Working with Instrument Data (GlobusWorld Tour - UMich)
Working with Instrument Data (GlobusWorld Tour - UMich)Globus This document discusses using Globus to automate the management and analysis of large scientific instrument data. It provides examples of challenges with managing large datasets from the Event Horizon Telescope and applying Globus services and automation to help address these challenges. Specific use cases discussed include building connectomes from microscopy data and applying deep learning to flag bad scanning electron microscope images. The document emphasizes that automation needs transparency, results need to be easily findable, and leveraging specialized services can help.
Deep Learning on Apache Spark at CERN’s Large Hadron Collider with Intel Tech...
Deep Learning on Apache Spark at CERN’s Large Hadron Collider with Intel Tech...Databricks This document discusses the application of deep learning and Apache Spark technologies at CERN's Large Hadron Collider (LHC) to efficiently filter and analyze the massive amounts of data produced during particle collisions. It highlights the technical infrastructure, data storage solutions, and analytics pipelines developed to handle the challenges of processing up to a petabyte of data per second and aims to improve event selection using machine learning. Overall, the integration of Spark, analytics zoo, and deep learning tools is shown to enhance the efficiency of real-time data processing in particle physics research.
Big data at experimental facilities
Big data at experimental facilitiesIan Foster 1) Scientists at the Advanced Photon Source use the Argonne Leadership Computing Facility for data reconstruction and analysis from experimental facilities in real-time or near real-time. This provides feedback during experiments.
2) Using the Swift parallel scripting language and ALCF supercomputers like Mira, scientists can process terabytes of data from experiments in minutes rather than hours or days. This enables errors to be detected and addressed during experiments.
3) Key applications discussed include near-field high-energy X-ray diffraction microscopy, X-ray nano/microtomography, and determining crystal structures from diffuse scattering images through simulation and optimization. The workflows developed provide significant time savings and improved experimental outcomes.
Deep Learning Pipelines for High Energy Physics using Apache Spark with Distr...
Deep Learning Pipelines for High Energy Physics using Apache Spark with Distr...Databricks Luca Canali presents a deep learning pipeline at CERN for high energy physics data processing, utilizing Apache Spark and TensorFlow to address key challenges in event filtering for particle collisions at the LHC. The pipeline aims to improve the selection of interesting events while reducing false positives and operational costs. Key findings emphasize the importance of efficient data preparation and scalable training techniques in optimizing performance.
Linking Scientific Instruments and Computation
Linking Scientific Instruments and ComputationIan Foster The document discusses advancements in scientific instruments and computational techniques, emphasizing the need for modular and extensible systems to handle the increasing complexity and volume of data generated by new sensors. It highlights the development of flows for tasks such as data collection, analysis, and visualization, which have significantly accelerated processes like protein structure determination. Additionally, the document calls for collaboration to enhance these methods and expand their application across various scientific fields.
Big Process for Big Data @ NASA
Big Process for Big Data @ NASAIan Foster The document discusses advancements in cyberinfrastructure aimed at enhancing research data management, emphasizing the importance of user experience and high-performance infrastructure. It highlights the development of tools like Globus Online for seamless data transfer and management, and introduces the concept of a 'Science DMZ' to optimize network performance for scientific workflows. Overall, it outlines a vision for a 21st-century research ecosystem that makes sophisticated data handling accessible and affordable for a broader audience.
Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Ian Foster This document discusses the development of data management services to help researchers more easily collect, move, share and analyze big data. It describes the Globus data management platform, which allows researchers to transfer large datasets between locations and share them with collaborators. The author discusses plans to expand Globus capabilities to support additional research workflows, and to develop these services sustainably through a software-as-a-service model. The goal is to automate and outsource common data tasks in a way that provides researchers with a seamless experience for managing their research data.
Globus Integrations (CHPC 2019 - South Africa)
Globus Integrations (CHPC 2019 - South Africa)Globus This document discusses enabling large-scale data-intensive science through integrating JupyterHub with Globus services. It describes how JupyterHub can be secured using Globus authentication to provide users tokens for remote Globus services from within notebooks. Examples shown include downloading data from Globus into a notebook, analyzing the data, and sharing results to an endpoint. The document also discusses extending this approach to more fully automate neuroscience data workflows using additional Globus services like Search, Transfer, and identity management.
Rpi talk foster september 2011
Rpi talk foster september 2011Ian Foster The document discusses the challenges faced by small and medium-scale scientific projects in managing the increasing volume and complexity of data. It advocates for the adoption of Software as a Service (SaaS) to enhance research IT services, streamline data movement, and reduce costs through economies of scale. The use of cloud-based solutions, such as Globus Online, is proposed to improve efficiency in data management and facilitate collaboration among researchers.
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceinside-BigData.com NERSC serves as the primary high-performance computing and data facility for the U.S. Department of Energy, supporting extensive research in physical sciences across various domains including biology and energy. The document discusses the evolution of NERSC's systems, emphasizing upcoming capabilities, real-time data analysis, and the integration of machine learning in scientific research. It highlights the critical role of supercomputing in advancing experimental science by enabling simulations, data analysis, and innovative scientific discoveries.
Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...
Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...Ian Foster The Advanced Photon Source (APS) at Argonne National Laboratory produces intense beams of x-rays for scientific research. Experimental data from the APS is growing dramatically due to improved detectors and a planned upgrade. This is creating data and computation challenges across the entire experimental process. Efforts are underway to accelerate the experimental feedback loop through automated data analysis, optimized data streaming, and computer-steered experiments to minimize data collection. The goal is to enable real-time insights and knowledge-driven experiments.
Grid is Dead ? Nimrod on the Cloud
Grid is Dead ? Nimrod on the CloudAdianto Wibisono This document discusses using hybrid cloud and grid infrastructure for high-throughput computational science. It provides an overview of the Nimrod toolkit, which supports parameter sweeps, optimization, and workflows across distributed resources. A recent experiment used Nimrod to complete jobs faster on grid resources than Amazon EC2. It also outlines a potential strawman project called GEMAP to enable grid-enabled microscopy across the Pacific using remote microscopes, compute clusters, storage, and visualization portals.
GlobusWorld 2020 Keynote
GlobusWorld 2020 KeynoteGlobus GlobusWorld 2020 highlights the mission of enhancing data-driven science efficiency through sustainable software, with a focus on collaborations for the 2019-nCoV project involving extensive databases and computing resources. It introduces advancements in Globus Connect v5, including improved user experience, credential management, and integration with various storage systems. The platform also emphasizes automation in research workflows, detailing automation capabilities for data transfer, management, and analysis across diverse computing environments.
Science Services and Science Platforms: Using the Cloud to Accelerate and Dem...
Science Services and Science Platforms: Using the Cloud to Accelerate and Dem...Ian Foster The document outlines Ian Foster's talk on utilizing cloud technology to enhance scientific discovery at the NorduGrid annual conference. It discusses the evolution of grid computing, advancements in data transfer and sharing, and the impact of cloud services on research efficiency and collaboration. Foster emphasizes the importance of a science platform to foster a discovery ecosystem that supports researchers in managing and analyzing data more effectively.
Accelerating Time to Science: Transforming Research in the Cloud
Accelerating Time to Science: Transforming Research in the CloudJamie Kinney The document outlines AWS's commitment to advancing scientific computing by providing tools and resources to accelerate research and innovation across various scientific disciplines. It discusses the services offered by AWS, such as high-performance computing and collaborative environments, and highlights notable case studies that demonstrate the impact of AWS in scientific research, including genomics and data-driven analytics. AWS also supports researchers through free usage credits and public data sets to enhance collaborative research efforts globally.
CPAC Connectome Analysis in the Cloud
CPAC Connectome Analysis in the CloudCameron Craddock This document discusses harnessing cloud computing for analyzing large neuroimaging datasets. It describes using the Amazon cloud to run the Configurable Pipeline for the Analysis of Connectomes (CPAC) on over 1,000 neuroimaging datasets. Running CPAC in the cloud provides scalable resources without hardware costs and allows processing many datasets in parallel. The results show cloud computing can significantly reduce costs and processing time compared to traditional computing for large neuroimaging projects.
Accelerating Data-driven Discovery in Energy Science
Accelerating Data-driven Discovery in Energy ScienceIan Foster The document discusses the evolution of data-driven discovery in energy science, emphasizing the development of new tools and collaborative research methodologies to enhance data access and sharing. It highlights the importance of eliminating data friction through services like Globus, which streamline data transfer, analysis, and publication processes. Additionally, it outlines the need for discovery engines to facilitate the integration of simulation and experimental data in order to address complex scientific questions.
Introduction to Galaxy and RNA-Seq
Introduction to Galaxy and RNA-SeqEnis Afgan The document provides an overview of a workshop on the Galaxy project and RNA-seq analysis, highlighting its accessibility, reproducibility, and integration of tools for data analysis in genomics. It discusses the importance of quality control, various RNA-seq techniques, and the steps involved in differential expression analysis, emphasizing the significance of replicates and data handling. The workshop aims to familiarize participants with Galaxy's capabilities using RNA-seq as a case study without delving into specific algorithmic details.
Ad
More from Ian Foster (19)
Global Services for Global Science March 2023.pptx
Global Services for Global Science March 2023.pptxIan Foster The document discusses advancements in global science services, particularly through the Globus platform, which facilitates remote access to computing resources and data management for researchers worldwide. It highlights the integration of AI and various technologies, including cloud-hosted services and secure authentication, to enhance the efficiency of research workflows while addressing challenges like data friction and the need for reliable execution. The text emphasizes the importance of a continuum-aware computing model to facilitate global collaboration and innovation in scientific research.
The Earth System Grid Federation: Origins, Current State, Evolution
The Earth System Grid Federation: Origins, Current State, EvolutionIan Foster The Earth System Grid Federation (ESGF) is a distributed network of climate data servers that archives and shares model output data used by scientists worldwide. ESGF has led data archiving for the Coupled Model Intercomparison Project (CMIP) since its inception. The ESGF Holdings have grown significantly from CMIP5 to CMIP6 and are expected to continue growing rapidly. A new ESGF2 project funded by the US Department of Energy aims to modernize ESGF to handle exabyte scale data volumes through a new architecture based on centralized Globus services, improved data discovery tools, and data proximate computing capabilities.
Better Information Faster: Programming the Continuum
Better Information Faster: Programming the ContinuumIan Foster This document discusses the computing continuum and efforts to enable better information faster through computation. It provides examples of how techniques like executing tasks closer to data sources or on specialized hardware can significantly accelerate applications. Programming models and managed services are explored for specifying and executing workloads across diverse infrastructure. There are still open questions around optimizing networks, algorithms, and applications for the computing continuum.
ESnet6 and Smart Instruments
ESnet6 and Smart InstrumentsIan Foster ESnet6 provides an ultra-fast and reliable network that enables new smart instruments for 21st century science. The network capacity has increased dramatically over time, with 2022 bandwidth being 500,000 times greater than 1993. This network allows rapid data transfer between facilities, such as replicating 7 petabytes of climate data between three labs. It also enables fast assembly and use of new instruments like high energy diffraction microscopy that can perform an analysis in 31 seconds. The integrated research infrastructure provided by Globus further supports use of remote resources and smart instruments that will drive scientific discovery.
A Global Research Data Platform: How Globus Services Enable Scientific Discovery
A Global Research Data Platform: How Globus Services Enable Scientific DiscoveryIan Foster The document discusses the Globus platform, a research data service aimed at reducing data friction and enhancing scientific discovery through cloud-based solutions. It outlines the platform's architecture, services, and lessons learned, emphasizing its hybrid free/subscription model for sustainable operation. Globus has facilitated significant data transfers and user engagement within a wide range of academic and research institutions.
Foster CRA March 2022.pptx
Foster CRA March 2022.pptxIan Foster The document discusses the concept of a national discovery cloud designed to enhance human intellect through advanced computing and data integration. It highlights the importance of creating new tools, platforms, and educational initiatives to facilitate research and innovation. Key challenges include ensuring privacy, balancing resource needs, and integrating diverse capabilities into a cohesive framework for the R&D community.
Big Data, Big Computing, AI, and Environmental Science
Big Data, Big Computing, AI, and Environmental ScienceIan Foster The document discusses three transformative technologies: exascale computing, big data, and artificial intelligence, which are significantly enhancing scientific research capabilities. It outlines the advancements in AI for science, particularly using deep learning for applications like automated cloud classification and climate modeling, while emphasizing the integration of large-scale data from satellites. Additionally, it describes the processing and classification of extensive MODIS satellite imagery to develop insights into cloud behavior and environmental phenomena.
AI at Scale for Materials and Chemistry
AI at Scale for Materials and ChemistryIan Foster The document discusses using artificial intelligence (AI) to accelerate materials innovation for clean energy applications. It outlines six elements needed for a Materials Acceleration Platform: 1) automated experimentation, 2) AI for materials discovery, 3) modular robotics for synthesis and characterization, 4) computational methods for inverse design, 5) bridging simulation length and time scales, and 6) data infrastructure. Examples of opportunities include using AI to bridge simulation scales, assist complex measurements, and enable automated materials design. The document argues that a cohesive infrastructure is needed to make effective use of AI, data, computation, and experiments for materials science.
Research Automation for Data-Driven Discovery
Research Automation for Data-Driven DiscoveryIan Foster This document discusses research automation and data-driven discovery. It notes that data volumes are growing much faster than computational power, creating a productivity crisis in research. However, most labs have limited resources to handle these large data volumes. The document proposes applying lessons from industry to create cloud-based science services with standardized APIs that can automate and outsource common tasks like data transfer, sharing, publishing, and searching. This would help scientists focus on their core research instead of computational infrastructure. Examples of existing services from Argonne National Lab and the University of Chicago Globus project are provided. The goal is to establish robust, scalable, and persistent cloud platforms to help address the challenges of data-driven scientific discovery.
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterIan Foster The document discusses the integration of Globus services with JupyterHub to enhance collaborative data science, enabling authentication, data access, movement, sharing, and search within Jupyter notebooks. It emphasizes the need for scalable, reproducible environments for data analysis through containers and incorporating parallel computing using the Parsl library. Additionally, it highlights features like remote data access, secure token management, and APIs for seamless interactions with distributed storage systems.
Team Argon Summary
Team Argon SummaryIan Foster Team Argon proposes a commons platform using reusable components to promote continuous FAIRness of data. These components include Globus Connect Server for standardized data access and transfer across storage systems, Globus Auth for authentication and authorization, and BDBags for exchange of query results and cohorts using a common manifest format. Together these aim to provide uniform, secure, and reliable access, transfer, and sharing of data while supporting identification, search, and virtualization of derived data products.
Thoughts on interoperability
Thoughts on interoperabilityIan Foster This document discusses lessons learned for achieving interoperability. It recommends having a clear purpose, starting with basic conventions like identifiers, monitoring commitments to build trust, and focusing on outward-facing interoperability through simple APIs and platforms rather than full software stacks. Observance of industry practices like authentication methods and cloud-based platforms is also advised to promote rapid development and distribution of applications.
Going Smart and Deep on Materials at ALCF
Going Smart and Deep on Materials at ALCFIan Foster The document discusses the development of an interactive and scalable data service for advanced materials science at the Argonne Leadership Computing Facility (ALCF), emphasizing the integration of advanced data analytics and machine learning to enhance simulation capabilities. It outlines methodologies for making data more accessible, automating processes, and creating reusable data objects to promote reproducibility in scientific research. Additionally, it highlights the application of time-dependent density functional theory and machine learning in predicting stopping power in materials, illustrating the significance of this service in improving computational efficiency and scientific discovery.
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...Ian Foster This document discusses computing challenges posed by rapidly increasing data scales in scientific applications and high performance computing. It introduces the concept of online data analysis and reduction as an alternative to traditional offline analysis to help address these challenges. The key messages are that dramatic changes in HPC system geography due to different growth rates of technologies are driving new application structures and computational logistics problems, presenting exciting new computer science opportunities in online data analysis and reduction.
Software Infrastructure for a National Research Platform
Software Infrastructure for a National Research PlatformIan Foster Ian Foster's talk at the 1st National Research Platform Workshop focuses on the importance of software infrastructure for a national research platform, emphasizing accessibility, performance, security, reliability, programmability, manageability, and sustainability. Globus is presented as a solution that facilitates data transfer, sharing, and publication across various storage systems, benefiting researchers and institutions. The platform supports over 10,000 active endpoints and boasts a 99.5% uptime, showcasing its capability to streamline collaboration and data management in scientific research.
Globus Auth: A Research Identity and Access Management Platform
Globus Auth: A Research Identity and Access Management PlatformIan Foster Globus Auth is an identity and access management platform designed for researchers, enabling secure and reliable file sharing, authentication, and authorization across various services and applications. It simplifies the integration of user login and OAuth2 support, allowing users to access services without needing local accounts and leveraging existing identities. With a focus on interoperability and security, Globus Auth is positioned to empower the research community by creating a cohesive ecosystem of integrated services.
Streamlined data sharing and analysis to accelerate cancer research
Streamlined data sharing and analysis to accelerate cancer researchIan Foster The document discusses how Globus serves as a data-sharing platform that accelerates cancer research by streamlining scientific data sharing and analysis. It highlights the challenges in data management, the capabilities of Globus in enhancing secure data transfers, and its role in the identification of germline mutations related to blood cancers. The emphasis is on the ease of use, secure access, and automation features of Globus, which have made it widely adopted in research environments.
Accelerating Discovery via Science Services
Accelerating Discovery via Science ServicesIan Foster The document discusses the acceleration of scientific discovery through the use of automated data management services, outsourcing, and cloud-based platforms like Globus. It highlights the importance of eliminating data friction and creating discovery engines to enhance collaboration and efficiency in scientific research. Key themes include the automation of data publication, sharing, and analysis, as well as the need for services that address the entire research lifecycle.
building global software/earthcube->sciencecloud
building global software/earthcube->sciencecloudIan Foster The document discusses the importance of building global software for scientific research, emphasizing that while automation is not fully achievable, economies of scale and network effects can enhance efficiency. It highlights the capabilities of research data management platforms, specifically mentioning Globus services and their extensive user base and file transfer capacity. Key takeaways include the necessity for software-as-service (SaaS) models in science and the need for global collaboration to ease data management challenges.
Ad
Recently uploaded (20)
TISSUE TRANSPLANTATTION and IT'S IMPORTANCE IS DISCUSSED
TISSUE TRANSPLANTATTION and IT'S IMPORTANCE IS DISCUSSEDPhoebeAkinyi1 Tissue transplant in immunology
Investigatory_project Topic:-effect of electrolysis in solar desalination .pdf
Investigatory_project Topic:-effect of electrolysis in solar desalination .pdfshubham997ku Cbse class 12. chemistry investigatory project
An Analysis Of The Pearl Short Story By John Steinbeck
An Analysis Of The Pearl Short Story By John SteinbeckBillyDarmawan3 John Steinbeck Popular Short Story
The Pearl adalah sebuah novella yang ditulis oleh John Steinbeck yang mengisahkan perjuangan seorang nelayan miskin bernama Kino dan istrinya, Juana, yang hidup sederhana di sebuah desa kecil Meksiko. Suatu hari, Kino menemukan sebuah mutiara raksasa — "the Pearl of the World" — yang diyakini akan mengubah nasib mereka.
Namun, alih-alih membawa kebahagiaan, mutiara itu justru menimbulkan keserakahan, kekerasan, dan penderitaan. Masyarakat di sekitarnya mulai menunjukkan niat jahat, dan bahkan keluarga Kino sendiri terjebak dalam konflik batin antara harapan dan kehancuran. Pada akhirnya, The Pearl menjadi simbol ironi: sesuatu yang tampak sebagai berkah berubah menjadi kutukan.
Melalui cerita ini, Steinbeck mengeksplorasi tema keserakahan, harapan, kemiskinan, dan ketimpangan sosial. Dengan gaya narasi yang lugas namun penuh makna simbolik, The Pearl menjadi refleksi tajam tentang sifat manusia dan dampak tragis dari keinginan yang tak terkendali.
lysosomes "suicide bags of cell" and hydrolytic enzymes
lysosomes "suicide bags of cell" and hydrolytic enzymeskchaturvedi070 Lysosomes are membrane-bound organelles found in most animal cells (rare in plants) that function as the digestive system of the cell. They contain hydrolytic enzymes that can break down various biomolecules.Single membrane enclosing the organelle.Contains over 50 types of digestive enzymes (hydrolases).Enzymes function best at acidic pH (~4.5–5.0), maintained by proton pumps in the lysosomal membrane.
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...ShrayasiRoy In an era where users are bombarded with endless choices, capturing attention and driving meaningful engagement isn't just about building features — it's about understanding what drives human behavior at its core. This presentation offers a deep dive into how psychological principles can inform smarter, more intuitive product decisions. It’s not just theory — it’s a hands-on blueprint for applying human-centered thinking at every stage of product development.
Grounded in behavioral science, consumer psychology, and cognitive design, this deck unpacks the key psychological drivers behind user motivation, decision-making, emotional engagement, and habit formation. From attention economics and dopamine-driven interactions to trust cues, loss aversion, and the paradox of choice — you'll see how to harness what the mind naturally does to build digital products people don’t just use… they return to, talk about, and even advocate for.
We’ll explore:
Cognitive biases that shape user perception and choices — and how to use them to your advantage without crossing ethical lines.
Emotional design principles that build trust, trigger desire, and turn features into feelings.
User behavior loops that build stickiness, deepen retention, and create emotional investments.
Psychological friction — when to reduce it for conversion, and when to add it for intentionality.
Dark patterns vs. ethical persuasion — the thin line between influence and manipulation.
A/B testing with a psychological lens — learning not just what works, but why it works.
This is more than UX research or product marketing fluff. It’s a call to build with empathy, backed by science, and sharpened by real-world product thinking. Whether you’re shaping onboarding flows, gamifying engagement, designing pricing models, or rethinking retention strategies — this deck will arm you with a new lens: the human mind.
Perfect for product managers, growth strategists, UX designers, behavioral scientists, and anyone serious about building products that resonate, retain, and inspire action.
Because at the heart of every great product… is a human.
Let’s start building for the brain — not just the screen.
EV REGENERATIVE ACCELERATION INNOVATION SUMMARY PITCH June 13, 2025.pdf
EV REGENERATIVE ACCELERATION INNOVATION SUMMARY PITCH June 13, 2025.pdfThane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER Summary: This document presents a breakthrough in electric vehicle (EV) innovation— the EV Regenerative Acceleration Innovation (EVRAI)—developed and patented by Thane Heins Energy Inc. of Ottawa, Canada, Peer-reviewed, validated and independently replicated at the University of Ottawa following performance confirmations at MIT. EVRAI now provides all EVs and hybrid vehicles with the ability to completely recharge themselves as they drive, eliminating the need for external plug-in recharging and reducing battery size requirements by up to 80%. This document outlines the new physics principles, internationally confirmed, validated replications, individual and commercial implications of this paradigm-shifting innovation in the context of sustainable mobility.
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...
How Psychology Can Power Product Decisions: A Human-Centered Blueprint- Shray...ShrayasiRoy2 In an era where users are bombarded with endless choices, capturing attention and driving meaningful engagement isn't just about building features — it's about understanding what drives human behavior at its core. This presentation offers a deep dive into how psychological principles can inform smarter, more intuitive product decisions. It’s not just theory — it’s a hands-on blueprint for applying human-centered thinking at every stage of product development.
Grounded in behavioral science, consumer psychology, and cognitive design, this deck unpacks the key psychological drivers behind user motivation, decision-making, emotional engagement, and habit formation. From attention economics and dopamine-driven interactions to trust cues, loss aversion, and the paradox of choice — you'll see how to harness what the mind naturally does to build digital products people don’t just use… they return to, talk about, and even advocate for.
We’ll explore:
Cognitive biases that shape user perception and choices — and how to use them to your advantage without crossing ethical lines.
Emotional design principles that build trust, trigger desire, and turn features into feelings.
User behavior loops that build stickiness, deepen retention, and create emotional investments.
Psychological friction — when to reduce it for conversion, and when to add it for intentionality.
Dark patterns vs. ethical persuasion — the thin line between influence and manipulation.
A/B testing with a psychological lens — learning not just what works, but why it works.
This is more than UX research or product marketing fluff. It’s a call to build with empathy, backed by science, and sharpened by real-world product thinking. Whether you’re shaping onboarding flows, gamifying engagement, designing pricing models, or rethinking retention strategies — this deck will arm you with a new lens: the human mind.
Perfect for product managers, growth strategists, UX designers, behavioral scientists, and anyone serious about building products that resonate, retain, and inspire action.
Because at the heart of every great product… is a human.
Let’s start building for the brain — not just the screen.
GBSN_ Unit 1 - Introduction to Microbiology
GBSN_ Unit 1 - Introduction to MicrobiologyAreesha Ahmad Microbiology for Nursing students - According to New PNC course curriculum - 2025
EV REGENERATIVE ACCELERATION INNOVATION SUMMARY PITCH June 13, 2025.pdf
EV REGENERATIVE ACCELERATION INNOVATION SUMMARY PITCH June 13, 2025.pdfThane Heins NOBEL PRIZE WINNING ENERGY RESEARCHER
Data Automation at Light Sources
- 1. Data Automation at Light Sources Ian Foster Argonne National Laboratory & The University of Chicago 1
- 2. Advanced Photon Source Argonne Leadership Computing Facility 1 km 5 μsec 2
- 3. von Laszewski et al., Real-time analysis, visualization, and steering of microtomography experiments at photon sources, SIAM Parallel Processing, 1999 I have been working with light sources for some time! “the data rates and compute power required ... are prodigious, easily reaching one gigabit per second and a teraflop per second [respectively]”
- 4. Ptychography: Use GPU cluster for 360x speedup, from 7 hours to 72 s [Deng, Vine, Chen, Nashed, Philips, Jin, Peterka, Ross, Jacobsen] Enable online analysis and use of fly scans Microtomography: Use 32K Mira BG/Q nodes to reduce reconstruction time from days to 2 mins [Bicer, Gursoy, Kettimuthu, De Carlo, Agrawal] Identify and correct experimental misconfiguration High-energy diffraction microscopy: 10K BG/Q nodes to reconstruct in 10 minutes [Sharma, Almer, Wozniak, Wilde, Foster] Zoom in on crack locations (switch far field near field) Coherence Brightness High Energy Micrometer porosity structure of shale samples Microstructure of a copper wire, 0.2mm diameter Work on high-speed analysis continues
- 5. We face a data crisis (and opportunity) New instrumentation means that data rates are growing much faster than Moore’s Law Neither humans nor computers can cope by using current methods We need new methods for designing experiments, managing data, analyzing data, and creating and delivering software “A knowledge-based society, connected by the Internet and powered by AI …” — Chen Chien-jen
- 8. Automate and outsource: (1) Data distribution Needs: Usable, efficient, reliable, secure, sustainable 8
- 9. Automate and outsource: (1) Data distribution Needs: Usable, efficient, reliable, secure, sustainable Outsource: (a) Petrel data store to hold data prior to/during distribution 9 Petrel online store petrel.alcf.anl.gov 94 Gbit/s Petrel—Blue Waters 2 petabytes 100 Gbps
- 10. Automate and outsource: (1) Data distribution Needs: Usable, efficient, reliable, secure, sustainable Outsource: (a) Petrel data store to hold data prior to/during distribution (b) Globus service for data transfer and sharing 10 2 petabytes 100 Gbps Globus APIs
- 11. globus.org
- 12. Automate and outsource: (1) Data distribution Needs: Usable, efficient, reliable, secure, sustainable Outsource: (a) Petrel data store to hold data prior to/during distribution (b) Globus service for data transfer and sharing Automate: (c) DMagic script uses Globus APIs to transfer data and configure permissions 12 http://dmagic.readthedocs.io Francesco de Carlo Given an experiment date: • Retrieve user info from APS scheduler • Create Globus “shared endpoint” and configure permissions • Monitor directory at beamline and use Globus to copy new files to endpoint • Email link to shared endpoint for data retrieval
- 13. Automate and outsource: (2) Publication and discovery Move to permanent location (or publish in place) Compute and record checksums Obtain and record metadata Assign persistent identifier Index for discovery 1313 2 petabytes 100 Gbps Globus APIs
- 14. Automate and outsource: (2) Publication and discovery Move to permanent location (or publish in place) Compute and record checksums Obtain and record metadata Assign persistent identifier Index for discovery 1414 Data Publication Indexing materialsdatafacility.org 2 petabytes 100 Gbps Globus APIs
- 15. Automate and outsource: (2) Publication and discovery 1515 Programmatic access (REST, Python, Jupyter) Web browse and search Data Publication Indexing materialsdatafacility.org 2 petabytes 100 Gbps Globus APIs
- 16. For each data, must apply quality control, assign identifiers, move to compute, extract features, eventually publish to public repository, … Building a different custom pipeline for every situation is impractical Automate and outsource: (3) End-to-end data pipelines
- 17. For each data, must apply quality control, assign identifiers, move to compute, extract features, eventually publish to public repository Building a different custom pipeline for every situation is impractical Automate: Trigger-action programming (“if this happens, then do that) Outsource: Cloud-based trigger-action service for reliability, scalability, ease of use, security, sustainability Automate and outsource: (3) End-to-end data pipelines
- 18. National Facility Local Storage and Compute • Quality Control • Assign Handle Beamline Instrument Globus Transfer Central Storage and Compute (CSC) • Feature extraction • Aggregate and convert format Archive Automate and outsource: (3) End-to-end pipelines with trigger-action programming
- 19. National Facility Local Storage and Compute • Quality Control • Assign Handle Beamline Instrument • Email / SMS notification Globus Transfer Central Storage and Compute (CSC) • Feature extraction • Aggregate and convert format Archive1 1 Rules • IF new files THEN run quality control scripts • IF quality is good THEN send email and transfer data to CSC Automate and outsource: (3) End-to-end pipelines with trigger-action programming
- 20. National Facility Local Storage and Compute • Quality Control • Assign Handle Beamline Instrument • Email / SMS notification Globus Transfer Central Storage and Compute (CSC) • Feature extraction • Aggregate and convert format Globus Transfer Archive • Set sharing ACLs • Set timer for publication to Materials Data Facility Data publication 1 2 1 Rules 2 • IF new files THEN run quality control scripts • IF quality is good THEN send email and transfer data to CSC • IF new files THEN run feature extraction • IF feature detected THEN transfer data to archival storage • IF time since ingest > 6 months THEN publish dataset to Materials Data Facility Automate and outsource: (3) End-to-end pipelines with trigger-action programming
- 21. Data Source Collector Storage and Compute • Capture dataset creation • Review center position APS beamline 32-ID ALCF Cooley Cluster • Generate preview and center images • Reconstruct image • Extract metadata Ingest in Globus Search Set sharing ACLs Data publication 1 2 1 Rules 2 • IF new HDF5 files THEN transfer to Cooley • IF new center_pos THEN initiate reconstruction • IF transfer complete THEN execute preview and center finding • IF results THEN return data to APS • IF reconstruction THEN transfer data to Petrel AND publish dataset ALCF Petrel Archive Visualize with Neuroglancer Another example: Mosaic tomography for neurocartography (N. Kasthuri, R. Chard, et al.)
- 22. globus.org
- 23. Automate and outsource: (4) Data transformation and analysis “beam misaligned” “…” Say you want to use a deep neural network for online identification of problems when running diffraction experiments
- 24. Automate and outsource: (4) Data transformation and analysis https://doi.org/10.1109/NYSDS.2017.8085045
- 25. Automate and outsource: (4) Data transformation and analysis ▪ Where are the model and trained weights? ▪ How do I run the model on my data? ▪ Should I run the model on my data? ▪ How can I retrain the model on new data? https://doi.org/10.1109/NYSDS.2017.8085045
- 26. DLHub [“beam off image”, …] model/xray/batch_predict Automate and outsource: (4) Data transformation and analysis ▪ Where are the model and trained weights? ▪ How do I run the model on my data? ▪ Should I run the model on my data? ▪ How can I retrain the model on new data? https://doi.org/10.1109/NYSDS.2017.8085045
- 27. DLHub [“beam off image”, …] model/xray/batch_predict Automate and outsource: (4) Data transformation and analysis ▪ Where are the model and trained weights? ▪ How do I run the model on my data? ▪ Should I run the model on my data? ▪ How can I retrain the model on new data? https://doi.org/10.1109/NYSDS.2017.8085045
- 28. Data and Learning Hub (DLHub): Overview • Collect, publish, categorize models/code/weights/data from many sources • Serve models via API to foster sharing, consumption, and access to data, training sets, and models • Automate training of models (using HPC as needed) as new data are available • Enable new science through reuse and synthesis of existing models TrainCollect Serve
- 29. DLHub: Collect, serve, train community models DLHub Collect Data 1) Register a model Train Model Register Model Model / transform containers Receive DOI Send to DLHub
- 30. DLHub Collect Data Receive predicted Properties Send compositions Call DLHub Find Model 2) Run a model Model / transform containers DLHub: Collect, serve, train community models Collect Data Receive DOI 1) Register a model Train Model Register Model Send to DLHub
- 32. 32
- 33. 33 Invoke model on data
- 34. Ben Blaiszik Steve TueckeKyle Chard Jim Pruyne Logan WardRachana Ananthakrishnan Ryan Chard Mike Papka Rick Wagner I reported on the work of many talented people Thanks also to: • Jon Almer, Francesco de Carlo, Hemant Sharma, Brian Toby, Stefan Vogt, Stephen Streiffer, Nicholas Schwarz, Doga Gursoy, and others, Advanced Photon Source • Tekin Bicer, Jonathan Gaff, Raj Kettimuthu, Justin Wozniak, and others, Argonne Computing We are grateful to our sponsors DLHub Globus IMaD Petrel Argonne Leadership Computing Facility
- 35. In summary More data demands new methods for designing experiments, managing data, analyzing data, and creating and delivering software We must automate and outsource to manage data, run pipelines, and train and run (machine learning) models I presented examples that illustrate what can be done: • High-speed storage services for data staging and distribution: Petrel • Cloud-based services for data transfer and sharing: Globus Transfer • Data publication and discovery services: Materials Data Facility • Cloud-based automation services: Globus Automate • Model and transformation services to encapsulate software: DLHub There are many opportunities, and great need, for collaboration To follow up: [email protected]