Atomate: a tool for rapid high-throughput computing and materials discovery
0 likes522 views
Atomate is a tool for automating materials simulations and high-throughput computations. It provides predefined workflows for common calculations like band structures, elastic tensors, and Raman spectra. Users can customize workflows and simulation parameters. FireWorks executes workflows on supercomputers and detects/recovers from failures. Data is stored in databases for analysis with tools like pymatgen. The goal is to make simulations easy and scalable by automating tedious steps and leveraging past work.
Atomate: a tool for rapid high-throughput computing and materials discovery
- 1. Atomate: a tool for rapid high-throughput computing and materials discovery Anubhav Jain Energy Technologies Area Lawrence Berkeley National Laboratory Berkeley, CA Ceder group meeting, April 2020 Slides (already) posted to hackingmaterials.lbl.gov
- 2. 2 “Civilization advances by extending the number of important operations which we can perform without thinking about them.” - Alfred North Whitehead
- 3. Outline 3 ① Vision of atomate ② History of atomate ③ Current implementation ④ Future of atomate ⑤ Practical things
- 4. A “black-box” view of performing a calculation 4 “something” Results! researcher What is the GGA-PBE elastic tensor of GaAs?
- 5. Unfortunately, the inside of the “black box” is usually tedious and “low-level” 5 lots of tedious, low-level work… Results! researcher What is the GGA-PBE elastic tensor of GaAs? Input file flags SLURM format how to fix ZPOTRF? q set up the structure coordinates q write input files, double-check all the flags q copy to supercomputer q submit job to queue q deal with supercomputer headaches q monitor job q fix error jobs, resubmit to queue, wait again q repeat process for subsequent calculations in workflow q parse output files to obtain results q copy and organize results, e.g., into Excel
- 6. All the low-level steps can lead to errors! Let’s take a look at two alternate universes: It is too easy to end up in the wrong timeline! 6 you have coffee copy files from previous simulation edit 5 lines run simulation, analyze data forget coffee copy files from previous simulation edit 4 lines but forget LHFCALC=F run simulation, looks fine at first, in a month you discover it was wrong 1 2 you
- 7. Today, it is difficult to learn and apply several computational procedures due to steep learning curve Because of the multiple low-level steps and subtleties that all need to be done correctly to apply computational methods, there is often a single group “expert” for each technique 7 “Alice knows how to do charged defect calculations.” “Bob is the one who can properly converge GW runs.” “Olga has all the scripts for phonon calculations.”
- 8. What would be a better way? 8 “something” Results! researcher What is the GGA-PBE elastic tensor of GaAs?
- 9. What would be a better way? 9 Results! researcher What is the GGA-PBE elastic tensor of GaAs? Workflows to run q band structure q surface energies ü elastic tensor q Raman spectrum q QH thermal expansion
- 10. Ideally the method should scale to millions of calculations 10 Results! researcher Start with all binary oxides, replace O->S, run several different properties Workflows to run ü band structure ü surface energies ü elastic tensor q Raman spectrum q QH thermal expansion q spin-orbit coupling
- 11. Atomate’s vision is to make it easy, automatic, and flexible to generate data with existing simulation packages 11 Results! researcher Run many different properties of many different materials!
- 12. Outline 12 ① Vision of atomate ② History of atomate ③ Current implementation ④ Future of atomate ⑤ Practical things
- 13. 13 The first attempt to create something like this in Ceder group was early/mid 2000s, with the “JCMC” code “relational database hell” DB design bottleneck • Java on its way out • Poor source code quality control • 134,000 lines of code • 511 classes • 54 packages
- 14. 14 Circa 2011, we solved the database problem by switching to MongoDB.We also migrated everything to Python Without going into too many details, it turned out these were better tools for us to be constructing code with. We also tightened up code quality control During this period, we created pymatgen and FireWorks which are still alive today and indeed tend to be growing. These codes were written to be general-purpose and for arbitrary end users. We also created MPWorks and used it for ~5 years. But this code was written mainly for ourselves to power the Materials Project. It was a “heavyweight” infrastructure, like a battleship. It could run a lot of calculations (good for solving big problems) but had a ton of components and prioritized comprehensiveness over ease–of–use (bad for solving small problems).
- 15. • We began the “atomate” project, which prioritized ease-of-use and ease-of-development over feature list • This turned out to be a very good idea, and atomate is definitely better than MPWorks • But, we still didn’t get everything right (more on that later) – Why is it taking so many iterations to get this code right? Because developing complex frameworks is more difficult than writing complex functions 15 Circa 2016, we decided that MPWorks needed to be redesigned to make it easier to solve small problems
- 16. • The Materials Project is powered by atomate for the last couple of years – Thus, atomate users are using the same exact tool as Materials Project • Persson, Jain, and Ong research groups use atomate for their work • Some examples of outside community usage • Members of ABINIT team (G.M. Rignanese, G. Hautier) plan to migrate all their ABINIT workflows to atomate – Exciting thing is that one can then run all this stuff without paying for anything apart from computing 16 Atomate is used in several different places and is quite well- tested now
- 17. Outline 17 ① Vision of atomate ② History of atomate ③ Current implementation ④ Future of atomate ⑤ Practical things
- 18. How can we design a simple yet powerful tool for performing simulations? • First, we’ll go through the four high-level steps in doing a theory study • In each step, we’ll talk about how software tools can help make you more productive and help you concentrate on things that truly require your attention, not mindless labor (e.g., manually resubmitting failed jobs) 18
- 19. Steps for theory & simulation 19 material generation simulation procedure calculation & execution data analysis (crystal, surface/interface, molecule, etc...) simulation parameters and sequence (workflow) job management, execution, error recovery databases, plotting, analysis, insight
- 20. Software technologies for theory / calculation 20 (automatic materials science workflows) Custodian (calculation error recovery) (materials analysis framework) Base packages Derived packages (workflow definition & execution) These are all open-source: (materials data mining)
- 21. Software stack for theory & simulation 21 material generation simulation procedure calculation & execution data analysis Custodian
- 22. Steps for theory & simulation 22 material generation simulation procedure calculation & execution data analysis (crystal, surface/interface, molecule, etc...) simulation parameters and sequence (workflow) job management, execution, error recovery databases, plotting, analysis, insight
- 23. Step 1: materials generation 23 material generation • Pymatgen can help generate models for: – crystal structures – molecules – systems (surfaces, interfaces, etc.) • Tools include: – order-disorder (shown at right) and SQS – interstitial finding – surface / slab generation – structure matching and analysis – get structures from Materials Project • Won’t cover materials generation in this presentation! Example: Order-disorder resolve partial or mixed occupancies into a fully ordered crystal structure (e.g., mixed oxide-fluoride site into separate oxygen/fluorine)
- 24. Steps for theory & simulation 24 material generation simulation procedure calculation & execution data analysis (crystal, surface/interface, molecule, etc...) simulation parameters and sequence (workflow) job management, execution, error recovery databases, plotting, analysis, insight
- 25. Step 2: simulation procedure 25 simulation procedure • Simulation procedure requires knowing: i. what sequence of calculations do I need to perform to compute my desired materials property? ii. for each calculation, what is a good set of parameters to use (e.g., functional, energy cutoff, k-point density, etc.) • Item (i) is largely handled by atomate • Item (ii) is largely handled by pymatgen, e.g. in the VaspInputSet classes and won’t be covered here – although atomate also contains some of this information • Overall, we are trying to standardize and automate simulation procedures as much as possible
- 26. Atomate contains a library of simulation procedures 26 VASP-based • band structure • spin-orbit coupling • hybrid functional calcs • elastic tensor • piezoelectric tensor • Raman spectra • NEB • GIBBS method • QH thermal expansion • AIMD • ferroelectric • surface adsorption • work functions • NMR spectra* • Bader charges* • Magnetic orderings* • SCAN functionals* Other • BoltzTraP • FEFF method • Q-Chem* *=added / major updates in past year Mathew, K. et al Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows, Comput. Mater. Sci. 139 (2017) 140–152.
- 27. Each simulation procedure in atomate is composed of multiple levels of detail / abstraction 27 Workflow – complete set of calculations to get a materials property Firework – one step in the Workflow (typically one DFT calculation) Firetask – one step in a Firework Starting with just a crystal structure, this workflow performs four calculations to get an optimized structure, optimized charge density, and band structure on two types of grids (uniform and line)
- 28. atomate allows you to leverage the prior efforts and knowledge of many researchers 28 K. Mathew J. Montoya S. Dwaraknath A. Faghaninia All past and present knowledge, from everyone in the group, everyone previously in the group, and our collaborators, about how to run calculations M. Aykol S.P. Ong B. Bocklund T. Smidt H. Tang I.H. Chu M. Horton J. Dagdalen B. Wood Z.K. Liu J. Neaton K. Persson A. Jain +
- 29. Workflow parameters can be customized at multiple levels of detail 29 1. Workflows have various high-level options 2. Fireworks also have options / flags (not shown) 3. Firetasks have most detailed number of options / flags (not shown) Example 1: “VASP input set” controls the rules that set DFT parameters (pseudopotentials, cutoffs, grid densities, etc) via pymatgen Example II: If “stability_check” is enabled, the later parts of the workflow are skipped if the structure is determined unstable to save computer time on uninteresting structures
- 30. You can build workflows from scratch or reuse components to assemble workflows Multiple workflows are built with the same components stacked together in different ways 30 These two workflows reuse almost all the same code between the two!
- 31. Atomate simulation procedures: the main ideas • Atomate allows you to start with a material (e.g., crystal structure) and get a full workflow of calculations needed to compute many types of materials properties • Atomate tries to guess sensible defaults in terms of calculation parameters, but you can always customize these parameters using: – Workflow options – Firework options – Firetask options – powerups – going back to pymatgen library 31
- 32. Steps for theory & simulation 32 material generation simulation procedure calculation & execution data analysis (crystal, surface/interface, molecule, etc...) simulation parameters and sequence (workflow) job management, execution, error recovery databases, plotting, analysis, insight
- 33. Step 3: calculation & execution 33 calculation & execution • Once you have the material and the simulation procedure (Workflow), you need to actually execute the workflow on your computing resource • This includes tasks like: – submission to calculation queues – customization of any computing-specific parameters • e.g., path to VASP executable, number of CPUs to parallelize over – recovering from failures / job resubmission – coordinating jobs across computing centers – managing location of jobs – tracking the progress of jobs • Almost all of this is handled by FireWorks (custodian is used for encoding job recover procedures) Custodian
- 34. FireWorks allows you to write your workflow once and execute (almost) anywhere 34 • Execute workflows locally or at a supercomputing center • Queue systems supported – PBS – SGE – SLURM – IBM LoadLeveler – NEWT (a REST-based API at NERSC) – Cobalt (Argonne LCF)
- 35. Dashboard with status of all jobs 35
- 36. Job provenance and automatic metadata storage 36 what machine what time what directory what was the output when was it queued when did it start running when was it completed
- 37. Detect and rerun failures • All kinds of failures can be detected and rerun – Soft failures (job quits with error code) – hard failures (computing center goes down) – human errors 37
- 38. Customize job priorities • Within workflow, or between workflows • Completely flexible and can be modified / updated whenever you want 38
- 39. Track output files remotely • Can bring up the last few lines of each of your output files – and can be combined with queries / filters 39 Now seems like a good time to bring up the last few lines of the OUTCAR of all failed jobs...
- 40. FireWorks workflow software: the main ideas • FireWorks is used to execute Workflow objects at supercomputing centers • It is very well-suited to high-throughput applications and has been used to execute millions of jobs and is well tested • There are many features and advantages to using FireWorks as your workflow manager, which has made it one of the most popular scientific workflow software in use today 40
- 41. An aside on “custodian”: instead of running VASP, have custodian run VASP on your behalf 41 • Custodian can wrap around an executable (e.g., VASP) – i.e., run custodian instead of directly running VASP • During execution, custodian will monitor output files and detect errors / problems – e.g., if ZPOTRF error detected, rerun with ISYM=0 – ever-expanding library of fixes (currently over 30 fixes for VASP!) • There is more you can do with custodian (e.g., simple workflows), but it won’t be covered here • Currently has fixes for: VASP, Q- Chem, NWChem
- 42. Steps for theory & simulation 42 material generation simulation procedure calculation & execution data analysis (crystal, surface/interface, molecule, etc...) simulation parameters and sequence (workflow) job management, execution, error recovery databases, plotting, analysis, insight
- 43. Step 4: data analysis 43 data analysis • Data analysis can involve: i. Creating databases of materials and their properties for easy searching and data retrieval ii. Conventional scientific analyses and scientific plotting (e.g., generating phase diagrams, plotting band structures) iii. Data mining methods • Item (i) is largely handled by atomate • Item (ii) is largely handled by pymatgen • Item (iii) is largely handled by matminer
- 44. Atomate – builders framework 44 “Builders” start with base collections in a database and create higher-level collections that summarize information or add metadata
- 45. pymatgen – examples of analyses 45 phase diagrams Pourbaix diagrams diffusivity from MDband structure analysis
- 46. Data analysis: the main ideas • Atomate generates databases that summarize your calculation results. Additionally The builders framework generates document collections that summarize data on a particular material from multiple calculations or from additional information. • Pymatgen is a powerful software for doing conventional scientific analyses with the computational data • Matminer, currently in pre-release, is software for doing data mining style analyses of materials 46
- 47. Doing simulations programmatically can become second nature – and very fast / automatic / scalable 47 material generation simulation procedure calculation & execution data analysis Custodian Results! researcher What is the GGA-PBE elastic tensor of GaAs? software infrastructure
- 48. Outline 48 ① Vision of atomate ② History of atomate ③ Current implementation ④ Future of atomate ⑤ Practical things
- 49. • After using atomate for ~3 years, we realize there are still things that are awkward to do • For example – Re-using components between workflows is harder than it should be (e.g., components don’t always “stack” and “connect” easily) – Workflow creator functions often use different signatures and conventions (harder to learn) – Changing certain parameters (e.g., switching to a vdW functional) can be non-intuitive • We are starting “atomate v2” to fix these things, but better to start with original atomate for now – Original atomate will still be around, and atomate v2 will be a smooth transition from atomate v1 49 Atomate v2
- 50. • Most of “atomate” (and all the various software tools) are largely programmatic, not interactive / GUI-driven • The goal of atomate was always to work up to a visual interface – Stage 1: Exploring results using a web interface rather than by database queries – Stage 2: Submitting pre-defined workflow templates using a web interface – Stage 3: Designing and submitting custom workflows using a web interface • Currently working on stage 1 50 Atomate GUI
- 51. 51 Atomate GUI prototype Done by UCB undergrad But really requires a dedicated, focused effort
- 52. Outline 52 ① Vision of atomate ② History of atomate ③ Current implementation ④ Future of atomate ⑤ Practical things
- 53. • Not going to lie, there is a learning curve and the current tools are really meant for programmers – When (if?) we get to “Stage 2” atomate GUI, this will no longer be the case • If you don’t know any programming, it will be a long road 53 How to actually get started No programming experience Python proficient pymatgen proficient atomate proficientMongoDB experience
- 54. • Papers: good general overview / vision but of no practical help • Online MP Workshop videos and tutorials: good way to get started if you have at least some knowledge, but usually not very deep. A “zero to one” situation • Code documentation: Most comprehensive way to learn, although some experience probably necessary • Help channels: Great if you already started and run into problems 54 What resources are available to learn?
- 55. Papers • Again, these are of little practical help and any specifics are often outdated 55 Ong, S. P. et al. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Comput. Mater. Sci. 68, 314–319 (2013). Jain, A. et al. FireWorks: a dynamic workflow system designed for high- throughput applications. Concurr. Comput. Pract. Exp. 22, 5037–5059 (2015). Mathew, K. et al. Atomate: A high- level interface to generate, execute, and analyze computational materials science workflows. Comput. Mater. Sci. 139, 140–152 (2017).
- 56. • Resources on http://workshop.materialsproject.org • Videos on MP channel of Youtube: – https://www.youtube.com/channel/UC6pqY-__Nu- mKkv0LMP8FJQ 56 Online MP workshop and videos
- 57. Online documentation • Online documentation is the most comprehensive writeup – www.pymatgen.org – https://materialsproject.github.io/fireworks/ – https://materialsproject.github.io/custodian/ – https://hackingmaterials.github.io/atomate/ • The online documentation includes installation, examples, tutorials, and descriptions of how to use the code • If you want to do “everything”, suggest starting with atomate and going from there 57
- 58. Help lists • A help forum for each code is at https://discuss.matsci.org 58
- 59. Questions? Comments? (a big thanks to everyone who supported and/or contributed to this software!) 59Slides (already) posted to hackingmaterials.lbl.gov