Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
3 likes1,127 views
This document discusses analyzing genomic data at scale using distributed machine learning tools like Spark, ADAM, and the Spark Notebook. It outlines challenges with genomic data like its large size and need for distributed teams in research projects. The document proposes sharing data, processes, and results more efficiently through tools like Shar3 that can streamline the data analysis lifecycle and allow distributed collaboration on genomic research projects and datasets.
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
- 1. Share and analyse genomic data at scale with Spark, Adam, Tachyon & the Spark Notebook by @DataFellas, Oct • 29th • 2015
- 2. Outline • Sharp intro to Genomics data • What are the Challenges • Distributed Machine Learning to the rescue • Projects: Distributed teams • Research: Long process • Towards Maximum Share for efficiency
- 3. Andy Petrella Maths Geospatial Distributed Computing Spark Notebook Trainer Spark/Scala Machine Learning Xavier Tordoir Physics Bioinformatics Distributed Computing Scala (& Perl) trainer Spark Machine Learning “There must be another way of doingthecredits” -- Robin Hood: Menin Tights (1993, Mel Brooks)
- 4. Analyse Genomic At Scale Spark, Adam, Spark Notebook • Sharp intro to Genomics data • What are the Challenges • Distributed Machine Learning to the rescue
- 5. What is genomics data? DNA? What makes us what we are… … a complex biochemical soup. With applications to medical diagnostics, drug response, disease mechanisms
- 6. On the production side Fast biotech progress… … can IT keep up?
- 7. On the production side Sequence {A, T, G, C} 3 billion characters (bases)
- 8. On the production side Sequence {A, T, G, C} 3 billion characters (bases) … x 30 (x 60) Massively parallel
- 10. Lots of data? 10’s millions
- 11. Lots of data! 10’s millions 1,000s 1,000,000s ...
- 12. ADAM: Spark genomics library http://www.bdgenomics.org
- 13. ADAM: Spark genomics library
- 14. ADAM: Spark genomics library
- 15. ADAM: Spark genomics library
- 16. ADAM: Spark genomics library Avro schema Parquet storage Genomics API
- 17. So what do we do with this? Study variations between populations Descriptive statistics Machine Learning (Population stratification or Supervised learning) … and share and replay!
- 18. The Spark Notebook … comes to the rescue. Spark: easy APIs Self described and consistent Easily shared (code) http://www.spark-notebook.io
- 22. So what do we do with this? … and share and replay! Code can be shared easily but we want better... How do we share data produced by the notebook? How do we publish the notebook as a service?
- 23. Share Genomic At Scale Spark, Tachyon, Mesos, Shar3 • Projects: Distributed teams • Research: Long process • Towards Maximum Share for efficiency
- 24. Projects Intrinsically involving many teams geolocally distributed in different countries or laboratories with different skills in Biology, Genetics, I.T., Medicine (, legal...)
- 25. Projects Require many types of data ranging from bio samples imagery textual archives/historical
- 26. Projects Of course Generally gather many people from several populations Note: This is very expensive and burns time as hell!
- 27. Projects 1.000 genomes (2008-2012): 200To 100.000 genomes (2013-2017): 20Po (probably more) 1.000.000 genomes (2016-2020): 0.2Eo (probably more) eQTL: mixing many sources
- 28. Projects Need proper data management between entities, yet coping with: amount of data heterogeneity of people distance between actors constraints related to data location
- 29. Projects Distributed friendly SCHEMAS + BINARY f.i. Avro
- 30. Research Research in medicine or health in general is LOOOOOOO…OOOOONG
- 31. Research Most reasons are quite obvious not have to be overlooked Lots of measures and validation Lots of control (including by Gov.) Lots of actors
- 32. Research As a matter of fact, research need to be conducted on data and to produce results But both are highly exposed to reuse, so what if we lose either of them?
- 33. Research However, we can get into troubles instantly without even losing them. What if we don’t track the processes to go from one to the other? In any scientific process: confrontation, replay and enhancement are key to move forward
- 34. This is misleading to think that sharing the code is enough. Remind: we look for data and results, not for code. The process includes the code, the context, the sources and so on, and all should be part of the data discovery/validation task Research
- 35. Assess the risk factor associated with a disease given mutations of a certain gene. More than 50 years of data collecting and modelling. Hundreds of researchers, each generation with new ideas. Replaying old processes on new data, new processes on old data Research
- 36. Share share share All these facts relate to our capacity to share our work and to collaborate. We need to share efficiently and accurately • data • process • results
- 37. Share share share The challenge resides in the workflow
- 38. Share share share “Create” Cluster Find sources (context, quality, semantic, …) Connect to sources (structure, schema/types,…) Create distributed data pipeline/Model Tune accuracy Tune performances Write results to Sinks Access Layer User Access ops data ops data sci sci ops sci ops data web ops data web ops data sci
- 39. Share share share “Create” Cluster Find sources (context, quality, semantic, …) Connect to sources (structure, schema/types,…) Create distributed data pipeline/Model Tune accuracy Tune performances Write results to Sinks Access Layer User Access ops data ops data sci sci ops sci ops data web ops data web ops data sci
- 40. Share share share Streamlining development lifecycle for better Productivity with Shar3
- 41. Share share share Analysis Production DistributionRendering Discovery Catalog Project Generator Micro Service / Binary format Schema for output Metadata
- 42. That’s all folks Thanks for listening/staying Poke us on Twitter or via http://data-fellas.guru @DataFellas @Shar3_Fellas @SparkNotebook @Xtordoir & @Noootsab Check also @TypeSafe: http://t.co/o1Bt6dQtgH