SeqsLab: a high performance genomics data analysis platform based on Apache Spark
Download as PPTX, PDF0 likes189 views
On the way to precision medicine, personal genomics is an essential topic since the genome holds the most fundamental information of individual. NGS is a comprehensive way to explore human genome, but with the intimidating amount of data and computation effort. Atgenomix SeqsLab is a high performance genomics data analysis platform aiming to close the Bio-IT gap.
![Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
def execute(cls, examples_tfrecord, ref_version, is_gpu, partition_id, extra_params, logger):
o, _ = pipe_call("nvidia-smi -L | awk '{print $2}'", True)
gpu_list= o.decode('utf-8').strip(":n").split(":n")
shuffle(gpu_list)
o, _ = pipe_call('nvidia-smi --query-gpu=memory.total --format=csv,nounits,noheader', True)
gpu_mem_mb= max([int(x) for x in o.decode('utf-8').strip().split('n')])
gpu_mem_ratio= str(int(100 / (int(gpu_mem_mb/ cv_mim_mem_mb) + 1)))
cmd = [BIN['deepvariant-call-variants'],'--execution_hardware', mode,
'--examples',examples_tfrecord,'--checkpoint', model,
'--outfile', cvo_tfrecord_path, '--percentage_gpu_memory', gpu_mem_ratio]
while True:
for gpu_idx in gpu_list:
env = os.environ.copy()
env['CUDA_VISIBLE_DEVICES'] = gpu_idx
try:
pipe_call(cmd, False, env)
eflag = True
except SeqPiperProcessError as e:
if str(e).find('CUDA_ERROR_OUT_OF_MEMORY')!= -1 or str(e).find('OOM') != -1:
continue
else:
raise e
if eflag:
break
t = random.randrange(5, 30)
time.sleep(t)
return cvo_tfrecord_path
1. Collect GPU list and shuffle it
2. Get GPU Memory and dynamically
transform to percentage
3. Specify GPU memory when
launching call_variants
4. Select a specific GPU in a round-
robin assignment
5. Sleep a few seconds if failed.
Google DeepVariant – Multiple GPU Support Issue](https://image.slidesharecdn.com/dataconf20200910f-200915015604/85/SeqsLab-a-high-performance-genomics-data-analysis-platform-based-on-Apache-Spark-40-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to SeqsLab: a high performance genomics data analysis platform based on Apache Spark (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Ad
SeqsLab: a high performance genomics data analysis platform based on Apache Spark
- 1. SeqsLab: a high-performance genomics data analysis platform based on Apache Spark Yun-Lung Li | Genomic Data Scientist | [email protected]
- 2. Road to Precision Medicine Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 3. Agenda • NGS - From DNA to Text Data • Dry Lab Challenges • Atgenomix SeqsLab • Case studies Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 4. Agenda • NGS - From DNA to Text Data • Dry Lab Challenges • Atgenomix SeqsLab • Case studies Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 5. From DNA to Insight FASTQ BAM FASTQVCF Whole Genome ~100GB Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 6. Variant Call Format (VCF) • ~5M SNP/indel Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 7. From DNA to Insight FASTQ BAM FASTQVCF Whole Genome ~100GB Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 8. Sequencing Industry Stats 2015 2020 2025 $600 Whole genomes >15,000 Illumina installed base $1,000 Whole genomes ~150 Petabases Sequence data https://s24.q4cdn.com/526396163/files/doc_presentations/2020/01/ILMN-at-JPM-13-Jan-2020-final.pdf https://www.biorxiv.org/content/10.1101/203554v1 10M+ Global samples sequenced UKBB, NIH’s All of US, Human Cell Atlas 60M+ Patients sequenced in healthcare context 1M+ Global samples sequenced 100GB per sample Whole genomes Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 9. Agenda BAM FASTQVCF Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. • NGS - From DNA to Text Data • Dry Lab Challenges • Atgenomix SeqsLab • Case studies
- 10. Dry Lab Challenges - Scalability • Data amount • Process complexity / time GATK Best Practice Workflow Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 11. Dry Lab Challenges – Evolving Open-Source Community https://assemblathon.org/post/48865310097/slides-from-an-assemblathon-2-talk https://github.com/broadinstitute/gatk/ Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 12. Dry Lab Challenges • Data scalability • Computation scalability • Evolving open-source community Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 13. Possible Solutions - Reimplementation • Edico Dragen (GATK in FPGA) • Nvidia Clara Parabricks (GATK in GPU) • Sentieon (GATK in C++) Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 14. Possible Solutions - Reimplementation • Edico Dragen (GATK in FPGA) • Nvidia Clara Parabricks (GATK in GPU) • Sentieon (GATK in C++) Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 15. Atgenomix – Scaling Out Existing Tools With Data Parallelization • Scaling with data parallelization • Run bioinformatics tools as is • On-demand Apache Spark clusters Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 16. Atgenomix SeqsLab - BIO-IT Platform in Cloud Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 17. SeqsLab - BIO-IT Platform in Cloud Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 18. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 19. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 20. Containerized SeqsLab Runtime Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 21. Azure Distributed Data Engineering Toolkit (AZTK) Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 22. User’s Workflows on On-demand Spark Clusters Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 23. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 24. BAM (Binary Alignment Map) BAM Human genome Human genome FASTQ Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. Read Mapping
- 25. Adaptive Data Parallelization – BAM Partitioning • Human genome is organized into chromosomes, so we can partition DNA data by chromosome without losing information. • Several partitioning strategies provide more partitions for better parallelization: - Centromere - Long ambiguous regions - Any customized region Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 26. • Columnar storage - eased data retrieval overhead • Binary, encoded, compressed - size reduction • Statistic metadata - pushdown predicate • Hive style partitioning BAM to SparkSQL – Avro, Parquet Bam sort Time Hardware samtools sort / index 326mins D13_v2 x1 samtools view 206mins D13_v2 x1 Bam partition 3101 Time Hardware Atgenomix Transform 5mins D13_v2 x40 Atgenomix BamSelect 8mins D13_v2 x40
- 27. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 28. Atgenomix Transform Azure Data Lake Storage Azure Data Lake Storage Avro Parquet PipeSeq – Read Mapping pipeTransform Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 29. Azure Data Lake Storage Atgenomix BamSelect Azure Data Lake Storage PipeSeq – Variant Calling PipeTransform Avro Parquet Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 30. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 31. SeqsLab - BIO-IT Platform in Cloud • Containerized user runtime + SeqsLab runtime • Data Parallelization - biological consideration • PipeSeq - bridge HDFS/Spark and bioinformatics tools • Computation resource recommendation Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 32. Computing Unit Catalog Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 33. Smart, Scalable and Simplified Computing Platform on Azure Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 34. Agenda Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. • NGS - From DNA to Text Data • Dry Lab Challenges • Atgenomix SeqsLab • Case studies
- 35. Case Study: GATK Best Practice on SeqsLab BWA-MEM AtgxTransform alignment flagstat | sorting PosBinSelect GATK4.1 HaplotypeCaller MarkDuplicate | BQSR | whatshap phasing | dbSNP annotate Runtime (min) Compute Cost (US$) 240d 240d 240d 160d 120 6.5 160d 160d 80d 80d 190 5.5 Illumina WGS 40X PE150bp GRCh38 (hs38d1, primary assembly plus decoy contigs) Parallelization Factor 708 FASTQ shards (>500k read pairs per shard) 155 BAM shards (~20m base pairs partitioned by contiguous unmasked Ns per shard) Computing Environment Azure WEST US 2 region (Azure Batch, Azure Data Lake Gen2) SeqsLab Compute Unit 80d (5 x D14v2 low-priority VMs) 160d (10 x D14v2 low-priority VMs) 240d (15 x D14v2 low-priority VMs) Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 36. Case Study: DeepVariant Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.https://www.nature.com/articles/nbt.4235.epdf
- 37. https://github.com/google/deepvariant/issues/90 • All CPU resource is occupied • Still have multiple threads Confidential - Atgenomix internal use only. © 2018 Atgenomix, Inc. Google DeepVariant – CPU/GPU Resource Limitation Issue FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 38. Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. https://github.com/google/deepvariant/blob/5e6fe205b984c6be116dcacafdfd83ce1df4d2e9/deepvariant/call_variants.py#L344 Google DeepVariant – CPU/GPU Resource Limitation Issue
- 39. Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. Issue When more GPUs are added, the performance didn’t improve. Observation 1. Low Volatile GPU-Util 2. Each process will allocate memory from all GPU cards Google DeepVariant – Multiple GPU Support Issue
- 40. Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. def execute(cls, examples_tfrecord, ref_version, is_gpu, partition_id, extra_params, logger): o, _ = pipe_call("nvidia-smi -L | awk '{print $2}'", True) gpu_list= o.decode('utf-8').strip(":n").split(":n") shuffle(gpu_list) o, _ = pipe_call('nvidia-smi --query-gpu=memory.total --format=csv,nounits,noheader', True) gpu_mem_mb= max([int(x) for x in o.decode('utf-8').strip().split('n')]) gpu_mem_ratio= str(int(100 / (int(gpu_mem_mb/ cv_mim_mem_mb) + 1))) cmd = [BIN['deepvariant-call-variants'],'--execution_hardware', mode, '--examples',examples_tfrecord,'--checkpoint', model, '--outfile', cvo_tfrecord_path, '--percentage_gpu_memory', gpu_mem_ratio] while True: for gpu_idx in gpu_list: env = os.environ.copy() env['CUDA_VISIBLE_DEVICES'] = gpu_idx try: pipe_call(cmd, False, env) eflag = True except SeqPiperProcessError as e: if str(e).find('CUDA_ERROR_OUT_OF_MEMORY')!= -1 or str(e).find('OOM') != -1: continue else: raise e if eflag: break t = random.randrange(5, 30) time.sleep(t) return cvo_tfrecord_path 1. Collect GPU list and shuffle it 2. Get GPU Memory and dynamically transform to percentage 3. Specify GPU memory when launching call_variants 4. Select a specific GPU in a round- robin assignment 5. Sleep a few seconds if failed. Google DeepVariant – Multiple GPU Support Issue
- 41. Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. 1. Each process will allocate memory from a specific GPU card 2. High Volatile GPU-Util Google DeepVariant – Multiple GPU Support Issue
- 42. DeepVariant-on-Spark Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. https://github.com/atgenomix/deepvariant-on-spark
- 43. DeepVariant-on-Spark Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. https://github.com/atgenomix/deepvariant-on-spark https://www.hindawi.com/journals/cmmm/2020/7231205/ 黃柏榕教授 長庚大學 生物醫學研究所
- 44. Process 1 WGS Cluster BWA (708 shards) 23m DS14_v2 x 20 BAM Sorting 7.6m BAM Select 8.4m NC12 x 20 DeepVariant (3K parts) 23m Execution Time Per Sample 1h2m Google DeepVariant on SeqsLab Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 45. Case Study: Population Study 96 Hours 60 Hours Illumina WGS 1475 Samples Joint Genotyping Reference Genome GRCh38 Data Amount 20TB GVCF Parallelization Factor 3101 VCF shards (~1M base pairs per shard) Pipeline Configuration GATK4 GenomicsDBImport | GenotypeGVCFs Computing Configuration 30xD14v2 (480 cores) Affymetrix TWB 69264 Sample Imputation Reference Genome GRCh38 Parallelization Factor 1891 Reference panel shards (~50K variants per shard, based on UK Biobank workflow) Reference Panels 1000Genomes (5096 haplotypes), TWBiobank (2902 haplotypes) Pipeline Configuration IMPUTE2 RefMerge | SHAPEIT4 Phasing | IMPUTE4 Computing Configuration 2xE64v4 (128 cores) | 11xD16v4 (176 cores) | 25xE32v4 (800 cores)
- 46. Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 47. Case Study: Assembly Algorithm Scaling Reference genome FASTQ Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 48. Case Study: Assembly Algorithm Scaling Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES.
- 49. Join Us to improve human health faster and better Confidential - Atgenomix internal use only. © 2020 Atgenomix, Inc.FOR RESEARCH USE ONLY. NOT FOR USE IN DIAGNOSTIC PROCEDURES. Software Engineer Bioinformatician 亞大基因科技成立於2015， 是一個擁有生物資訊和大數 據分析技術雙重優勢的軟體 研發團隊，專注致力於發展 創新的人類基因序列數據挖 掘技術及分析軟體平台，進 而幫助醫院醫師提供更好的 個人化醫療。亞大基因立足 台灣，放眼國際，矢志成為 亞洲最大的基因大數據分析 軟體公司。歡迎有志之士一 起加入！
- 50. One platform for all your genomic workloads Yun-Lung Li | Genomic Data Scientist | [email protected]