Age of Language Models in NLP
Download as PPTX, PDF0 likes221 views
This document contains an agenda for a webinar on the age of language models in NLP. The agenda includes discussions on word embeddings, sequence modeling, advanced language models like BERT and Transformers, attention mechanisms, and case studies. It also provides overviews of Tyrone systems' high-performance AI platform using NVIDIA A100 GPUs and its Tyrone Kubyts technology for revolutionizing deep learning environments.
Age of Language Models in NLP
- 1. The Age of Language Models in NLP Tuesday | 23rd June, 2020 LIVE WEBINAR Presented by
- 2. AGENDA 1 . About Tyrone World’s high performing AI platform system – A100 Get Development, Training, Inference in one Era of Modern Mixed Workloads Tyrone Kubyts™ 2. Word Embeddings How Word embeddings create a context based relationship How to create Word Embeddings 3. Sequence Modelling Introduction of Deep learning in NLP Overview on the model architecture to use 4. Advanced Language Models Overview of the Language models How are they created Transformers BERT , GP2 etc 5. NLP Attention Mechanism Overview on attention Mechanism 6. Case Studies
- 3. Tyrone Systems at a Glance
- 4. NVIDIA HGX A100 PERFORMANCE New Tensor Core for AI & HPC New Multi-instance GPU New Hardware Engines Increase in GPU interconnect bandwidth Increase in GPU memory Increase in memory bandwidth Speedup in AI performance
- 5. 54 Billion XTORS 3rd Gen Tensor cores Sparsity Acceleration Multi Instance GPU 3rd GEN NVLINK & NVSwitch
- 6. NVIDIA A100 Greatest Generational Leap – 20X Volta 54B XTOR | 826mm2 | TSMC 7N | 40GB Samsung HBM2 | 600 GB/s NVLink Peak Vs Volta FP32 TRAINING 312 TFLOPS 20X INT8 INFERENCE 1,248 TOPS 20X FP64 HPC 19.5 TFLOPS 2.5X MULTI INSTANCE GPU 7X GPUs
- 7. New tf32 tensor cores on A100 20X Higher FLOPS for AI, Zero Code Change 20X Faster than Volta FP32 | Works like FP32 for AI with Range of FP32 and Precision of FP16 No Code Change Required for End Users | Supported on PyTorch, TensorFlow and MXNet Frameworks Containers
- 8. Most flexible ai platform with MULTI-INSTANCE GPU (MIG) Optimize GPU Utilization, Expand Access to More Users with Guaranteed Quality of Service Up To 7 GPU Instances In a Single A100: Simultaneous Workload Execution With Guaranteed Quality Of Service: All MIG instances run in parallel with predictable throughput & latency Flexibility to run any type of workload on a MIG instance Right Sized GPU Allocation: Different sized MIG instances based on target workloads Amber GPU Mem GPU GPU Mem GPU GPU Mem GPU GPU Mem GPU GPU Mem GPU GPU Mem GPU GPU Mem GPU
- 9. ONE SYSTEM FOR ALL ai INFRASTRUCTURE AI Infrastructure Re-Imagined, Optimized, and Ready for Enterprise AI-at-Scale any job | any size | any node | anytime Analytics Training Inference Flexible AI infrastructure that adapts to the pace of enterprise • One universal building block for the AI data center • Uniform, consistent performance across the data center • Any workload on any node - any time • Limitless capacity planning with predictably great performance with scale
- 10. Game-changing performance for innovators 9x Mellanox ConnectX-6 200Gb/s Network Interface 450GB/sec Peak Bi-directional Bandwidth Dual 64-core AMD Rome CPUs and 1TB RAM 3.2X More Cores to Power the Most Intensive AI Jobs 8x NVIDIA A100 GPUs with 320GB Total GPU Memory 12 NVLinks/GPU 600GB/sec GPU-to-GPU Bi-directional Bandwidth 15TB Gen4 NVME SSD 4.8TB/sec Bi-directional Bandwidth 2X More than Previous Generation NVSwitch 6x NVIDIA NVSwitches 25GB/sec Peak Bandwidth 2X Faster than Gen3 NVME SSDs
- 11. 2U GPU server up to 4 NVIDIA HGX ™ A100 GPU Camarero DAS7TGVQ-24RT Tyrone NVIDIA A100 based SERVERS • Supports 4x A100 40GB SXM4 GPUs • Supports CPU TDP up to 280W • Dual AMD EPYC™ 7002 Series Processors in up to 128 Cores • Flexible storage with 4 hotswap for SAS, SATA or NVMe • PCI-E Gen 4 NVLink for fast GPU-GPU connections • 32 DIMM Slots that allow up to 8TB of 3200Mhz DDR4 memory • 4 Hot-swap heavy duty fans • 2x 2200W Redundant Power Supplies, Titanium Level PCI-E Gen 4 NEW LAUNCH NVIDIA NVLink
- 12. 4U GPU server up to 8 NVIDIA HGX ™ A100 GPU Tyrone NVIDIA A100 based SERVERS NVIDIA NVLink & NVSwitch NEW LAUNCH • Supports up to 8 double-width GPUs, • Supports CPU TDP up to 280W • Dual AMD EPYC™ 7002 Series Processors in up to 128 Cores • Flexible storage with 4 hotswap for SAS, SATA or NVMe • PCI-E Gen 4 NVLink for fast GPU-GPU connections • 32 DIMM Slots that allow up to 8TB of 3200Mhz DDR4 memory • 4 Hot-swap heavy duty fans • 2x 2000W Redundant Power Supplies, Titanium Level
- 13. 4U GPU server up to 8 NVIDIA HGX ™ A100 GPU Tyrone NVIDIA A100 based SERVERS NVIDIA NVLink COMING SOON • Supports Intel Xeon • Supports NVLink • 8 x NVIDIA Tesla A100 SXM4
- 14. Delivers 4XFASTER TRAINING than other GPU-based systems Your Personal AI Supercomputer Power-on to Deep Learning in Minutes Pre-installed with Powerful Deep Learning Software Extend workloads from your Desk-to-Cloud in Minutes
- 15. Mixed Workloads Convergence of AI|HPC| Cloud | Containers
- 16. The Era of Modern Mixed Workload F L E X I B L E Is the usage going to be constant? O P T I M I Z A T I O N Is optimal utilization required? R E S I L I E N C E Do we need the application to run all the time. E A S E Is ‘ease of maintenance’ key? S C A L A B I L I T Y & S P E E D Do we have one size that fits all?
- 17. Connectivity and usage Virtual Desktop Laptop Tyrone Cloud Manager Tyrone Cloud Manager Laptop
- 18. Run Multiple Applications simultaneously Flow Architecture Revolutionizing Deep Learning CPU-GPU Environment 10X20X30X40X50X60X70X SPEED WITH TYRONE KUBYTS™ CLIENT Compatible Workstations has a repository of : 50 containerized applications 100s of Containers CLOUD
- 19. Tyrone KUBITS : Revolutionizing Deep Learning CPU-GPU Environment Run different applications simultaneously Check for Tyrone KUBITS Compatible Workstations Get access to over 100+ Containers on Tyrone KUBITS Cloud. High scalability Affordable price Has both GPU & CPU Optimized Containers Design a simple Workstation or Large Clusters with KUBITS technology. Talk to our experts & build the right workstation within your budget. KUBITS CLOUDCOMPATIBLE
- 20. AGENDA 1 . About Tyrone World’s high performing AI platform system – A100 Get Development, Training, Inference in one Era of Modern Mixed Workloads Tyrone Kubyts™ 2. Word Embeddings How Word embeddings create a context based relationship How to create Word Embeddings 3. Sequence Modelling Introduction of Deep learning in NLP Overview on the model architecture to use 4. Advanced Language Models Overview of the Language models How are they created Transformers BERT , GP2 etc 5. NLP Attention Mechanism Overview on attention Mechanism 6. Case Studies
- 21. Word Embedding • Word Embedding is a language modeling technique used for mapping words to vectors of real numbers. • It represents words or phrases in vector space with several dimensions. • Word embeddings can be generated using various methods like neural networks, co-occurrence matrix, probabilistic models, etc. • Word2Vec consists of models for generating word embedding. These models are shallow two layer neural networks having one input layer, one hidden layer and one output layer. Word2Vec utilizes two architectures
- 22. CBOW – Continuous Bag of Words • CBOW model predicts the current word given context words within specific window. The input layer contains the context words and the output layer contains the current word. • The hidden layer contains the number of dimensions in which we want to represent current word present at the output layer.
- 23. Skip Gram – Word Embeddings • Skip gram predicts the surrounding context words within specific window given current word. The input layer contains the current word and the output layer contains the context words. • The hidden layer contains the number of dimensions in which we want to represent current word present at the input layer.
- 24. Advanced Language Models and Transformers ELMo ULMFit BERT Transformer
- 25. Transformer
- 26. BERT Architecture • Transformer is an attention-based architecture for NLP • Transformer composed of two parts: Encoding component and Decoding component • BERT is a multi-layer bidirectional Transformer encoder
- 28. BERT vs GPT
- 29. Q&A Session Hirdey Vikram [email protected] India (North) Niraj [email protected] India (South) Vivek [email protected] India (East) Navin [email protected] India (West) Anupriya [email protected] Singapore Arun [email protected] UAE Agam [email protected] Indonesia Contact our team if you have any further questions after this webinar [email protected] to our AI Experts