Scribd
Upload
16 views

How to Build LLMs From Scratch

The document outlines the process of building Large Language Models (LLMs) from data collection to evaluation. It includes steps such as data scraping, preprocessing, model architecture selection, post-training alignment, deployment optimization, and performance benchmarking. Each phase emphasizes the importance of data quality, model training, and continuous improvement.

Uploaded by

Abdul Samad Nasir
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
16 views

How to Build LLMs From Scratch

The document outlines the process of building Large Language Models (LLMs) from data collection to evaluation. It includes steps such as data scraping, preprocessing, model architecture selection, post-training alignment, deployment optimization, and performance benchmarking. Each phase emphasizes the importance of data quality, model training, and continuous improvement.

Uploaded by

Abdul Samad Nasir
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

How to

Build LLMs from Data to Evaluation


© Crafted & illustrated by: Dr. Maryam Miradi

Data Collection (Web : Preprocessing and Pretraining incl. Model Architecture


1 Web Scraping & Data 2 Tokenization : Dataset 3
Selection: Defining the Architecture +
GatheringScraping & Structuring + Metadata Compute & Infrastructure Setup +
Pre-Training

Curation) + Data Filtering & Generation + Data Formatting Pretraining the Model + Training
Cleaning + for Training Optimizations

Model Alignment (Post- Model Deployment & Evaluation & Benchmarking:


4 Training & RLHF):
5 Optimization: 6 Benchmarking Performance +
Supervised Fine-Tuning Quantization & Red-Teaming & Adversarial
Post-Training

(SFT) + Reinforcement Compression + Serving & Testing


Learning from Human API Deployment +
Feedback (RLHF) + Continuous Monitoring &
Constitutional AI & Safety Improvement
Fine-Tuning
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

Data Collection (Web :


1 Web Scraping & Data
GatheringScraping &
Curation) + Data Filtering &
Cleaning +

Pre-Training
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

Preprocessing and
2 Tokenization : Dataset
Structuring + Metadata
Generation + Data Formatting
for Training

Pre-Training
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

3 Pretraining incl. Model Architecture


Selection: Defining the Architecture +
Compute & Infrastructure Setup +
Pretraining the Model + Training
Optimizations

Pre-Training
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

Model Alignment (Post-Training &


4 RLHF): Supervised Fine-Tuning (SFT)
+ Reinforcement Learning from
Human Feedback (RLHF) +
Constitutional AI & Safety Fine-
Tuning

Post-Training
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

Model Deployment &


5 Optimization:
Quantization &
Compression + Serving &
API Deployment +
Continuous Monitoring &
Improvement

Post-Training
How to
Build LLMs
from Data to Evaluation
© Crafted & illustrated by: Dr. Maryam Miradi

Evaluation & Benchmarking:


6 Benchmarking Performance +
Red-Teaming & Adversarial
Testing

Post-Training

You might also like