![9/13
JOIN
`bigquery-public-data.github_repos.licenses` AS l
ON
f.repo_name = l.repo_name
WHERE
NOT c.binary
AND ((f.path LIKE '%.py')
AND (c.size BETWEEN 1024 AND 1048575))
Step 2: Setting up the tokenizer and model
To train a GPT model, we need a tokenizer. Here we have used an existing tokenizer (e.g.,
GPT-2) and trained it on the dataset mentioned above with
the train_new_from_iterator() method.
# Iterator for Training
def batch_iterator(batch_size=10):
for _ in tqdm(range(0, args.n_examples, batch_size)):
yield [next(iter_dataset)["content"] for _ in range(batch_size)]
# Base tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
base_vocab = list(bytes_to_unicode().values())
# Load dataset
dataset = load_dataset("lvwerra/codeparrot-clean", split="train", streaming=True)
iter_dataset = iter(dataset)
# Training and saving
new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(),
vocab_size=args.vocab_size,
initial_alphabet=base_vocab)
new_tokenizer.save_pretrained(args.tokenizer_name, push_to_hub=args.push_to_hub)
Next, a new model is initialized using the same hyperparameters as GPT-2 large (1.5B
parameters). This model can be used to adjust the embedding layer to fit a new tokenizer, by
adding some stability tweaks. The code snippet for the same is mentioned below:
tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_name)
# Configuration
config_kwargs = {"vocab_size": len(tokenizer),
"scale_attn_by_layer_idx": True,
"reorder_and_upcast_attn": True}
# Load model with config and push to hub
config = AutoConfig.from_pretrained('gpt2-large', **config_kwargs)
model = AutoModelForCausalLM.from_config(config)
model.save_pretrained(args.model_name, push_to_hub=args.push_to_hub)](https://image.slidesharecdn.com/howtobuildagptmodel-230413103339-513350d1/85/How-to-build-a-GPT-model-pdf-9-320.jpg)
![10/13
With a streamlined tokenizer and a newly established model, we are ready to begin the model
training process.
Step 3: Implementing the training loop
Prior to commencing the training, it’s necessary to configure the optimizer and the schedule
for the learning rate. Here, a support function is used for exclusion to prevent weight decay
on biases and LayerNorm weights.
def get_grouped_params(model, args, no_decay=["bias", "LayerNorm.weight"]):
params_with_wd, params_without_wd = [], []
for n, p in model.named_parameters():
if any(nd in n for nd in no_decay): params_without_wd.append(p)
else: params_with_wd.append(p)
return [{"params": params_with_wd, "weight_decay": args.weight_decay},
{"params": params_without_wd, "weight_decay": 0.0},]
optimizer = AdamW(get_grouped_params(model, args), lr=args.learning_rate)
lr_scheduler = get_scheduler(name=args.lr_scheduler_type, optimizer=optimizer,
num_warmup_steps=args.num_warmup_steps,
num_training_steps=args.max_train_steps,)
We can now move forward with composing the core training cycle.It will resemble a typical
PyTorch training cycle with some modifications. You’ll notice that
accelerator functions are utilized here, instead of PyTorch’s native methods. Additionally,
after every evaluation, the model is transferred to the accelerator.
# Train model
model.train()
completed_steps = 0
for step, batch in enumerate(train_dataloader, start=1):
loss = model(batch, labels=batch, use_cache=False).loss
loss = loss / args.gradient_accumulation_steps
accelerator.backward(loss)
if step % args.gradient_accumulation_steps == 0:
accelerator.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()
completed_steps += 1
if step % args.save_checkpoint_steps == 0:
eval_loss, perplexity = evaluate(args)
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(model)
unwrapped_model.save_pretrained(args.save_dir, save_function=accelerator.save)](https://image.slidesharecdn.com/howtobuildagptmodel-230413103339-513350d1/85/How-to-build-a-GPT-model-pdf-10-320.jpg)
How to build a GPT model.pdf
- 1. 1/13 How to build a GPT model? leewayhertz.com/build-a-gpt-model Introduced by OpenAI, powerful Generative Pre-trained Transformer (GPT) language models have opened up new frontiers in Natural Language Processing (NLP). The integration of GPT models into virtual assistants and chatbots boosts their capabilities, which has resulted in a surge in demand for GPT models. According to a report published by Allied Market Research, titled “Global NLP Market,” the global NLP market size was valued at $11.1 billion in 2020 and is estimated to reach $341.5 billion by 2030, growing at a CAGR of 40.9% from 2021 to 2030. Interestingly, the demand for GPT models are a major contributor to this growth. GPT models are a collection of deep learning-based language models created by the OpenAI team. Without supervision, these models can perform various NLP tasks like question- answering, textual entailment, text summarization, etc. These language models require very few or no examples to understand tasks. They perform equivalent to or even better than state-of-the-art models trained in a supervised fashion. The most trained GPT model -GPT-3, has 175 billion learning parameters, making it ten times more powerful than any language model. It has the edge over other models in that it can perform tasks without extensive tuning; it only requires little textual-interactional demonstration, and the model does the rest. An advanced trained GPT model can make life
- 2. 2/13 easier by performing language translation, text summarization, question answering, chatbot integration, content generation, sentiment analysis, named entity recognition, text classification, text completion, text-to-speech synthesis and much more. This article deeply delves into all aspects of GPT models and discusses the steps required to build a GPT model from scratch. What is a GPT model? GPT stands for Generative Pre-trained Transformer, the first generalized language model in NLP. Previously, language models were only designed for single tasks like text generation, summarization or classification. GPT is the first generalized language model ever created in the history of natural language processing that can be used for various NLP tasks. Now let us explore the three components of GPT, namely Generative, Pre-Trained, and Transformer and understand what they mean. Generative: Generative models are statistical models used to generate new data. These models can learn the relationships between variables in a data set to generate new data points similar to those in the original data set. Pre-trained: These models have been pre-trained using a large data set which can be used when it is difficult to train a new model. Although a pre-trained model might not be perfect, it can save time and improve performance. Transformer: The transformer model, an artificial neural network created in 2017, is the most well-known deep learning model capable of handling sequential data such as text. Many tasks like machine translation and text classification are performed using transformer models. GPT can perform various NLP tasks with high accuracy depending on the large datasets it was trained on and its architecture of billion parameters, allowing it to understand the logical connections within the data. GPT models, like the latest version GPT-3, have been pre- trained using text from five large datasets, including Common Crawl and WebText2. The corpus contains nearly a trillion words, allowing GPT-3 to perform NLP tasks quickly and without any examples of data. Working mechanism of GPT models GPT is an AI language model based on transformer architecture that is pre-trained, generative, unsupervised, and capable of performing well in zero/one/few-shot multitask settings. It predicts the next token (an instance of a sequence of characters) from a sequence of tokens for NLP tasks, it has not been trained on. After seeing only a few examples, it can achieve the desired outcomes in certain benchmarks, including machine translation, Q&A and cloze tasks. GPT models calculate the likelihood of a word appearing in a text given that it appears in another text primarily based on conditional probability. For example, in the
- 3. 3/13 sentence, “Margaret is organizing a garage sale…Perhaps we could purchase that old…” the word chair is more likely appropriate than the word ‘elephant’. Also, transformer models use multiple units called attention blocks that learn which parts of a text sequence to be focused on. One transformer might have multiple attention blocks, each learning different aspects of a language. LeewayHertz Output Probabilities Feed Forward Multi- Head Attention Multi- Head Attention Nx Nx Positional Encoding Positional Encoding Outputs (shifted right) Inputs Add & Norm Add & Norm Add & Norm Add & Norm Feed Forward Add & Norm Linear Softmax Masked Multi-Head Attention Input Embedding Output Embedding Transformer architecture A transformer architecture has two main segments: an encoder that primarily operates on the input sequence and a decoder that operates on the target sequence during training and predicts the next item. For example, a transformer might take a sequence of English words and predict the French word in the correct translation until it is complete.
- 4. 4/13 The encoder determines which parts of the input should be emphasized. For example, the encoder can read a sentence like “The quick brown fox jumped.” It then calculates the embedding matrix (embedding in NLP allows words with similar meanings to have a similar representation) and converts it into a series of attention vectors. Now, what is an attention vector? You can view an attention vector in a transformer model as a special calculator, which helps the model understand which parts of any given information are most important in making a decision. Suppose you have been asked multiple questions in an exam that you must answer using different information pieces. The attention vector helps you to pick the most important information to answer each question. It works in the same way in the case of a transformer model. The multi-head attention block initially produces these attention vectors. They are then normalized and passed into a fully connected layer. Normalization is again done before being passed to the decoder. During training, the encoder works directly on the target output sequence. Let us say that the target output is the French translation of the English sentence “The quick brown fox jumped.” The decoder computes separate embedding vectors for each French word of the sentence. Additionally, the positional encoder is applied in the form of sine and cosine functions. Also, masked attention is used, which means that the first word of the French sentence is used, whereas all other words are masked. This allows the transformer to learn to predict the next French words. These outputs are then added and normalized before being passed on to another attention block which also receives the attention vectors generated by the encoder. Alongside, GPT models employ some data compression while consuming millions upon millions of sample texts to convert words into vectors which are nothing but numerical representations. The language model then unpacks the compressed text into human-friendly sentences. The model’s accuracy is improved by compressing and decompressing text. This also allows it to calculate the conditional probability of each word. GPT models can perform well in “few shots” settings and respond to text samples that have been seen before. They only require a few examples to produce pertinent responses because they have been trained on many text samples. Besides, GPT models have many capabilities, such as generating unprecedented-quality synthetic text samples. If you prime the model with an input, it will generate a long continuation. GPT models outperform other language models trained on domains such as Wikipedia, news, and books without using domain-specific training data. GPT learns language tasks such as reading comprehension, summarization and question answering from the text alone, without task-specific training data. These tasks’ scores (“score” refers to a numerical value the model assigns to represent the likelihood or probability of a given output or result) are not the best, but they suggest unsupervised techniques with sufficient data and computation that could benefit the tasks. Here is a comprehensive comparison of GPT models with other language models.
- 5. 5/13 Feature GPT BERT (Bidirectional Encoder Representations from Transformers) ELMo (Embeddings from Language Models) Pretraining approach Unidirectional language modeling Bidirectional language modeling (masked language modeling and next sentence prediction) Unidirectional language modeling Pretraining data Large amounts of text from the internet Large amounts of text from the internet A combination of internal and external corpus Architecture Transformer network Transformer network Deep bi-directional LSTM network Outputs Context-aware token-level embeddings Context-aware token-level and sentence-level embeddings Context-aware word- level embeddings Fine-tuning approach Multi-task fine- tuning (e.g., text classification, sequence labeling) Multi-task fine-tuning (e.g., text classification, question answering) Fine-tuning on individual tasks Advantages Can generate text, high flexibility in fine-tuning, large model size Strong performance on a variety of NLP tasks, considering the context in both directions Generates task- specific features, considers context from the entire input sequence Limitations Can generate biased or inaccurate text, requires large amounts of data Limited to fine-tuning and requires task-specific architecture modifications; requires large amounts of data Limited context and task-specific; requires task-specific architecture modifications Prerequisites to build a GPT model To build a GPT (Generative Pretrained Transformer) model, the following tools and resources are required: A deep learning framework, such as TensorFlow or PyTorch, to implement the model and train it on large amounts of data. A large amount of training data, such as text from books, articles, or websites to train the model on language patterns and structure.
- 6. 6/13 A high-performance computing environment, such as GPUs or TPUs, for accelerating the training process. Knowledge of deep learning concepts, such as neural networks and natural language processing (NLP), to design and implement the model. Tools for data pre-processing and cleaning, such as Numpy, Pandas, or NLTK, to prepare the training data for input into the model. Tools for evaluating the model, such as perplexity or BLEU scores, to measure its performance and make improvements. An NLP library, such as spaCy or NLTK, for tokenizing, stemming and performing other NLP tasks on the input data. Besides, you need to understand the following deep learning concepts to build a GPT model: Neural networks: As GPT models implement neural networks, you must thoroughly understand how they work and their implementation techniques in a deep learning framework. Natural language Processing (NLP): For GPT modeling processes, tokenization, stemming, and text generation, NLP techniques are widely used. So, it is necessary to have a fundamental understanding of NLP techniques and their applications. Transformers: GPT models work based on transformer architecture, so understanding it and its role in language processing and generation is important. Attention mechanisms: Knowledge of how attention mechanisms work is essential to enhance the performance of the GPT model. Pretraining: It is essential to apply the concept of pretraining to the GPT model to improve its performance on NLP tasks. Generative models: Understanding the basic concepts and methods of generative models is essential to understand how they can be applied to build your own GPT model. Language modeling: GPT models work based on large amounts of text data. So, a clear understanding of language modeling is required to apply it for GPT model training. Optimization: An understanding of optimization algorithms, such as stochastic gradient descent, is required to optimize the GPT model during training. Alongside this, you need proficiency in any of the following programming languages with a solid understanding of programming concepts, such as object-oriented programming, data structures, and algorithms, to build a GPT model. Python: The most commonly used programming language in deep learning and AI. It has several libraries, such as TensorFlow, PyTorch, and Numpy, used for building and training GPT models. R: A popular programming language for data analysis and statistical modeling, with several packages for deep learning and AI.
- 7. 7/13 Julia: A high-level, high-performance programming language well-suited for numerical and scientific computing, including deep learning. How to create a GPT model? A step-by-step guide Building a GPT model involves the following steps: Step 1: Data preparation To prepare a dataset to build a GPT model, the following steps can be followed: Data collection: You need to collect a large amount of text data, such as books, articles, and websites, to use it as the training data for your GPT model. Data cleaning: You should remove any irrelevant information, such as HTML tags or irrelevant headers, and standardize the text format. Tokenize the data: Divide the text into smaller units, such as words or subwords, to enable the model to learn the language patterns and structure. Data pre-processing: Perform any necessary pre-processing tasks on the data, such as stemming, removing stop words, or converting the text to lowercase. Split the data: Divide the cleaned and pre-processed data into different sets, such as training, validation, and test sets to evaluate the model’s performance during training. Batch creation: Create batches of the training data to feed into the model during training. Depending on the requirements of the model, this can be done randomly or sequentially. Convert the data to tensor: TensorFlow and PyTorch are some basic data structures used in deep learning frameworks. So, you need to convert the data into tensors. It is essential to ensure that the data is of high quality, diverse, and in sufficient quantity to train the GPT model effectively and avoid overfitting. Step 2: Model architecture selection Model architecture selection is a crucial step in building a GPT model. It primarily depends on the type of data and task being addressed. While choosing an architecture, you need to consider the following factors: Task complexity: The task complexity should be analyzed properly to identify the factors that can impact the architecture, such as the size of the output space, the presence of multi-label or multi-class outputs, the presence of additional constraints, etc. For example, complex tasks may require more layers or sophisticated attention mechanisms.
- 8. 8/13 Data characteristics: You need to identify the characteristics of the data being processed, which include the length of the sequences, the presence of structured or unstructured data, and the size of the vocabulary. For example, longer sequences may require deeper networks, while convolutional neural networks benefit the structured data. Computational constraints: The choice of architecture also depends on the memory requirement of the computational resources available along with GPU resources. For example, larger models may require more memory and computational resources. Ultimately, the choice of architecture is a trade-off between the desired performance, the computational resources available, and the complexity of the task and data. So, it needs careful experimentation and iteration to determine the best architecture for a given task. Step 3: Model training Model training is the most crucial phase of the entire GPT model-building process, as in this step, the model is exposed to vast amounts of text data and learns to predict the next word in a sequence based on the input context. During the training process, the model’s parameters are adjusted in a way that its predictions become more accurate and it achieves a certain level of performance. The quality of the training data and the choice of hyperparameters greatly influence the performance of the final model, making model training a critical component in the development of GPT models. Here we will describe how to train a large GPT-2 model that can auto-complete your Python code. You can get the code from Github by searching with the string, codeparrot. Here are the basic steps followed in building the model: Step 1: Data generation Before training the model, we need a large training dataset. For this Python code generation model, you can access the GitHub dump available on Google’s BigQuery, which is filtered for all Python files and is a 180 GB dataset with 22 million files. The SQL query to create the dataset is the following: SELECT f.repo_name, f.path, c.copies, c.size, c.content, l.license FROM `bigquery-public-data.github_repos.files` AS f JOIN `bigquery-public-data.github_repos.contents` AS c ON f.id = c.id
- 9. 9/13 JOIN `bigquery-public-data.github_repos.licenses` AS l ON f.repo_name = l.repo_name WHERE NOT c.binary AND ((f.path LIKE '%.py') AND (c.size BETWEEN 1024 AND 1048575)) Step 2: Setting up the tokenizer and model To train a GPT model, we need a tokenizer. Here we have used an existing tokenizer (e.g., GPT-2) and trained it on the dataset mentioned above with the train_new_from_iterator() method. # Iterator for Training def batch_iterator(batch_size=10): for _ in tqdm(range(0, args.n_examples, batch_size)): yield [next(iter_dataset)["content"] for _ in range(batch_size)] # Base tokenizer tokenizer = GPT2Tokenizer.from_pretrained("gpt2") base_vocab = list(bytes_to_unicode().values()) # Load dataset dataset = load_dataset("lvwerra/codeparrot-clean", split="train", streaming=True) iter_dataset = iter(dataset) # Training and saving new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=args.vocab_size, initial_alphabet=base_vocab) new_tokenizer.save_pretrained(args.tokenizer_name, push_to_hub=args.push_to_hub) Next, a new model is initialized using the same hyperparameters as GPT-2 large (1.5B parameters). This model can be used to adjust the embedding layer to fit a new tokenizer, by adding some stability tweaks. The code snippet for the same is mentioned below: tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_name) # Configuration config_kwargs = {"vocab_size": len(tokenizer), "scale_attn_by_layer_idx": True, "reorder_and_upcast_attn": True} # Load model with config and push to hub config = AutoConfig.from_pretrained('gpt2-large', **config_kwargs) model = AutoModelForCausalLM.from_config(config) model.save_pretrained(args.model_name, push_to_hub=args.push_to_hub)
- 10. 10/13 With a streamlined tokenizer and a newly established model, we are ready to begin the model training process. Step 3: Implementing the training loop Prior to commencing the training, it’s necessary to configure the optimizer and the schedule for the learning rate. Here, a support function is used for exclusion to prevent weight decay on biases and LayerNorm weights. def get_grouped_params(model, args, no_decay=["bias", "LayerNorm.weight"]): params_with_wd, params_without_wd = [], [] for n, p in model.named_parameters(): if any(nd in n for nd in no_decay): params_without_wd.append(p) else: params_with_wd.append(p) return [{"params": params_with_wd, "weight_decay": args.weight_decay}, {"params": params_without_wd, "weight_decay": 0.0},] optimizer = AdamW(get_grouped_params(model, args), lr=args.learning_rate) lr_scheduler = get_scheduler(name=args.lr_scheduler_type, optimizer=optimizer, num_warmup_steps=args.num_warmup_steps, num_training_steps=args.max_train_steps,) We can now move forward with composing the core training cycle.It will resemble a typical PyTorch training cycle with some modifications. You’ll notice that accelerator functions are utilized here, instead of PyTorch’s native methods. Additionally, after every evaluation, the model is transferred to the accelerator. # Train model model.train() completed_steps = 0 for step, batch in enumerate(train_dataloader, start=1): loss = model(batch, labels=batch, use_cache=False).loss loss = loss / args.gradient_accumulation_steps accelerator.backward(loss) if step % args.gradient_accumulation_steps == 0: accelerator.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() lr_scheduler.step() optimizer.zero_grad() completed_steps += 1 if step % args.save_checkpoint_steps == 0: eval_loss, perplexity = evaluate(args) accelerator.wait_for_everyone() unwrapped_model = accelerator.unwrap_model(model) unwrapped_model.save_pretrained(args.save_dir, save_function=accelerator.save)
- 11. 11/13 if accelerator.is_main_process: hf_repo.push_to_hub(commit_message=f"step {step}") model.train() if completed_steps >= args.max_train_steps: break Done! That’s the code to train a full GPT-2 model. (However, you need to access the full code from the GitHub location as mentioned above) Step 4: Model evaluation Model evaluation is an important step you need to perform when building a GPT model, as it provides insight into how well the model is performing. The metrics used for evaluation vary depending on the task, but some common metrics include accuracy, perplexity, and F1 score. To perform an evaluation in a GPT model, you must first set aside a portion of your training data for validation. During the training process, you can periodically evaluate the model on this validation set rather than the training set. You can then compare the model’s performance on the validation set to its performance on the training set to check for overfitting. When evaluating the model, you can calculate various metrics based on the model’s predictions and compare them to the actual outputs. For example, you can calculate the model’s accuracy by comparing its predictions to the true labels, or you can calculate the perplexity of the model by evaluating how well it predicts the next word in a sequence. After evaluating the model, you can use the metrics to make informed decisions about how to improve the model, such as adjusting the learning rate, changing the model architecture, or increasing the amount of training data. Regular model evaluation and adjustment help refine the model and produce a high-performing GPT model. Things to consider while building a GPT model Removing bias and toxicity As we strive to build powerful generative AI models, we must be aware of the tremendous responsibility that comes with it. It is crucial to acknowledge that models such as GPT are trained on vast and unpredictable data from the internet, which can lead to biases and toxic language in the final product. As AI technology evolves, responsible practices become increasingly important. We must ensure that our AI models are developed and deployed ethically and with social responsibility in mind. Prioritizing responsible AI practices is vital in reducing the risks of biased and toxic content while fully unlocking the potential of generative AI to create a better world.
- 12. 12/13 It is necessary to take a proactive approach to ensure that the output generated by AI models is free from bias and toxicity. This includes filtering training datasets to eliminate potentially harmful content and implementing watchdog models to monitor output in real-time. Furthermore, leveraging first-party data to train and fine-tune AI models can significantly enhance their quality. This allows customization to meet specific use cases, improving overall performance. Improving hallucination It is essential to acknowledge that while GPT models can generate convincing arguments, they may not always be based on factual accuracy. Within the developer community, this issue is known as “hallucination,” which can reduce the reliability of the output produced by these AI models. To overcome this challenge, you need to consider the measures as taken by OpenAI and other vendors, including data augmentation, adversarial training, improved model architectures, and human evaluation to enhance the accuracy of the output and decrease the risk of hallucination and ensure output generated by the model is as precise and dependable as possible. Preventing data leakage Establishing transparent policies is crucial to prevent developers from passing sensitive information into GPT models, which could be incorporated into the model and resurfaced in a public context. By implementing such policies, we can prevent the unintentional disclosure of sensitive information, safeguard the privacy and security of individuals and organizations, and avoid any negative consequences. This is essential to remain vigilant in safeguarding against potential risks associated with the use of GPT models and take proactive measures to mitigate them. Incorporating queries and actions Current generative models can provide answers based on their initial large training data set or smaller “fine-tuning” data sets, which are not real-time and historical. However, the next generation of models will take a significant leap forward. These models will possess the capability to identify when to seek information from external sources such as a database or Google or trigger actions in external systems, transforming generative models from isolated oracles to fully connected conversational interfaces with the world. By enabling this new level of connectivity, we can unlock a new set of use cases and possibilities for these models, creating a more dynamic and seamless user experience that provides real-time, relevant information and insights. Endnote
- 13. 13/13 GPT models are a significant milestone in the history of AI development, which is a part of a larger LLM trend that will grow in the future. Furthermore, OpenAI’s groundbreaking move to provide API access is part of its model-as-a-service business scheme. Additionally, GPT’s language-based capabilities allow for creating innovative products as it excels at tasks such as text summarization, classification, and interaction. GPT models are expected to shape the future internet and how we use technology and software. Building a GPT model may be challenging, but with the right approach and tools, it becomes a rewarding experience that opens up new opportunities for NLP applications. Want to get a competitive edge in your industry with cutting-edge GPT technology? Contact LeewayHertz’s AI experts to take your solution’s natural language processing capabilities to the next level!