Scribd
Upload
45 views

Bert Fine Tuning (AutoRecovered)

This document provides instructions for fine-tuning BERT for classification using local machines with TensorFlow. It describes preprocessing data by tokenizing sentences and one-hot encoding labels. It also discusses creating a BERT layer using BertModelLayer and freezing original BERT parameters. The document notes making a model by joining sequential layers after the BERT layer and using dropouts to reduce overfitting.

Uploaded by

Dinesh Yedakula
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
45 views

Bert Fine Tuning (AutoRecovered)

This document provides instructions for fine-tuning BERT for classification using local machines with TensorFlow. It describes preprocessing data by tokenizing sentences and one-hot encoding labels. It also discusses creating a BERT layer using BertModelLayer and freezing original BERT parameters. The document notes making a model by joining sequential layers after the BERT layer and using dropouts to reduce overfitting.

Uploaded by

Dinesh Yedakula
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

1.

Fine-Tuning BERT for Classification using local machine


with internal GPU Reference

STEP 1: INSTALLING REQUIRED PACKAGES


1.Transformers : Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert)
provides general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural
Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models
in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch.
pip install transformers

2.pytorch :

pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

STEP 2: PREPROCESSING DATA using transformers


Import BERT Model and BERT Tokenizer from transformers. Tokenizer is used to generate tokens from the
textdata which could be useful for further process
from transformers import AutoModel, BertTokenizerFast
bert = AutoModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

STEP 3: PREPROCESSING DATA using transformers


tokenize and encode sequences of train and test data using tokenizer.batch_encode_plus()
converting interger sequences to tensors using torch.tensor()

STEP 4: Freeze BERT Parameters


STEP 5: Defining Model
Create own neural network architecture and pass the pre-trained BERT to our define architecture and
push the model to GPU.Architecure can be defined by sequential layer with dropouts,activations. Lastly
appending optimizer from hugging face transformers

If data is imbalanced dataset we can use class weight from sklearn.utils.class_weight


Compute the class weights convert class weights to tensor using torch.tensor function ad setting right
epochs

STEP 6: FineTuning BERT


One of finetuning if we are using pytorch we need to write own function to perform the finetuning.

2.Fine-Tuning BERT for Classification using GCP Platform


with GCP TPU’s. Reference

STEP 1: Create a Cloud Storage bucket to hold dataset and


model output.

This section provides information on setting up Cloud Storage bucket and a Compute Engine
VM.Open a Cloud Shell window.

Open Cloud Shell


1. Create a variable for your project's ID.
export PROJECT_ID=project-id

2. Configure gcloud command-line tool to use the project where you want to create Cloud TPU.

gcloud config set project ${PROJECT_ID}

3. Create a Cloud Storage bucket using the following command:


gsutil mb -p ${PROJECT_ID} -c standard -l us-central1 -b on gs://bucket-name

4. Launch a Compute Engine VM and Cloud TPU using the ctpu up command.

$ ctpu up --tpu-size=v3-8 \
--machine-type=n1-standard-8 \
--zone=us-central1-b \
--tf-version=1.15.3 \
--name=bert-tutorial

5. The configuration you specified appears. Enter y to approve or n to cancel.


6. When the ctpu up command has finished executing, verify that your shell prompt has changed
from username@project to username@vm-name. This change shows that you are now logged into
your Compute Engine VM.
gcloud compute ssh bert-tutorial --zone=us-central1-b

As you continue these instructions, run each command that begins with (vm)$ in your VM session
window.

STEP 2: Clone the BERT repository and other required files.

Clone the BERT repository

From your Compute Engine virtual machine (VM), clone the BERT repository.

(vm)$ git clone https://github.com/google-research/bert

Download download_glue_data.py

This tutorial uses the General Language Understanding Evaluation (GLUE) benchmark to evaluate
and analyze the performance of the model. To use this benchmark, download
the download_glue_data.py script using the following git clone command:

(vm)$ git clone https://gist.github.com/60c2bdb54d156a41194446737ce03e2e.git


download_glue_data

Download the GLUE data

Next, run the download_glue_data.py on your Compute Engine VM.

(vm)$ python3 download_glue_data/download_glue_data.py --data_dir $HOME/glue_data -


-tasks all
STEP 3: Run the training job
Train the model

From your Compute Engine VM, run the following command.

python3 ./bert/run_classifier.py \
--task_name=${TASK_NAME} \
--do_train=true \
--do_eval=true \
--data_dir=${GLUE_DIR}/${TASK_NAME} \
--vocab_file=${BERT_BASE_DIR}/vocab.txt \
--bert_config_file=${BERT_BASE_DIR}/bert_config.json \
--init_checkpoint=${BERT_BASE_DIR}/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=2e-5 \
--num_train_epochs=3.0 \
--output_dir=${STORAGE_BUCKET}/${TASK_NAME}-output/ \
--use_tpu=True \
--tpu_name=${TPU_NAME}

STEP 4: Verify the output results.

Verify your results

The training should take less than 5 minutes. When the training completes, you should see results
similar to the following:

***** Eval results *****


eval_accuracy = 0.845588
eval_loss = 0.64990824
global_step = 343
loss = 0.34979442
3.Fine-Tuning BERT for Classification using local machine
with TensorFlow Reference

STEP 1:Preprocessing
The BERT layer requires in input an array of sequences with a defined max length for each sequence.Create

an instance of the BERT FullTokenizer, that requires in input the corpora used for training the BERT

model.Using the tokenizer, the data preparation is as follows:

Split data into training set, training labels, testing set and testing labels and Shuffling each sentence

set then Tokenized each sentence set using the tokenizer described above. Appending the tokens

which completely depends upon the dataset type. Performed one hot encoder to each label of the

label set

STEP 2 :BERT Layer


BertModelLayer wrapper is used to create the Keras Layer.

bert_config.json which contains all the parameters required for creating the
layer.To freeze all the original layer wrapper into the BertModelLayer class it
would be better keep all bert parameters False

Code for creating bert layer is below


Import bert

import os

def createBertLayer():
global bert_layer

bertDir = os.path.join(modelBertDir, "multi_cased_L-12_H-768_A-12")


#bert containing directory

bert_params = bert.params_from_pretrained_ckpt(bertDir)
#imputing pretrained parmas to bert model

bert_layer = bert.BertModelLayer.from_params(bert_params, name="bert")


#creating bert layer

bert_layer.apply_adapter_freeze()

Making Model
After creating bert layer we can join sequential layers right after it,and can
use dropouts to reduce overfitting

- Dinesh Yedakula
- 4th Year
- Mechanical Engineering
- IIt Dharwad

You might also like