Faculty of Computing

Artificial Intelligence - CS-370

Fall 2023
BSCS11 C
Lab 12: Back Propagation

Date: 15th Dec 2023

Instructor: Dr. Imran Malik
Lab Eng: Shakeela

CS370: Artificial Intelligence Fall 2023 Page 1

 Lab 13: Back Propagation
Introduction:

Back propagation is a widely used algorithm in machine learning for training artificial neural
networks. It is a supervised learning algorithm that uses gradient descent to optimize the neural
network's weights and biases. In this lab, we will explore the concept of back propagation and
how it can be used to train a neural network.

Objectives:

The main objectives of this lab are:

 Understand the concept of back propagation.

 Learn how to train a neural network using back propagation.
 Learn how to implement back propagation using Python.
 Apply back propagation to a real-world dataset.

Tools:

Python 3.x, NumPy, Matplotlib, Jupyter Notebook or Google Colab

Description:

We will implement a neural network using Python and NumPy. We will use the MNIST dataset,
which contains a large number of handwritten digit images. We will preprocess the dataset, train
the neural network using back propagation, and evaluate its performance on a test set.

MNIST Dataset:

This dataset contains 70,000 images of handwritten digits, split into 60,000 training examples
and 10,000 test examples. The goal is to correctly classify each digit image into its
corresponding numerical digit (0-9).

The MNIST dataset contains images of size 28x28 pixels. Each pixel value is an integer between
0 and 255, representing the grayscale value of the pixel. The dataset is commonly used as a
benchmark for image recognition tasks in machine learning.

Here is a link to the MNIST dataset: http://yann.lecun.com/exdb/mnist/. However, MNIST

dataset is also saved in all machine learning libraries e.g. PyTorch, Tensorflow and Scikit-learn.
It can be used by importing from these libraries as well.

CS370: Artificial Intelligence Fall 2023 Page 2

Methodology:

1. Prepare the dataset: Load the dataset and split it into training and test sets. Normalize
the input features to bring them to a similar scale.
2. Define the model architecture: Decide on the number of input and output neurons, as
well as the number of hidden layers and neurons in each hidden layer. Also, choose an
appropriate activation function for each layer.
3. Initialize the weights and biases: Assign random values to the weights and biases of the
network.
4. Forward propagation: Pass the input through the network and compute the output.
5. Compute the loss: Calculate the difference between the predicted output and the actual
output using an appropriate loss function.
6. Backward propagation: Compute the gradients of the loss function with respect to the
weights and biases using the chain rule of differentiation.
7. Update the weights and biases: Adjust the weights and biases in the opposite direction
of the gradients to reduce the loss.
8. Repeat steps 4-7 for multiple epochs: Train the model for a fixed number of epochs or
until the loss stops decreasing.
9. Evaluate the model performance: Test the model on the test set and calculate metrics
such as accuracy, precision, and recall.
10. Make predictions: Use the trained model to predict the output for new input data.

Tasks:
 Load and preprocess the MNIST dataset using Python.
 Build a neural network using Python and NumPy with
o Different activation functions i.e., sigmoid, ReLu and hyperbolic tangent (tanh)
function on hidden layers and softmax on output layer.
o Different sizes of hidden layers i.e., 32, 64, 128 and 256.
o Use accuracy as evaluation metric.
 Train and evaluate the performance of the neural network.
 Compare and visualize the results of the neural network.
 Save the best performing network model using model.save() method.

CS370: Artificial Intelligence Fall 2023 Page 3

import numpy as np
from sklearn.model_selection import train_test_split
from tensorflow.keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(x_train.shape[0], -1) / 255.0

x_test = x_test.reshape(x_test.shape[0], -1) / 255.0

x_train, x_val, y_train, y_val = train_test_split(x_train, y_train,

test_size=0.1, random_state=42)

def one_hot_encode(labels, num_classes):

return np.eye(num_classes)[labels]

y_train = one_hot_encode(y_train, 10)

y_val = one_hot_encode(y_val, 10)
y_test = one_hot_encode(y_test, 10)

class NeuralNetwork:
def __init__(self, input_size, hidden_sizes, output_size, activations):
self.input_size = input_size
self.hidden_sizes = hidden_sizes
self.output_size = output_size
self.activations = activations

self.weights, self.biases = self.initialize_parameters()

def initialize_parameters(self):
sizes = [self.input_size] + self.hidden_sizes + [self.output_size]
weights = [np.random.randn(sizes[i], sizes[i+1]) for i in
range(len(sizes)-1)]
biases = [np.zeros((1, sizes[i+1])) for i in range(len(sizes)-1)]
return weights, biases

def forward_propagation(self, x):

layer_outputs = [x]
for i in range(len(self.hidden_sizes)):
x = self.activations[i](np.dot(x, self.weights[i]) + self.biases[i])
layer_outputs.append(x)

CS370: Artificial Intelligence Fall 2023 Page 4

 output = self.softmax(np.dot(x, self.weights[-1]) + self.biases[-1])
layer_outputs.append(output)
return layer_outputs

@staticmethod
def softmax(x):
exp_values = np.exp(x - np.max(x, axis=1, keepdims=True))
return exp_values / np.sum(exp_values, axis=1, keepdims=True)

def train_neural_network(model, x_train, y_train, x_val, y_val, epochs,

learning_rate):
for epoch in range(epochs):
layer_outputs = model.forward_propagation(x_train)

loss = -np.sum(y_train * np.log(layer_outputs[-1])) / len(x_train)

print(f"Epoch {epoch+1}/{epochs}, Loss: {loss}")

gradients = model.backward_propagation(layer_outputs, y_train)

model.update_parameters(gradients, learning_rate)

val_outputs = model.forward_propagation(x_val)[-1]
val_predictions = np.argmax(val_outputs, axis=1)
val_accuracy = np.mean(val_predictions == np.argmax(y_val, axis=1))
print(f"Validation Accuracy: {val_accuracy}")

def backward_propagation(self, layer_outputs, y_true):

gradients = []

output_error = layer_outputs[-1] - y_true

gradients.append(np.dot(layer_outputs[-2].T, output_error))

for i in range(len(self.hidden_sizes)-1, -1, -1):

hidden_error = np.dot(output_error, self.weights[i+1].T) *
self.activations[i](layer_outputs[i], derivative=True)
gradients.insert(0, np.dot(layer_outputs[i].T, hidden_error))
output_error = hidden_error

return gradients

def update_parameters(self, gradients, learning_rate):

CS370: Artificial Intelligence Fall 2023 Page 5

 for i in range(len(self.weights)):
self.weights[i] -= learning_rate * gradients[i]
self.biases[i] -= learning_rate * np.sum(gradients[i], axis=0,
keepdims=True)

import matplotlib.pyplot as plt

def plot_results(history):
plt.plot(history)
plt.title('Training Loss Over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.show()

model = NeuralNetwork(input_size=784, hidden_sizes=[64, 32], output_size=10,

activations=[sigmoid, relu, softmax])
train_neural_network(model, x_train, y_train, x_val, y_val, epochs=10,
learning_rate=0.01)
plot_results(training_loss_history)

def save_model(self, filename):

model_params = {'weights': self.weights, 'biases': self.biases}
with open(filename, 'wb') as file:
pickle.dump(model_params, file)

model.save_model('best_model.pkl')

CS370: Artificial Intelligence Fall 2023 Page 6