![Advantages of Feedforward Neural Networks:
Less complex, easy to design & maintain.
Fast and speedy [One-way propagation].
FNN are used in
linear regression - is used to predict the value of a variable based
on the value of another variable. The variable you want to predict is
called the dependent variable. The variable you are using to predict
the other variable's value is called the independent variable.
classification problems - requires that problem be classified into
one of two or more classes. Example, an email of text can be
classified as belonging to one of two classes: “spam“ and “not
spam“.
Disadvantages of feedforward Neural Networks:
Cannot be used for deep learning [due to absence of dense layers
and back propagation]](https://image.slidesharecdn.com/ann-unit2-231127082650-98cd5fcd/85/ANN-UNIT-2-pptx-61-320.jpg)
ANN - UNIT 2.pptx
- 1. ARTIFICIAL NEURAL NETWORKS UNIT - II Department: AIML Staff Name: P.Vidyasri SRM INSTITUTE OF SCIENCE AND TECHNOLOGY, CHENNAI
- 2. Unit - II Components of artificial neural networks The concept of time in neural networks Components of neural networks Connections Propagation function and network input Activation Threshold value Activation function Common activation functions Output function Learning strategies Network topologies Feedforward networks Recurrent networks Completely linked networks Bias neuron Representing Neurons Orders of activation Synchronous activation Asynchronous activation Input and output of data
- 3. The concept of time in neural networks Current time - (t) Next time step - (t + 1) Preceding one - (t - 1) From a biological point is not very plausible. One neuron does not wait for the next neuron. It significantly simplifies the implementation.
- 4. Components of neural networks Designed based on the inner workings of biological brains. Imitate the functions of interconnected neurons. A technical neural network consists of large number of simple processing units, the Neurons. Neurons are connected to each other by directed communication links, which are associated with weights. Weight is an information used by neural net to solve problems. Weight values can be fixed or may vary. The weight changes indicate the overall performance of neural network.
- 6. Neural Network A neural network is a sorted triple (N, V, w) with two sets N , V and a function w, where N is the set of neurons and V is a set of connections. The function Defines the weights, where w(i, j), the weight of the connection between neuroni andneuron j. The strength of a connection between two neurons i and j is referred as wi,j .
- 7. The weights can be implemented in a square weight matrix W or a weight vector W. With the row number of the matrix indicating where the connection begins. The column number of the matrix indicating, which neuron is the target. The numeric 0 marks a non-existing connection. This matrix representation is also called Hinton diagram.
- 9. Connections Carryinformation, processed by neurons. Data are transferred between neurons via connections. Connections with the connecting weight being either excitatory or inhibitory. Excitatory – Neurons that release neurotransmitters to make the post- synaptic neuron to generate an action potential. Inhibitory - Neurons that release neurotransmitters to make the post- synaptic neuron to generate less action potential. Neurotransmitter: A chemical released from a neuron following an action potential. Travels across the synapse to excite or inhibit the target neuron.
- 10. Propagation function & Network input The propagation function converts vector inputs to scalar network inputs. For a neuron j, The propagation function receives the outputs From other neurons i1, i2, . . . , in (which are connected to j). Transforms them. The connecting weights wi,j into the network input netj. Processed by the activation function. The network input is the result of the propagation function.
- 12. Let I = i1, i2, . . . , in be the set of neurons, such that Then the network input of j, called netj, is calculated by the propagation function fprop as follows: The multiplication of the output of each neuron i by wi,j, and the summation of the results represents netj
- 14. Activation state / Activation The activation is the "switching status" of a neuron. The reactions of the neurons to the input values depend on this activation state. Let j be a neuron. The activation state assigned to j is aj. Indicates the extent of the neuron’s activity. Results from the activation function called activation. Hidden layer performs activation function, sometimes output layer also.
- 16. Threshold value Neurons get activated if the network input exceeds their threshold value Near the threshold value, the activation function of a neuron reacts. From the biological point of view, the threshold value represents the threshold at which a neuron starts firing. Let j be a neuron. The threshold value represents Θj Uniquely assigned to j. Marks the position of the maximum gradient value of the activation function.
- 18. The activation function When comparing with a neuron-based model that is in our brains, the activation function is “what is to be fired to the next neuron”. An activation function is a function that is added into an ANN to help the ”network learn complex patterns in the data”. Activation function decides, whether a neuron should be activated or not by calculating weighted sum and further adding bias with it. Determines the activation of a neuron dependent on network input and threshold value. The activation aj of a neuron j depends on the previous activation state of the neuron and the external input.
- 19. The activation function transforms the network input netj, as well as the previous activation state aj(t - 1) into a new activation state aj(t), with the threshold value Θ. The most important feature in an activation function is its ability to add non-linearity into a neural network. A neural network without an activation function is essentially just a linear regression model with limited abilities.
- 20. Derivatives of an activation function represent a slope on a curve, they can be used to find maxima and minima of functions, when the slope, is zero. Without activation function: Network would be less powerful. Will not be able to learn the complex patterns from the data, including images, speech, videos, audio, etc. This is certainly not what we want from a neural network. With activation function: Make sense of complicated, high dimensional and non-linear big data sets. Extract knowledge from such complicated big data sets.
- 23. A linear activation function takes the form: y = mx It takes the inputs, multiplied by the weights for each neuron, and creates an output signal proportional to the input. A linear activation function has two major problems: Not possible to use backpropagation: It’s difficult to train the model understand. Hard to understand which weights in the input neurons can provide a better prediction. All layers of the neural network collapse into one: Turns the neural network into just one layer. That’s why linear activation function is hardly used in deep learning.
- 25. Modern neural network models use non-linear activation functions. Non-linear functions address the problems of a linear activation function: Non-linear functions allow backpropagation. They allow “stacking” of multiple layers of neurons to create a deep neural network. Multiple hidden layers of neurons are needed to learn complex data sets with high levels of accuracy.
- 27. Common activation functions Binary step/ Heaviside activation function: If the input to the activation function is greater than a threshold, then the neuron is activated. Else it is deactivated; its output is not considered for the next hidden layer. f(x) = 1, x >= 0 = 0, x < 0 The gradient of the binary step function is zero which causes a hindrance in the back propagation process. If you calculate the derivative of f(x) with respect to x, it comes out to be 0. f'(x) = 0, for all x In backpropagation, the derivative of the activation function is used to calculate how much the weights of each neuron need to be adjusted to minimize the error.
- 29. Linear / Identity activation function: The function is defined as f(x) = ax Here the activation is proportional to the input. The variable ‘a’ in this case can be any constant value. The gradient here does not become zero. It is a constant which does not depend upon the input value x at all. This implies that the weights and biases will be updated during the backpropagation process but the updating factor would be the same. The neural network will not really improve the error since the gradient is the same for every iteration. The network will not be able to train well and capture the complex patterns from the data. Hence, linear function might be ideal for simple tasks.
- 31. Sigmoid / Fermi / Logistic activation function: It is one of the most widely used non-linear activation function. Sigmoid transforms the values between the range 0 and 1. f(x) = Unlike the binary step and linear functions, sigmoid is a non-linear function. This is a smooth S-shaped function and is continuously differentiable. The derivative of this function comes out to be f'(x) = (1+e−x) / 2e−x
- 32. The gradient values are significant for range -3 and 3 but the graph gets much flatter in other regions. As the gradient value approaches zero, the network is not really learning. The Fermi function can be expanded by a temperature parameter T into the form The smaller this parameter, the more does it compress the function on the x axis.
- 34. Tanh / Hyperbolic tangent activation function: The tanh function is very similar to the sigmoid function. The only difference is that it is symmetric around the origin. The range of values in this case is from -1 to 1. Thus the inputs to the next layers will not always be of the same sign. Similar to sigmoid, the tanh function is continuous and differentiable at all points. The gradient of the tanh function is steeper as compared to the sigmoid function. Usually tanh is preferred over the sigmoid function since it is zero centred.
- 36. ReLU activation function: ReLU stands for Rectified Linear Unit, a non-linear activation function. The main advantage of using the ReLU function over other activation functions is that it does not activate all the neurons at the same time, computationally efficient than other activation function. This means that the neurons will only be deactivated if the output of the linear transformation is less than 0. f(x)=max(0,x) For the negative input values, the result is zero, neuron does not get activated. f'(x) = 1, x>=0 = 0, x<0 If you look at the negative side of the graph, you will notice that the gradient value is zero. During the backpropagation process, the weights and biases for some neurons are not updated. This can create dead neurons which never get activated. This is taken care of by the ‘Leaky’ ReLU function.
- 39. Output function An output function may be used to process the activation once again. The output function is defined globally. The output function: Calculates the output value oj of the neuron j from its activation state aj. The output function of a neuron j calculates the values which are transferred to the other neurons connected to j. Often this function is the identity, i.e., the activation aj is directly output.
- 40. Learning strategy Learning strategies adjust a network to fit our needs. The learning strategy is an algorithm that can be used to change and thereby train the neural network. So that the network produces a desired output for a given input.
- 41. Network topologies Layers: Input Layer: The number of neurons in the input layer should be equal to the attributes or features in the dataset. It provides information from the outside world to the network. No computation is performed at this layer Nodes just pass on the information to the next layer. N-Hidden Layer: Nodes of this layer are not exposed to the outer world. Hidden layer performs all sort of computation on the features entered through the input layer. Transfer the result to the output layer. Output Layer: This layer brings up the information learned by the network to the outer world.
- 42. Hinton diagram - Feedforward network
- 43. Completely linked neural network: Every neuron is always allowed to be connected to every other neuron, as a result every neuron can become an input neuron. Permit connections between all neurons, except for direct network with laterally recurrent neurons. The matrix W may be unequal to 0 everywhere, except along its diagonal.
- 44. Shortcut connections skiplayers: Connections that skip one or more levels. These connections may only be directed towards the outputlayer. Like the feedforward network, the connections are not directed towards the next layer but towards any other subsequent layer.
- 45. Direct recurrence neural network: Some networks allow neurons to be connected to themselves, which is called direct recurrence or self- recurrence, start and end at the same neuron. We expand the feedforward network by connecting a neuron j to itself, with the weights of these connections being referred to as wj,j. The diagonal of the weight matrix W may be different from 0.
- 46. Indirect recurrence neural network: If connections are allowed towards the input layer, called indirect recurrences. A neuron j can use indirect forwards connections to influence itself. On a feedforward network, now with additional connections between neurons and their preceding layer being allowed. Therefore, below the diagonal of W is different from 0.
- 47. Lateral recurrence neural network: Connections between neurons within one layer are called lateral recurrences. Each neuron often inhibits the other neurons of the layer and strengthens itself. As a result, only the strongest neuron becomes active (winner-takes-all scheme). A laterally recurrent network permits connections within one layer.
- 48. Bias neuron A bias neuron is a neuron whose offset value is always 1. The threshold value is an activation function parameter of a neuron, that indicates the activity of a neuron. The bias does not depend on any input value. A bias neuron is used to represent neuron biases as connection weights, which enables any weight training algorithm to train the biases at the same time. Thresholdvaluesareimplemented asconnection weights and can directly be trained together with the connectionweights, which considerablyfacilitates the learningprocess. In absence of bias, model will train over point passing through origin only. In accordance with real-world scenario, the model should be more flexible.
- 51. Instead of including the threshold value in the activationfunction, it is now included in the propagation function. The processing done by the neuron is: output = sum (weights * inputs) + bias For example, consider an equation: y=mx+c
- 53. If the linear combination is greater or lesser than some threshold value, produces an output of 1 or 0 respectively. The Greek letter Sigma ∑ is used to represent summation, and the subscript i is used to iterate over input (x) and weight (w) pairings. To make things a little simpler for training later, let’s make a small readjustment to the above formula. Let’s move the threshold to the other side of the inequality and replace it with what’s known as the neuron’s bias.
- 54. Effectively, bias = — threshold. The threshold value is subtracted from the network input, i.e. it is part of the network input. We randomly assign numbers for weights and as the neural network trains. It makes incremental changes to those weights to produce more accurate outputs. The threshold shifted to the left side of the equation becomes the bias. Increasing the bias decreases the threshold. Decreasing the bias increases the threshold.
- 56. Feedforward Network Each neuron in one layer has only directed connections to the neurons of the next layer. Every neuron i is connected to all neurons of the next layer. The decision making are based on the current input. It doesn’t memorize the past data, and there’s no future scope. For an example: Give the word "neuron" as an input and it processes the word character by character. By the time it reaches the character "r," it has already forgotten about "n," "e" and "u," which makes it almost impossible for this type of neural network to predict which character would come next.
- 58. To better understand how feedforward neural network’s function, let’s solve a simple problem. Predicting if it's raining or not when given three inputs: x1 - day/night x2 – temperature x3 - month Let's assume the threshold value to be 20 (Actual value), and if the output is higher than 20 then it will be raining, otherwise it's a sunny day. Given a data tuple with inputs (x1, x2, x3) as (0, 12, 11), initial weights of the feedforward network (w1, w2, w3) as (0.1, 1, 1) and biases as (1, 0, 0). Multiplication of weights and inputs: The input is multiplied by the assigned weight values, which this case would be the following: (x1* w1) = (0 * 0.1) = 0 (x2* w2) = (1 * 12) = 12 (x3* w3) = (11 * 1) = 11
- 59. Adding the biases: The product found in the previous step is added to their respective biases. The modified inputs are then summed up to a single value. (x1* w1) + b1 = 0 + 1 (x2* w2) + b2 = 12 + 0 (x3* w3) + b3 = 11 + 0 Transfer function: An transfer function is the mapping of summed weighted input to the output of the neuron. Weighted sum = (x1* w1) + b1 + (x2* w2) + b2 + (x3* w3) + b3 = 24 (Predicted value) It governs the inception at which the neuron is activated and the strength of the output signal. Output signal: The weighted sum in our example is greater than 20, the perceptron predicts it to be a rainy day.
- 60. Calculating the Loss or cost function: A loss function quantifies how “good” or “bad” a given model is in classifying the input data. The loss is calculated as the difference between the actual output and the predicted output. Cost or Loss function= Y_{predicted/expected} - y_{original/actual} The function that is used to compute this error is known as loss function or cost function.
- 61. Advantages of Feedforward Neural Networks: Less complex, easy to design & maintain. Fast and speedy [One-way propagation]. FNN are used in linear regression - is used to predict the value of a variable based on the value of another variable. The variable you want to predict is called the dependent variable. The variable you are using to predict the other variable's value is called the independent variable. classification problems - requires that problem be classified into one of two or more classes. Example, an email of text can be classified as belonging to one of two classes: “spam“ and “not spam“. Disadvantages of feedforward Neural Networks: Cannot be used for deep learning [due to absence of dense layers and back propagation]
- 62. Introduction to Recurrent Neural Network (RNN) RNNs are a powerful and robust type of neural network. Belong to the most promising algorithms in use because it is the only one with an internal memory. Like many other deep learning algorithms, recurrent neural networks are relatively old. They were initially created in the 1980’s, but only in recent years have we seen their true potential. An increase in computational power along with the massive amounts of data that we now must work with, and the invention of long short- term memory (LSTM) in the 1990s.
- 63. Because of their internal memory, RNN’s can remember important things about the input they received, which allows them to be very precise in predicting. The nodes in different layers of the neural network are compressed to form a single layer of recurrent neural networks. A, B, and C are the parameters of the network. Works on the principle of saving the output of a particular layer and feeding this back to the input in order to predict the output of the layer. Unlike FNN, in RNN the output of the network at time “t” is used as network input at time “t+1”.
- 64. FNN vs RNN Few issues in the feed-forward neural network: Cannot handle sequential data. Depends on the current input. Cannot memorize previous inputs. Bad at predicting what’s coming next. Recurrent neural networks: an algorithm for sequential data, used by Apple's Siri and Google's voice search. the first algorithm that remembers its input, due to an internal memory. When it makes a decision, it considers the current input and what it has learned from the inputs it received previously.
- 66. Types of Recurrent Neural Networks One to One RNN: This type of neural network is known as the Vanilla Neural Network. It's used for general machine learning problems, which has a single input and a single output.
- 67. Image classification: • Single-label image classification is a traditional image classification problem where each image is associated with only one label or class. • For instance, an image of a cat can be labeled as “cat” and nothing else.
- 68. One to Many RNN: This type of neural network has a single input and multiple outputs. An example of this is the image caption.
- 69. Image Captioning: Here, let’s say we have an image for which we need a textual description. We have a single input – the image, and a series or sequence of words as output.
- 70. Many to One RNN: This RNN takes a sequence of inputs and generates a single output. Sentiment analysis is a good example of this kind of network where a given sentence can be classified as expressing positive or negative sentiments.
- 71. Sentiment Classification: This can be a task of simply classifying tweets into positive and negative sentiment. The input would be a tweet of varying lengths, while output is of a fixed type and size.
- 72. Many to Many RNN: This RNN takes a sequence of inputs and generates a sequence of outputs. Machine translation is one of the examples.
- 73. Language Translation: Given an input in one language, RNNs can be used to translate the input into different languages as output. The number of inputs and outputs do not match, e.g., in language translation we pass in “n” words in English and get “m” words in Italian.
- 74. Understanding a Recurrent Neuron in detail An RNN remembers each and every information through time. It is useful in time series prediction only because of the feature to remember previous inputs. Let’s take a character level RNN where we have a word “Hello”. We provide the first 4 letters i.e., h,e,l,l and ask the network to predict the last letter i.e., ’o’. The vocabulary of the task is just 4 letters {h,e,l,o}. In real case scenarios involving natural language processing, the vocabularies include the words in entire Wikipedia database, or all the words in a language. This is called Long Short-Term Memory
- 76. Let’s see how the above structure be used to predict the fifth letter in the word “hello”. The letter “h” has nothing preceding it, let’s take the letter “e”. So at the time the letter “e” is supplied to the network, a recurrence formula is applied to the letter “e” and the previous state which is the letter “h”. These are known as various time steps of the input. So if at time “t”, the input is “e” and at time “t-1”, the input was “h”. The recurrence formula is applied to “e” and “h” both and we get a new state. The formula for the current state can be written as: ht - Current hidden state ht-1 - Previous hidden state xt - Current input state
- 77. We have four inputs to be given to the network, during a recurrence formula, the same function and the same weights are applied to the network at each time step. Taking the simplest form of a recurrent neural network, let’s say that the activation function is tanh. We can write the equation for the state at time “t” as: whh - Weight at recurrent neuron wxh - Weight at input neuron Once the final state is calculated we can go on to produce the output We can calculate the output state as: Yt - Output Why- Weight at output layer
- 78. Let me summarize the steps in a recurrent neuron: • A single time step of the input is supplied to the network i.e. xt is supplied to the network. • We then calculate its current state using a combination of the current input and the previous state i.e. we calculate ht • The current ht becomes ht-1 for the next time step. • We can go as many time steps as the problem demands and combine the information from all the previous states. • Once all the time steps are completed the final current state is used to calculate the output yt • The output is then compared to the actual output and the error is generated. • The error is then backpropagated to the network to update the weights and the network is trained.
- 81. Representing neurons We can either write its name or its threshold value into a neuron. Neurons can represent their type of data processing.
- 82. Order of activation It is very important in which order the individual neurons receive and process the input and output the results. Distinguished int two model classes: Synchronous activation: All neurons change their values synchronously. All neurons of a network calculate network inputs at the same time by means of the propagation function, activation by means of the activation function and output by means of the output function. Activation cycle iscomplete. Asynchronous activation: The neurons do not change their values simultaneously but at different points of time. There exist different orders.
- 83. (i) Random order: With random order of activation, a neuron i is randomly chosen and its neti, ai and oi are updated. Some neurons are repeatedly updated during one cycle, and others, however, not at all. This order of activation is not always useful. (ii) Random permutation: Each neuron is chosen exactly once, but in random order, during one cycle. Initially, a permutation of the neurons is calculated randomly and therefore defines the order of activation. Then the neurons are successively processed in this order. For all orders either the previous neuron activations at time t or, if already existing, the neuron activations at time t + 1, for which we are calculating the activations, can be taken as a starting point.
- 84. (iii) Topological order: The neurons are updated during one cycle and according to a fixed order. This order is defined by network topology. This procedure can only be considered for non-cyclic, i.e.,non- recurrent networks, since otherwise there is no order of activation. In feedforward networks, the input neurons would be updated first, then the inner neurons and finally the output neurons. Given the topological activation order, we just need one single propagation. (iv) Fixed orders ofactivation during implementation: When implementing feedforward networks, for example, it is common practice to establish the activation order once based on the topology and then apply that order without further verification during runtime. But this is not necessarily useful for networks that are capable to change their topology.
- 85. Input and output data Many types of neural networks permit the input of data. These data are processed and can produce output. As an example, if feedforward network has two input neurons and two output neurons, then Input vector: Data is put into a neural network by using the components of the input vector as network inputs of the input neurons. A networkwith“n” inputneuronsneeds“n” inputs x1, x2, . . . , xn. They are considered as input vector x = (x1, x2, . . . , xn). Therefore, the input dimension isreferred to as “n”. Output vector: Data is output by a neural network by the output neurons adopting the components of the output vector in their output values. A network with “m” output neurons provides “m” outputs y1, y2, . . . , ym. They are regarded as output vector y = (y1, y2, . . . , ym). Therefore, the output dimension is referred to as “m”.
- 86. • The derivative of the activation function determines how the neuron's activation changes concerning changes in its input. • This, in turn, affects how much the weights of the connected neurons should be updated during training.