Advanced topics in artificial neural networks
- 1. SWPNA.C Asst.Prof. IT Department SriDevi Women’s Engineering College
- 2. ADVANCED TOPICS IN ARTIFICIAL NEURAL NETWORKS Alternative Error Functions: Adding a penalty term for weight magnitude. we can add a term to E that increases with the magnitude of the weight vector. This causes the gradient descent search to seek weight vectors with small magnitudes, thereby reducing the risk of over fitting. One way to do this is to redefine E as Each weight is multiplied by the constant upon each iteration. Swapna.C
- 3. Adding a term for errors in the slope, or derivative of the target function. Minimizing the cross entropy of the network with respect to the target values. Swapna.C
- 4. Altering the effective error function can also be accomplished by weight sharing, or "tying together" weights associated with different units or inputs. The idea here is that different network weights are forced to take on identical values, usually to enforce some constraint known in advance to the human designer. The various units that receive input from different portions of the time window are forced to share weights. The net effect is to constrain the space of potential hypotheses, thereby reducing the risk of over fitting and improving the chances for accurately generalizing to unseen situations. Swapna.C
- 5. Alternative Error Minimization Procedures One optimization method, known as line search, involves a different approach to choosing the distance for the weight update. A second method, that builds on the idea of line search, is called the conjugate gradient method. Here, a sequence of line searches is performed to search for a minimum in the error surface. On the first step in this sequence, the direction chosen is the negative of the gradient. On each subsequent step, a new direction is chosen so that the component of the error gradient that has just been made zero, remains zero. Swapna.C
- 6. Recurrent Networks Recurrent networks are artificial neural networks that apply to time series data and that use outputs of network units at time t as the input to other units at time t + 1. One limitation of such a network is that the prediction of y(t + 1 ) depends only on x(t) and cannot capture possible dependencies of y (t + 1 ) on earlier values of x. Swapna.C
- 7. Swapna.C
- 8. Dynamically Modifying Network Structure One idea is to begin with a network containing no hidden units, then grow the network as needed by adding hidden units until the training error is reduced to some acceptable level. The CASCADE- CORRELATION Algorithm (Fahlman and Lebiere 1990) is one such algorithm. CASCADE-CORRELATIONS by constructing a network with no hidden units. Swapna.C
- 9. second idea for dynamically altering network structure is to take the opposite approach. Instead of beginning with the simplest possible network and adding complexity, we begin with a complex network and prune it as we find that certain connections are inessential. One way to decide whether a particular weight is inessential is to see whether its value is close to zero. A second way, which appears to be more successful in practice, is to consider the effect that a small variation in the weight has on the error E. Swapna.C