Created Wednesday 16 July 2014
The momentum term helps prevent the algorithm from getting stuck in local minimums. Note that there is no guarantee that the momentum works but it has been found to work in practice.
Notation
- Please read the notation page
Calculating the Momentum
- Let `w_(kj)^(**(l))` = the value of the weight from the previous iteration of the backprop algorithm.
- If there is no previous value, let `w_(kj)^(**(l)) = 0`.
Then the momentum term is
`\ \ alpha w_(kj)^(**(l))`
where `alpha` is the momentum parameter.
See also how to pick the momentum parameter