Bearded-android-docs

Created Friday 13 December 2013

Convolution is a term taken from image processing: from a large array of values, take smaller rectangular patches, treat each as inputs to a single neuron, and store the outputs in a smaller array.

A picture says 1,000 words: http://ufldl.stanford.edu/tutorial/index.php/File:Convolution_schematic.gif

• Let `X_n` be an `n xx n` matrix of real-number values.
• Let `X_s` be a smaller matrix of size `(n-s+1) xx (n-s+1)` called a feature map that will be created by sampling patches of size `s xx s` from the larger matrix.
• Each unit (or cell) of `X_s` has a corresponding neuron `(sigma, bb W, b)` where `sigma` is the activation function, `bb W` is the set of weights, and `b` is the bias.
• `bb W` has `s^2` weights: `bb W = (w_1, ... w_(s^2))`.
• The weights are also called the kernels. The set of weights `bb W` is called the kernel.
• The receptive field size of the neuron is `s^2`.
• The neurons all share the same weights i.e., there is a single set of weights for `X_s`.

To calculate the values of `X_s`:

• Let `x_s(i,j)` be the value of `X_s` at position `(i,j)` (row i, column j).
• Let `x_n(i,j)` be the value of `X_n` at position `(i,j)` (row i, column j).
• Let patch `P_s(i,j)` be the sub-array of values of `X_n` of size `s xx s` and top-left corner `(i, j)`. `P_s(i,j)` is a square matrix:

`\ \ \ \ P_s(i,j)
= [(x_n(i,j),x_n(i,j+1),...,x_n(i,j+s))
,(x_n(i+1,j),x_n(i+1,j+1),...,x_n(i+1,j+s))
,(vdots, vdots, vdots, vdots)
,(x_n(i+s,j),x_n(i+s,j+1),...,x_n(i+s,j+s))]`

Even though `P_s(i,j)` is a matrix, think of it as a "flattened" vector of size `s^2`:

`\ \ bb P_s(i,j) = (x_n(i,j),x_n(i,j+1),...,x_n(i,j+s),...,x_n(i+s,j),x_n(i+s,j+1),...,x_n(i+s,j+s))`

Define `x_s(i,j)` as:

`\ \ x_s(i,j) = sigma(bb P_(ns)(i,j) * bb W + b)`

Construction of feature map `X_s` without applying the activation function `sigma` or the bias is called convolution with kernel `bb W` (i.e., if `X'_s` is defined as `x'_s(i,j) = bb P_(ns)(i,j) * bb W`, then `X'_s` is the convolution of kernel `bb W`).

Resources

• http://www.mathworks.com/help/vision/ref/2dconvolution.html