I. Deep Learning

To repeat: all neural networks can be divided into two parts: a forward propagation phase, where the information "flows" forward to compute predictions and the error;

And the backward propagation phase, where the backpropagation algorithm computes the error derivatives and update the network weights.

For one it is important to know we don’t perform the loops that the summation notation might imply.

Loops are known to be highly inefficient computationally, so we will want to avoid them.

Fortunately, we can use matrix operations (i.e. vectorization) to achieve the exact same result much faster.

In the example below the we don’t perform 12 different operations, but simply 1 operation.

The standard form of dimensionality reduction technique we might use is PCA to shrink the features from a 2D array (matrix) of 100 feature to 5 features by forming an efficient representation on a new 2D array (represented on the bottom right).

The techniques for 3D and 4D arrays are not widely used, one is called Matrix Product State, these would allow you to go from 3D→3D and 4D→4D.

Since the development of neural networks a models known as an Autoencoder has become popular to learn an efficient lower-dimensional representation of the data within the same rank.

Autoencoders are very flexible non-linear decomposition methods and can work with tensors of any rank, including rank two (2D), three (3D), and four (4D) tensors.

You can move from a 4D tensor (tesseract) to a 3D tensor using Matrix Product State.

You can move from 3D (tensor) to 2D (matrix) using the Tucker and CANDECOMP decompositions.

We can also use PCA to go from a 2D array to a 1D array, by only choosing the first principal component.

It wasn’t long until developers from finance realized that you could use the same format above to solve problems in finance.

Tucker Balch (JP Morgan AI Research), created a large sample of financial time-series images encoded as candlestick (Box and Whisker) charts.

They labeled the samples following three algebraically-defined binary trade strategies (Murphy, 1999).

The authors realized that there are various ways in which one can “visualize” stock market data.

We can also add a lot of technical feature and let the CNN find the appropriate direction that the technical indicator identified.

Moreover, we can vary the image resolution (pixels) as a robustness test to see the variation in performance.

Here the purpose was not to predict the direction of the stock, but instead see if they can capture the labels of technical indicator methods.

The non-linear equivalent of the ARIMA model is called a Recurrent Neural Network (RNN)

Below is how you can convert a Feed-Forward (Multilayer Perceptron) Neural Network into a Recurrent Neural Network.

When an RNN model has to calculate the new hidden state it uses feature inputs like FFN, but also the previous hidden state.

The FNN (MLP) model is the standard NN that purely relies on the input without taking previous information into account for subsequent predictions.

However, we can do something more interesting, we can force several $z$ values to become parameters on a distribution $\mathcal{N}\left(\mu, \sigma^{2}\right)$ instead of simple unconstrained scalar values.

The mean $\mu$ and variance $\sigma^{2}$ together forms its own normal probability distribution that we can use to sample a new latent space from.

Sampling from the distribution creates sample that can be decoded and turned into new outputs.

Discriminative models (nets) like CNNs or MLPs attempt to learn the probability of $Y$, whereas Generative models (nets) like VAEs and GANs attempt to learn a simulator of the data $X$.

Probabilistically we can say generative models attempt to model the full joint distribution $P(X,Y)$, whereas discriminative models want the posterior probability $P(Y, X)$.