The Application of Stock Index Price Prediction with Neural Network

There are 4 possible methods of machine learning to perform stock index price position, including 3 models: Multilayer Perceptron (MLP), Long Short Term Memory (LSTM), Convolutional Neural Networks(CNNs) and an attention-based neural network.

The data contains:

SP500

CSI300

and Nikkei225 Index from 3 different financial markets. There are 7 variables chosen as the inputs containing daily training data, technical indicators and macroeconomic variables. All the introduced models have better accuracy in the financial market than the developing index. 

Most capital is traded around the world in the stock market. One can take advantage of the market if they have the proper models to predict stock price, which is represented as a financial time series. Time series is a chronological sequence of observed behavior, and it aims at building models to simulate futures based on past values. Past and future behavior is not deterministic.

p(Xt+d|Xt, Xt-1, ...) = f(Xt, Xt-1, ...) 

Financial time series are primarily based on nonlinear relationships. Artificial Neural Networks (ANN) and Support Vector Regression (SVR) tend to have good accuracies. Combing deep learning networks with financial prediction is regarded as one of the most charming topics in the stock market.

These are the following 4 networks again: 

1. Multilayer Perceptron (MLP)

2. Long Short Term Memory (LSTM)

3. Convolutional Networks (CNN)

4. Uncertainty-aware Attention (UA). 

LSTM is a type of Recurrent Neural Network (RNN). 

LSTMs are composed of cell, input, output, and forget gate. It remembers the values given over time interval and the 3 gates that regulate the flow of information into and out of the cell. It is very popular in image processing and time series predictions, and were made to deal with vanishing gradients.

Convolution contains convolution, pooling, and fully connected layer, with a nonlinear activation function between each layer. It uses convolution, which is a linear operation in place of matrix multiplications, and it can reduce the complexity of a network.

UA is an attention-based model using RNNs as its basic structure, using 2 RNNs with raw time series data and generates temporal attention and variable attention using inference. This article compares 4 of these methods.

You can have Google Domestic Trends as macroeconomic factors in order to model the SP500 index. 

The researchers collected 25 domestic trends, considered as representation of the public. The researchers determined and normalized the window size and determined the observation window using the LSTM model. industries. 

The dataset is taken from the Automobile, Banking, and IT They compare this to ARIMA and GARCH models (other comparisons), using them as benchmarks. There were also many specificities used in terms of performing these models (Kenma-Vorst, Levy Approximation, etc.)

The price of financial markets are solely based on news and events. Using NLP in a deep neural network drove the market prediction and made a 6% prediction as compared to state-of-the-art baseline methods.

A wavelet transform eliminates noise by decomposing the stock price series. Stacked autoencoders are then used to extract features from the data. These contain the input layer, the hidden layer, and the output layer. Every Autoencoder was trained using the same gradient descent algorithm. LSTMs finally predict stock prices. This was the work of one researcher. 

Another method is by using a convolutional neural network. They might not reflect the time domain information properly. This neural network performs binary classification, and the output has either 1 or 0 with 1 representing the stock price going up and 0 representing the stock price going down. This also yields a decent performance in addition to RNNs. 

The SOCNN, or significant offset convolutional neural network is a combination of AR model and CNN. It helps compute asynchronous time series and it is the weighted sum of adjusted coefficients, depending mainly of the input of the network. One network represents the significance of observed data and the other represents predictors that are independent of position in time. We want to determine the significance between data that is/is not time significant. 

A temporal convolutional network uses a hierarchy of convolutional networks to perform segmentation. There are temporal convolutional filters that can cover long-range patterns, and update the parameters over a series of time steps.

Anther researcher proposed a attention-based recurrent neural network. The authorse select and encode a series that affects the outcome the most, by referring to the previous state. Then, the model will select relative states with designed temporal addition mechanism, and the model can then choose the most related input variables of a time series. 

Now to the methodology part. 

MLPs contain a finite number of continuous layers an has at least 3 layers: an input layer, a hidden layer, and an output layer. We can subsequently have a set of input vectors into a set of output vectors, which consists of layers to the next layer.

Due to the vanishing gradient, it can be hard for RNN to learn long-term dependencies, since they have a repeated chain in one neural cell.

LSTMs use memory cells, and as a result is a solution for vanishing gradients. LSTMs have 4 layers that are interacting with each other. 

Ct is the cell state of a LSTM.

The cell state runs through a chain keeping information flow unchanged and an LSTM can delete or modify information of a cell state regulated by a gate mechanism, a way for information to pass selectively. Gates have both a sigmoid neuron layer and a a multiplication layer pointwise. 

Every gate has expression σ(WiX + bi). Here is the picture of the LSTM: 

Step 1 is to decide that to discard from the cell state. This decision is made by a 'forget gate layer' and outputs a vector ranging from 0 to 1 for a certain cell state Ct-1. It keeps the information if the value equals to 1 and discard the information if the value equals to 0, represented by the following information: 

ft = σ(Wf * [ht-1, xt] + bf) 

The next layer that I want to discuss is the input gate layer. This is where one sigmoid layer determines the updated values and one tanh layer creates a new candidate value that will be added to the new state information.  (p. 6) The I and Ct levels will subsequently be used to update the state at hand.

it =  σ(Wi * (ht-1, xt) + bi)

~Ct = tanh(Wc * [ht-1, xt] + bc)

Ct = ft * Ct-1 + it * ~Ct. 

* represents the element-wise product.

The last thing I want to determine is the output gate layer, which is obviously determining directly what the output is. The following denotes the output equations:

ot = σ(W0 * [ht-1, xt] + b0)

ht = ot * tanh(Ct).

A fully connected layer generates predictions based on the previous layers.

Convolutional neural networks are usually applied in image processing, but they can be applied in terms of the 2D series time data in this case. There are 2 important properties that reduce the number of parameters. Each neuron has a local perceptron and combines local information at a higher level, and each neuron acts as a filter to extract features from local input data. 

 

The filter slides all over the input data to capture the local information by capturing the convolutional information on each path and different convolutional kernels help to generate various different types of information.

Pooling helps to reduce the number of data and parameters to avoid overfitting. It compresses features to the maximum value or mean value of a specific area. Fully connected layers go through convolutional layers, pooling layers several times to get optimal results through dimensionality reduction and feature extraction. 

The last topic that I want to talk about is uncertainty-aware attention. This is an attention-based models that uses RNNs to generate all sorts of attention. The 2 types of attention weights are assumed as Gaussian distribution with input dependent noise produced with 2 RNNs, generating attentions with small variance when the contribution of each features are estimated and it is confident in this estimate. We want to generate 2 attentions within timesteps α and features β.

gi, ..., g1 = RNNα(vi, ..., v1, ω),  hi, ... h1 = RNNβ(vi, ..., v1; ω)

ej = wα(transpose)gj + bα for all j = from 1 to i. 

α1, ..., αi = Softmax(e1, ... ei)    βj = tanh(dj) 

2 Attentions are produced through softmax and tanh, with e and d as logits generated from outputs g and h. 

c = Σ (j = 1 to n) αjβj ⊙ vj

^y = fc(c), fc is fully connected layer.

3 stock indices are chosen from 3 different models with 8 years of time period collected. We should ideally choose datasets from different market conditions. 

One dataset is the CSI300 from mainland China, considered as a developing market. We choose 7 variables as inputs and divide them into multiple sets. They will also test on the S&P 500 index and the Nikkei225 financial index from the Tokyo Market. We choose 7 variables and divide them into 3 sets.

Here are the variables: 

Daily Trading Data

Open/Close Price (The daily trading open/closing price)

Trading Volume (Daily Trading Volume)

Technical Indicators

MACD - Moving average convergence divergence, trend following characteristics and momentum

ATR - Measures the volatility of the price

Macroeconomic Variables

Exchange Rate - US Dollar Index

Interest Rate - Interbank Offered Rate

90% of the data are used to train models and 10% of the data are used to test the models.

There are 5 hidden layers with the RELU activation function. The sigmoid and Tanh function are used in the LSTM model. For CNN, each convolution layer has the same kernel size of 3, and for all methods, time step is 20 learning rate is 0.001 and the Mean-squared error is used as the measurement for performance. This uses PyTorch and the loss function implements Adam.

Performance is calculated with the Mean Absolute percentage error, the root mean squared error, and the correlation coefficient. 

yi is the original series and _yi is the mean of the original series, where ^yi is the predicted series and _^yi is the mean of the predicted series. MAPE measures size of error compared to relative average of the error.

The attention-based mode surprisingly achieved the best results in terms of financial time series prediction, followed closely by the convolutional neural network in all metrics. The predicted results yields promising data. 

The top represents the forecast period and the bottom represents the fitted time series. We also have evidenced that a developed financial market is much less susceptible to noise than underdeveloped markets. 

Predicting series is tough due low signal-noise ratio. We might consider combining linear and non-linear aspects in the time series. A problem researchers can face is selecting proper indicators. An appropriate model is constructed using an unsupervised learning method so the model can extract vital information to reduce dimensions and parameters for training.