F. Modeling Phase

Most data arrive without a clean label and you either have to extract a label from datasets or hand-develop your own labels.

That doesn’t mean that we will blindly use the stock price other steps might be involved:

In previous sections we have looked at developing features $X$ in such a way that the model can best extract patterns from the data.

In the past we have shown that one broadly has two types of supervised machine learning labels:

Model selection spans the function to be learned (random forest, neural network), the objective function to optimize (log loss, mean squared error), and the learning parameters (hyperparameters).

Bias — The bias of a model is the difference between the expected prediction and the correct model that we try to predict for given data points.

Variance — The variance of a model is the variability of the model prediction for given data points.

Bias/variance tradeoff — The simpler the model, the higher the bias, and the more complex the model, the higher the variance.

With model validation we want to validate the accuracy, robustness, and explainability of the model.

Every modeller has to decide how much weight to give to each of the validation criteria types:

Quality of Model = Accuracy + Robustness + Explainability:

In general it is a trade-off — more accurate models could be less robust and explainable, and visa versa, more robust and explainable models could be less accurate.

In a previous lecture we have shown that we can validate the performance and quality of a model by taking the measures against validation/development data, there are two main methods.

In a context of a binary classification, here are the main metrics that are important to track in order to assess the performance of the model.

The confusion matrix is used to have a more complete picture when assessing the performance of a model, here we are predicting whether the sentiment of the CEO in the earnings call is Malicious or Benign.