Home
Login
Home
ArXiv
SSRN
Seminars
GitHub

C. Classification Models

Lectures
4
Difficulty
Medium
Homework
Yes
Small Description
Predicting categorical values.
journals
Gallery
Classification Topics

Binary Classification

Bankruptcy Prediction (EX1)
3rd Class Preparation Exercise

Introduction to Classification

In regression, recall that the output yy is a a real number. That is, y∈Ry \in \mathbb{R}. In classification, however, the output is usually an integer. That is, y∈{1,2,…,V}y \in \{1, 2, \dots , V\}, where VV is the number of labels.
All-in-all, classification is mostly like regression. You mostly already know how to do it! The objective function is the same:
min⁡f∈H∑n=1NL(y(n),f(x(n)))+λJ(f).\min_{f \in \mathcal{H}} \sum_{n=1}^{N} L(y^{(n)}, f(x^{(n)})) + \lambda J(f).
•
The only difference is that we need to make some small changes to the function ff and to the loss function LL.
•
We will show that with some small adaptions K-NNs, decision trees, and linear methods all work for classification.

Classification vs. Regression

•
The difference between classification and regression is actually more subtle than it initially seems.
•
Sometimes it can even by quite hard to tell whether you have a classification problem or a regression problem.
We want to predict how many stars new users will give a particular app. Should we use regression or classification? It is actually a difficult choice to make. In some cases, the answer is straightforward:
•
If y∈{1,2}y \in \{1, 2\}, it's better to use classification.
•
If y∈{1,2,…,100}y \in \{1, 2, \dots , 100\}, it's better to use regression.
A. Linear Classification
Medium

Kernel Models

We will talk about kernel methods in an entirely supervised context (regression and classification) although you can also use kernel methods for unsupervised learning.
•
In some respects kernel methods can be thought of as a model somewhere between linear models and nearest neighbors models.
•
All kernel methods (like SVM) uses the kernel trick to effectively transform our dataset to the kernel space and then performs a linear regression in kernel space.
•
The concept of kernels are simple, it attempts to make the data linearly separable so that you can use a linear model, which can be very helpful for classification models.
•
The figure above shows that process for a SVM, a type of kernel model for classification.
•
As part of this class, I just want you to have an intuitive sense for this model is doing.

Components

•
A Kernel is a function that is used to map lower-dimensional data points into higher dimensional, and hopefully, linearly separable data points.
•
There are many types of kernels that you can use to perform this trick, such as Linear kernels, Radial basis function kernels, Polynomial kernel, Sigmoid kernel, etc.
B. Kernel Classification
Medium

K-NN Classification

Consider a case where there are V=5V = 5 possible classes, xx is some information about a student, and yy represents their favorite module in this course.
1.
Regression (r)
2.
Classification (c)
3.
Kernels (k)
4.
Probabilistic (p)
5.
Unsupervised (u)
There is no meaningful order here. Suppose each student has some skill in "math" and "Python", represented by a positive real number, and we want to predict yy based on these attributes. Suppose we have the following data:
Let’s assume that we have the orange query point (new data point) above.
•
What would we predict if K=3? (Remember, K=The number of neighbors)
•
We see that among its 3 nearest neighbors, 'c' is the most common class.
•
Hence, we pick 'c' as our classification.
The algorithm is simply this: Find the nearest neighbors and output the most common class.
C. KNN Classification
Easy

Classification Trees

Just like with regression trees, the prediction function is
f(x)=∑m=1McmI[x∈Rm].f(x) = \sum_{m=1}^{M} c_{m} \mathbb{I} [x \in R_m].
The difference is that now, cmc_m is an integer that represents a class. As before I\mathbb I is the indicator function, and RmR_m is the mthm^\text{th} region. If the function inside the indicator is true it returns a 11 otherwise it returns a 00.
To train, we need to address the following issues:
•
How do we choose the constants cmc_m? The answer is we choose the most common class in RmR_m.
•
How do we grow the tree? Similar to regression trees, we "grow" stumps and then traverse recursively.
•
How do we choose a split? We cannot use the MSE\mathrm{MSE} because we are doing classification, not regression.

How to split classification trees?

We use proxy measures to split the decision trees. Consider the following quantity:
p^mv=1Nm∑x(n)∈RmI[y(n)=v].\hat{p}_{mv} = \frac{1}{N_m} \sum_{x^{(n)} \in R_m} \mathbb{I} [y^{(n)} = v].
p^mv\hat p_{mv} defines the the fraction of the data in region mm that has class vv. How do we know when a region represents a good split in terms p^mv?\hat p_{mv}?
•
Intuitively, we want p^mv\hat{p}_{mv} to have "low randomness" for all mm.
•
For example, suppose that all data in every region had the same class. This would be great!
D. Classification Trees
Medium

Diversity in Models

How can we get some model diversity:

Bagging versus Boosting

We will pay particular attention to Bagging and Boosting methods.

Stacking and Voting

There is two other type of methods called Stacking and Voting.
•
Unlike bagging and boosting, stacking and voting may be (and normally is) used to combine models of different types. (linear models, trees, neural networks)
•
The point of Stacking and Voting is to explore a space of different models for the same problem
•
For Voting, we have simple predefined rules of how the predictions should be added together.
•
With Stacking you don’t have an empirical formula for the weight function, rather it introduces a meta-level (use another model/approach) to establish the model weights.
•
For Stacking the original model’s predictions, becomes features in the new meta-model.

Boosting

E. Ensemble Models
Hard

No Homework

There is no homework for the classification week material, this is simply some code that you can go through that might help you for your future assignments.

Loan Classification:

Finance is full of potential use-cases for classification like predicting market-regimes, predicting whether the stock price will be higher or lower in the next month or the next minute. Here are a few examples where we are credit ratings, loan classifications, and venture capital success.

Multi-classification:

Machine Learning for Company Credit Rating Prediction
•
A colleague was nice enough to provide you with the data from the paper.
Note, I haven’t developed code for this project yet, but I have developed a similar model is the past (but the data is proprietary and can’t be shared). If you are interested in learning from this code you can find it here.

Advanced Bankruptcy Prediction

If you want to start to understand some advanced modelling and data processing steps, you can look through the following example: (you can download the data here). It's just some rough coding, but it will give you an idea of a developer’s thought process.
F. Weekly Example
Medium
TOP