Home
Login
Home
ArXiv
SSRN
Seminars
GitHub

E. Preprocessing Phase

Lectures
6
Difficulty
Easy
Homework
No
Small Description
Preparing data for modelling.
journals
Gallery
Preprocessing Methods

Exploratory Data Analysis (EDA)

The package is built on the original open source project ‣ and although some of these methods help you perform EDA, they can impossibly cover all the operations.
You can explore different packages yourself for example have a look at Klib and Dtale (Demo).

Summary Statistics

You can do more to your data than simply df.describe() in Python or summary(df) in R. Perhaps you can run slightly more advanced stats like that found on datatile.
A. Exploratory Data Analysis

Wrangling

•
The end result of the data wrangling process is to get data into machine readable format, which normally means transforming all the data to numbers, and to put all the numbers into a matrix format.
•
We can break data wrangling (munging) into four different tasks.
•
Data wrangling is often a subjective procedure –there are no hard set rules. It requires a lot of empirical experience to understand datasets and how ML algorithms will be affected by various cleaning procedures
Given the above steps, we can define wrangling as all the data preprocessing necessary before we start augmenting structured data.

1. Convert Unstructured Data

Text Vectorization

Structured data consists of formatted data with clearly defined data types that can fit into vector space (e.g., excel spreadsheets), whereas unstructured data consists of sensors, text files, audio and video files, etc. (your downloads folder).
•
The predominant type of unstructured data that we would deal with in finance is text files.
•
We will touch on this topic more in the NLP module, so we will now just show some simple methods to convert unstructured data into structured data.
•
The techniques used to turn Text into features can be referred to as Text Vectorization techniques, since they all aim at one purpose: turning text into vectors.
•
Before you turn the text into number, you might first identify some issues (errors) that have to be fixed, so that the number values correctly represents the dataset.
B. Data Wrangling
Data augmentation because it is not just features, but also labels, augmentation because it is not just feature engineering, but also data generation, sampling, and preprocessing methods. There are five data augmentation techniques that we will discuss.
Interacting (Many to One Column)
3.
Mapping (Many to Many Column)
4.
Extracting (Column/s to Row/s)
5.
Synthesizing (Many to Many Rows) 
•
There is another type called Exploding (One to Many Columns) that is not included, for example the Signitures Method can convert a single feature into multiple features.
•
Currently we are also only focusing on a two dimensional dataset, there are additional extraction methods for higher dimensional datasets.
•
Some methods can create a three dimensional dataset out of a two dimensional dataset like Recurrence Plots, Gramian Angular Fields, and Markov Transition Fields.
•
And some methods can create two dimensional dataset out of three dimensional datasets like the Tucker and CANDECOMP decomposition.
•
It is unlikely that you would ever have to deal with these augmentation methods, so we ignore them here.
•
We are also only looking at numerical augmentation methods and not audio, text, or image augmentation methods.
C. Data Augmentation

Feature Selection

Feature section is a column to column data augmentation technique, we have isolated it here, because it is a very important augmentation method.
There are a thousands ways to select features, one of the most common methods is to select all the features and to let the model decide what is the most important features.
You could also use your domain experience and select the ones that you think matters from your personal experience, or based on past research.
However there are several reasons why you might still want to perform data-driven feature selection methods, it offers a:

Three Methods

Embedded Methods

Filter Methods

Wrapper Methods

Conclusion

•
Filter methods measure the relevance of features by their correlation with dependent variable while wrapper methods measure the usefulness of a subset of feature by actually training a model on it.
•
Filter methods are much faster compared to wrapper methods as they do not involve training the models. On the other hand, wrapper methods are computationally very expensive as well.
•
Greedy filter methods are then also much faster than non-greedy methods, but can lead to substandard outcomes.
D. Feature Selection
TOP