K. Reinforcement Learning

(1) Uncertainty means that the problems under consideration involve random variables that evolve over time; the technical term for this is Stochastic Processes.

Random variables that evolve over time can be found in nature (e.g., weather) and in business (e.g., customer demand or stock prices).

(2) Optimal decisions refers to using optimization methods to maximize a well-defined goal.

(3) Sequential refers to the fact that as we move forward in time, as the random variables evolve the optimal decisions have to be adjusted to the "changing circumstances."

All problems in Reinforcement Learning are theoretically modeled as maximizing the return in a Markov Decision Process, or simply, an MDP. An MDP is characterized by 4 things:

A Markov decision process is often denoted as $\mathcal{M}=\langle\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}\rangle$. Let us now look into them in a bit more detail (Sometimes $\mathcal{T}$ is used for the transition function, instead of $\mathcal{P}$).

$\mathcal{S}$ describes the legitimate values the state can take at time $t$ is denoted as $S_{t} \in \mathcal{S}$ where $\mathcal{S}$ is referred to as the state space.

The set of states, $\mathcal{S}$, can be either discrete or continuous in value.

An example of a finite discrete state space is the state we used for the Atari game, i.e. the past 4 frames.

There are three reasons why solving a MDP is fundamentally a hard problem.

Taking many complex financial factors into account, DRL trading agents build a multi-factor model and provide algorithmic trading strategies, which are difficult for human traders to execute and understand.

By now you should know that there are five elements to any sequential decision problem.  Before you start developing a mathematical model, you need to identify three of these (no analytics needed):

WIdely overlooked in the stochastic optimization community is the power of the methods that are most widely used in practice:

PFAs - some sort of parameterized model (e.g. order-up-to, buy low, sell high) CFAs - Some form of parameterized optimization problem Deterministic DLAs - such as google maps. Note that deterministic lookaheads can be useful in some settings (such as google maps), but you can almost always make deterministic lookaheads even better by introducing tunable parameters.

In effect, all of these involve some form of tunable parameter. Tuning can be hard. Normally we prefer to do this in a simulator, but simulators can be hard to build. Without a simulator, most of the time we are using the intuition of someone who understands the problem.

I first highlighted the importance of modeling uncertainty as the foundation for developing flexible supply chains that can respond to uncertainty. Then, after listing the five dimensions of every sequential decision problem, I focused on the first four: