Value Iteration and Policy Iteration - Model Based Reinforcement Learning Method - Machine Learning

RL09 Value Iteration and Policy Iteration Model Based Reinforcement Learning Machine Learning

Model Based Reinforcement Learning

In model-based reinforcement learning algorithm, the environment is modelled as a Markov Decision Process (MDS) with following elements:

* A set of states
* A set of actions available in each state
* Transition probability function from current state (st) to next state (st+1) under action a
* Reward function: reward received on transition from current state (st) to next state (st+1) under action a

There are two common approaches to find optimal policy using recursive relation of Bellman Equation

1. Value Iteration: In this method, the optimal policy is obtained by iteratively computing the optimal state value function V(s) for each state until it converges. In this method policy function in not computed explicitly during iteration, rather optimal state value function is computed by choosing the action that maximizes Q value for a given state.

Algorithm

2. Policy Iteration: In this method, we start with a baseline policy and improve it iteratively to obtain optimal policy. There are two steps

1. Policy evaluation: in this step we evaluate the value function for current policy
2. Policy improvement: in this step policy is improved at each step by selecting the action that maximizes the Q value.

Algorithm

Shortcomings of Value Iteration and Policy Iteration Methods

1. These methods are computationally feasible only for finite small Markov Decision Processes, i.e., small number of time steps and small number of states.

2. These methods cannot be used for games or processes where model of environment, i.e., Markov Decision Process, is not known beforehand. Rather than the model we are given a simulation model of the environment and the only way to collect information about the environment is by interacting with it.

Artificial intelligence
Machine learning
Learning from data
Learning paradigms
Reinforcement learning
Environment
Agent
State
Action
Policy
Reward
Discount rate
Value
State value
Value function
State value function
Action value
Q value
Markov property
Process
Stochastic process
Markov process
Discrete time
State transition probability
Matrix
step
Model Based Reinforcement Learning
Markov Decision Process
Value Iteration
Policy Iteration
Policy Evaluation
Policy Improvement