Policy and Value Iteration

Last Time

• How is a Markov decision process defined?

• What is a policy?

• How do we evaluate policies?

Guiding Questions

• How do we reason about the future consequences of actions in an MDP?

• What are the basic algorithms for solving MDPs?

Value-Based Policy Evaluation

Dynamic Programming and Value Backup

Policy Iteration

Algorithm: Policy Iteration

Given: MDP \((S, A, R, T, \gamma, b)\)

1. initialize \(\pi\), \(\pi'\) (differently)
2. while \(\pi \neq \pi'\)
3.     \(\pi \gets \pi'\)
4.     \(U^\pi \gets (I - \gamma T^\pi )^{-1} R^\pi\)
5.     \(\pi'(s) \gets \underset{a \in A}{\text{argmax}} \left(R(s, a) + \gamma \sum_{s' \in S} T(s' | s, a) U^\pi (s') \right) \quad \forall s \in S\)
6. return \(\pi\)

Value Iteration

Algorithm: Value Iteration

Given: MDP \((S, A, R, T, \gamma, b)\), tolerance \(\epsilon\)

1. initialize \(U\), \(U'\) (differently)
2. while \(\lVert U - U' \rVert_\infty > \epsilon\)
3.     \(U \gets U'\)
4.     \(U'(s) \gets \underset{a \in A}{\text{max}} \left(R(s, a) + \gamma \sum_{s' \in S} T(s' | s, a) U (s') \right)  \quad \forall s \in S\)
5. return \(U'\)

• Returned \(U'\) will be close to \(U^*\)!
• \(\pi^*\) is easy to extract: \(\pi^*(s) = \arg\max( R(s, a) + \gamma E[U^*(s)])\)

Bellman's Equations

Guiding Questions

• How do we reason about the future consequences of actions in an MDP?

• What are the basic algorithms for solving MDPs?