Incomplete Information Dynamic Games

Incomplete Information

Partially Observable Markov Decision Process (POMDP)

\(\mathcal{S}\) - State space
\(T:\mathcal{S}\times \mathcal{A} \times\mathcal{S} \to \mathbb{R}\) - Transition probability distribution
\(\mathcal{A}\) - Action space
\(R:\mathcal{S}\times \mathcal{A} \to \mathbb{R}\) - Reward
\(\mathcal{O}\) - Observation space
\(Z:\mathcal{S} \times \mathcal{A}\times \mathcal{S} \times \mathcal{O} \to \mathbb{R}\) - Observation probability distribution

Alleatory

Epistemic (Static)

Epistemic (Dynamic)

Partially Observable Markov Game

Alleatory

Epistemic (Static)

Epistemic (Dynamic)

Interaction

\(\mathcal{S}\) - State space
\(T(s' \mid s, \bm{a})\) - Transition probability distribution
\(\mathcal{A}^i, \, i \in 1..k\) - Action spaces
\(R^i(s, \bm{a})\) - Reward function
\(\mathcal{O}^i, \, i \in 1..k\) - Observation space
\(Z(o^i \mid \bm{a}, s')\) - Observation probability distribution

Partially Observable Markov Game

Hierarchy of Problems

Belief updates?

Reduction to Simple Game

Pruning in Dynamic Programming

Extensive Form Game

(Alternative to POMGs that is more common in the literature)

Similar to a minimax tree for a turn-taking game
Chance nodes
Information sets

Extensive Form Game

(Alternative to POMGs that is more common in the literature)

Similar to a minimax tree for a turn-taking game
Chance nodes
Information sets

Extensive Form Game

Extensive-form game definition (\(h\) is a sequence of actions called a "history"):

Finite set of \(n\) players, plus the "chance" player
\(P(h)\) (player at each history)
\(A(h)\) (set of actions at each history)
\(I(h)\) (information set that each history maps to)
\(U(h)\) (payoff for each leaf node in the game tree)

King-Ace Poker Example

4 Cards: 2 Aces, 2 Kings
Each player is dealt a card
P1 can either raise (\(r\)) the payoff to 2 points or check (\(k\)) the payoff at 1 point
If P1 raises, P2 can either call (\(c\)) Player 1's bet, or fold (\(f\)) the payoff back to 1 point
The highest card wins

King-Ace Poker Example

4 Cards: 2 Aces, 2 Kings
Each player is dealt a card
P1 can either raise (\(r\)) the payoff to 2 points or check (\(k\)) the payoff at 1 point
If P1 raises, P2 can either call (\(c\)) Player 1's bet, or fold (\(f\)) the payoff back to 1 point
The highest card wins

Extensive to Matrix Form

Exponential in number of info states!

A more interesting example: Kuhn Poker

Fictitious Play in Extensive Form Games

This slide not covered on exam

Heinrich et al. 2015 "Fictitious Self Play in Extensive-Form Games"

Incomplete Information Dynamic Games