Games

Multi-agent interaction

Up to this point:

  • Stochastic Uncertainty
  • Optimization

This week:

  • Interaction Uncertainty
  • Game Solution Concepts

 

 

Kriegspiel

Outline

  1. Matrix Games
    1. Linear Programming
    2. Regret Matching
  2. Turn-taking Games
    1. Minimax Trees
    2. \(\alpha\)-\(\beta\) Pruning
    3. Monte Carlo Tree Search
  3. Imperfect Information Games
    1. Extensive-form Games
    2. Counterfactual Regret Minimization

Sources:

  • AI: A Modern Approach by Russel and Norvig
  • "An Introduction to Counterfactual Regret Minimization in Games" by Neller and Lanctot

Game Solution Concepts: Nash and Correlated Equilibria

Battle of the Sexes

Rock Paper Scissors

Definitions:

  • Strategy/Policy: Probability distribution of actions for a player
  • Strategy Profile: Set of strategies for all players
  • Nash Equilibrium: A strategy where there is no incentive for any player to unilaterally change strategy
  • Correlated Equilibrium: Strategy profile with a shared random source

 

Calculating Mixed-Strategy Equilibria

By hand

\[\sigma_\text{Gary}(M) = x\]

 

\(U_\text{Monica}(\sigma_\text{Gary}, M) = 2x\),

\(U_\text{Monica}(\sigma_\text{Gary}, G) = 1-x\)

If \(x=1/3\), these are equal, so Monica is indifferent. By symmetry, \(\sigma = \{(\frac{2}{3},\frac{1}{3}), (\frac{1}{3},\frac{2}{3}\}\) is a Nash Equilibrium.

Regret Matching

For action profile \(\mathbf{a}\), player \(i\)'s regret for not playing action \(a'\) is: \[R(a'_i) =  U(a'_i, a_{-i}) - U(\mathbf{a})\] that is, the utility missed out on by not playing \(a'\).

 

Positive regret means that action would have been better.

Regret Matching:

For some number of iterations:

  1. Compute a regret matching profile (\(\sigma \propto \text{normalized cumulative regret}\))
  2. Add strategy profile to profile sum
  3. Select an action
  4. Compute regrets
  5. Add regrets to cumulative regrets

Turn-taking Games

Terminology

  • Max and Min players
  • deterministic
  • two player
  • zero-sum
  • perfect information

Minimax Trees

Minimax Tree

MDP Expectimax Tree

\[V(s) = \text{max}_{a \in \mathcal{A}}\left(R(s, a) + \text{E}[V(s')]\right)\]

\[V_1(s) = \text{max}_{a \in \mathcal{A_1}}\left(R(s, a) + \text{min}_{a' \in A_2} (R(s', a') + V(s''))\right)\]

Tree Backup Example

Why is this harder than an MDP? (think back to sparse sampling)

Alpha-Beta Pruning

Evaluation Functions

Examples of evaluation functions:

  • Rollout using default policy
  • Neural Network
  • Weighted combination of features (e.g. number of Pawns, Knights, etc.)

Deep Blue

  • Defeated world champion Gary Kasparov in may 1997
  • 30 IBM RS/600 processors for alpha-beta search
  • Routinely reached depth 14, sometimes up to 40
  • Evaluation function with 8000 features
  • Opening book of 4000 positions
  • Database of 700,000 grandmaster games

MCTS for Games

Break Exercise

  • Start at 5
  • Max can move up/down
  • Min moves left/right
  • Each player gets 2 turns
  • On green squares, Max wins
  • Who has advantage and what is first move?

Bonus: what if 5 was not green?

Incomplete Information

Incomplete Information

Incomplete Information

Extensive-form game definition (\(h\) is a sequence of actions called a "history"):

  • Finite set of \(n\) players, plus the "chance" player
  • \(P(h)\) (player at each history)
  • \(A(h)\) (set of actions at each history)
  • \(I(h)\) (information set that each history maps to)
  • \(U(h)\) (payoff for each leaf node in the game tree)

 

King-Ace Poker Example

  • 4 Cards: 2 Aces, 2 Kings
  • Each player is dealt a card
  • P1 can either raise (\(r\)) the payoff to 2 points or check (\(k\)) the payoff at 1 point
  • If P1 raises, P2 can either call (\(c\)) Player 1's bet, or fold (\(f\)) the payoff back to 1 point
  • The highest card wins

Extensive to Matrix Form

Exponential in number of info states!

Counterfactual Regret Minimization

Kuhn Poker Example

3 Cards: 1, 2, and 3

Solution (found by hand in 1950s):

  • Player 1 may pass on a 3 with arbitrary probability \(\gamma\), but then they must bet on a 1 w.p. \(\gamma/3\) and bet on a 2 in the second round w.p. \(\gamma/3 + 1/3\)
  • Player 2 must bet 1/3 of the time when holding a 1 after a pass and bet 1/3 of the time when holding a 2 with a bet

Matrix to Extensive Form

Takeaways

  • Games do not have a single optimal solution; Instead they have equilibria
  • Regret matching is a tool for finding Nash equilibria in Matrix games
  • Turn-taking zero-sum games of perfect information can be solved using minimax trees with Alpha-beta pruning or MCTS
  • Imperfect information can be represented as an extensive-form game with strategies conditioned on information sets.
  • Strategies for imperfect information games often involve stochastic "bluffing"
  • Counterfactual regret minimization can be used to solve extensive-form games