Artificial Intelligence

Game Playing
 Artificial intelligence : Games

• Games are a big deal in AI, Tictactoe, backgammon

etc.
• Games are interesting to AI because they are too hard to
solve
• One of the first examples of AI is the computerized game of Nim made in
1951
• https://www.archimedes-lab.org/game_nim/play_nim_game.html
 Artificial intelligence : Games

• One of the first examples of AI is the computerized game of Nim made in

1951
• https://www.archimedes-lab.org/game_nim/play_nim_game.html
• In 1951, using the Ferranti Mark 1 machine of the University of
Manchester, Christopher Strachey wrote a checkers program and Dietrich
Prinz wrote one for chess
• Arthur Samuel's checkers program developed in 1960s introduced
several concepts on game playing - search tree, pruning, heuristics
• Coined the term machine learning – “rote learning
(or generalization learning)”
• https://www.youtube.com/watch?v=iT_Un3xo1qE
Game playing machines
Checkers:
• Chinook ended 40-year-reign of human world
champion Marion Tinsley in 1994.

• Used an endgame database defining perfect play for

all positions involving 8 or fewer pieces on the board,
a total of 443,748,401,247 positions.

• https://cardgames.io/checkers/
Game playing machines
Chess:
• In 1949, Claude E. Shannon in his paper
“Programming a Computer for Playing Chess”,
suggested Chess as an AI problem for the
community.
• Deep Blue defeated human world champion Gary
Kasparov in a six-game match in 1997.
• In 2006, Vladmir Kramnik, the undisputed world
champion, was defeated 4-2 by Deep Fritz.
 Deep Blue : Man vs Machine
Deep Thought : chess playing machine project started in
CMU
Deep Blue : First computer to beat a chess champion in
1996
Kasparov said that he sometimes saw deep intelligence
and creativity in the machine's moves
Game playing machines
Go: b > 300! Google Deep mind Project AlphaGo. In
2016, AlphaGo beat both Fan Hui, the European Go
champion and Lee Sedol the worlds best player.
Game playing : Tree search
Game playing : 8 queens
https://mypuzzle.org/sl
iding

https://www.brainmetri
x.com/8-queens/
Game Playing : Tic Tac Toe

https://playtictactoe.org/
“Search” Solutions !

• Goal formulation, based on the current situation and the agent’s

performance measure, is the first step in problem solving.

• Problem formulation is the process of deciding what actions and states

to consider, given a goal.

• The process of looking for a sequence of actions that reaches the goal is
called search.
• A search algorithm takes a problem as input and returns a solution in the
form of an action sequence. Once a solution is found, the actions it
recommends can be carried out. This is called the execution phase.

• Intelligent agents use the design philosophy “formulate, search,

execute” design
8 Queens Puzzle
Goal of the 8-queens problem is to place eight queens on a chessboard such that no queen
attacks any other. (A queen attacks any piece in the same row, column or diagonal.)

http://brainmetrix.com/8-queens/
8 Queens Puzzle
8 puzzle problem • http://mypuzzle.org/sliding
 Travelling Problem

• The initial state that the agent

starts : In(Arad).

• ACTIONS(s) returns the set of

actions From In(Arad), the
applicable Find the path from Arad to Bucharest
• actions are {Go(Sibiu),
Go(Timisoara), Go(Zerind)}.

• TRANSITION MODEL model,

specified by a function
RESULT(s, a) that returns the
state that results from action a in
state
• RESULT(In(Arad),Go(Zerind))
= In(Zerind) .
• GOAL STATE {In(Bucharest
)}.
• PATH COST cost function that
assigns a numeric cost to each
path.
Search
Trees
Tree Terminology
Node — denotes some state/value
1 Root — The top node in a tree, the prime ancestor.
Child — A node directly connected to another node when
moving away from the root, an immediate descendant.
Parent — The converse notion of a child, an immediate
ancestor.
Leaf — A node with no children.
Internal node — A node with at least one child.
Edge — The connection between one node and another.
Depth — The distance between a node and the root.
Level — the number of edges between a node and the root
+1
Height — The number of edges on the longest path
between a node and a descendant leaf.
Breadth — The number of leaves.
Sub Tree — A tree T is a tree consisting of a node in T and
all of its descendants in T.
Binary Tree — is a tree data structure in which each node
has at most two children, which are referred to as the left
child and the right child.
Branching Factor – maximum number of branches for any
node
Breadth first search
Breadth first search

memory requirements are a bigger problem for breadth-first search than is the execution time.

exponential-complexity search problems cannot be solved by uninformed methods for

any but the smallest instances.
 Depth First Search

depth-first search requires storage of only O(bd) nodes, depth-first search would require 156
kilobytes instead of 10 exabytes at depth d = 16, a factor of 7 trillion times less space.

Time complexity :
BFS vs DFS
Breadth first search and depth first
search
 Informed Search : Best first search

a node is selected for expansion based on an

evaluation function, f(n).

a component of f a heuristic function, denoted h(n):

h(n) = estimated cost of the cheapest path from the

state at node n to a goal state. if n is a goal node, then

h(n) = 0.

straight- line distance heuristic,

in Romania, one might estimate the cost of the

cheapest path from Arad to Bucharest via the straight-

line distance from Arad to Bucharest.

 Informed Search : A* search

Evaluates nodes by combining

g(n) : the cost to reach the node, and

h(n) : the cost to get from the node to the goal:

f(n) = g(n) + h(n)

f (n) = estimated cost of the cheapest solution

through n .

Try first is the node with the lowest value of

g(n) + h(n)
Types of games

W e are mostly interested in deterministic games, fully o b -

servable environments, z ero - su m, w here t w o agents a c t al-
ternately.
 Zero-sum Games
• Adversarial: Pure competition.
• Agents have different values on the outcomes.
• One agent maximizes one single value, while the
other minimizes it.
• Each move by one of the players is called a “ply.”
O n e function: one agents maximizes it and one
minimizes it!
Embedded thinking...
E m b e d d e d thinking or backw ard reasoning!

• One agent is trying to figure out what to do.

• How to decide? He thinksabout the consequences of
the possible actions.
• He needs to think about his opponent as well...
• The opponent is also thinking about what to do etc.
• Each will imagine what would be the response from
the opponent to their actions.
• This entails an embedded thinking.
Adversarial Search

• There is an opponent we can’t control planning

again us!
• Game: optimal solution is not a sequence of actions
but a strategy (policy) If opponent does a, agent
does b, else if opponent does c, agent does d, etc.

• Tedious and fragile if hard-coded (i.e., implemented

with rules)
• Good news: Games are modeled as search
problems and use heuristic evaluation
functions.
Single player...
Assume we have a tic-tac-toe with one player.
L et’scall him M ax and havehim play threemoves only
for the sake of the example.
Single player...
Single player...

In the case of one player, nothing will prevent Max from

winning (choose the path that leads to the desired
utility here 1), unless there is another player who will
do everything to make Max lose, let’s call him Min (the
Mean :))
Adversarial search: minimax
• T w o players: Max and Min
• Players alternate turns
• Max moves first
• Max maximizes results
• Min minimizes the result
• Compute each node’s minimax value’s the best
achievable utility against an optimal adversary
• Minimax value = best achievable payoff against best
play
Minimax example
Formalization
• The initial s t a t e
• P layer( s) : defines which player has the move in state s.Usually
taking turns.

• Actions(s): returns the set of legal moves in s

• T r a n s it io n function: S * A - > S defines the result of a move

• Terminal test: True when the game is over, False

otherwise. States where game ends are called t e r m i n a l
states

• Utility(s,p): utility f unction or objective function for a game

that ends in terminal state s for player p. In Chess, the
outcome is a win, loss, or draw with values +1, 0, 1/2. For
tic-tac-toe we can use a utility of +1, -1, 0.
Adversarial search: minimax
• Find the optimal strategy for Max:
– Depth-first search of the game tree
– An optimal leaf node could appear at any depth
of the tree
– Minimax principle: compute the utility of being
in a state assuming both players play optimally
from there until the end of the game
– Propagate minimax values up the tree once
terminal nodes are discovered
Adversarial search: minimax
• If state is terminal node: Value is utility(state)
• If state is M A X node: Value is highest value of all
successor node values (children)

• If state is MIN node: Value is lowest value of all

successor node values (children)
Adversarial search: minimax

>
Minimax example
Minimax example
Minimax example
Minimax example
Minimax example
Minimax algorithm
Properties of minimax
• Optimal (opponent plays optimally) and complete
(finite tree)
• D F S time: O(bd)
• D F S space: O(bd)
– Tic-Tac-Toe
-5 legal moves on average, total of 9 moves (9
plies).
- 59 = 1,953,125
--9! = 362, 880 terminal nodes
– Chess
-b~ 35 (average branching factor)
- d ~ 100 (depth of game tree for a typical game)
--bd~ 35100 ~ 10154 nodes
– G o branching factor starts at 361 (19X19 board)
Multiplayer Games
Case of limited resources
• Problem: In real games, we are limited in time, so we
can’t search the leaves.

• T o be practical and run in a reasonable amount of

time, min- imax can only search to some depth.

• More plies make a big difference.

• Solution:
1 . R eplace terminal utilities with an
evaluation function for non-
terminal positions.
2. Use pruning: eliminate large parts of the tree.
↵—Ø pruning

A two-ply game tree.

↵—Ø pruning
↵—Ø pruning
↵ —Ø pruning
Which values are necessary?
↵—Ø pruning

Minimax(root) = max(min(3, 12,8), min(2, X , Y ), min(14, 5,2))

↵—Ø pruning

Minimax(root) = max(min(3, 12,8), min(2, X , Y ), min(14, 5,2))

= max(3, min(2, X , Y ), 2)
↵—Ø pruning

Minimax(root) = max(min(3, 12,8), min(2, X , Y ), min(14, 5,2))

= max(3, min(2, X , Y ), 2)

= max(3,Z,2) where Z = min(2,X , Y ) 2

 ↵—Ø pruning
Minimax(root) = max(min(3,12,8),min(2,X , Y ),min(14,5,2))

= max(3,min(2,X , Y ),2)

= max(3,Z,2) where Z = min(2,X , Y) <= 2

= 3
↵—Ø pruning

Minimax(root) = max(min(3, 12,8), min(2, X , Y ), min(14, 5,2))

= max(3, min(2, X , Y ), 2)

= max(3, Z,2) where Z = min(2, X , Y ) 2

= 3
M i n i m a x decisions are independent of t h e values of X and Y .
 ↵—Ø pruning

Minimax(MIN) = min(max(3, 1,2), max(12, A, B ))

= min(3, max(12, A, B ))

= min(3, Z) where Z >= max(12, A, B ) 2

= 3
M i n i m a x decisions are independent of t h e values of A and B .
↵—Ø pruning
• Strateg y: Just like minimax, it performs a D F S .
↵—Ø pruning
• Strategy: Just like minimax, it performs a D F S .
• Parameters: Keep track of two bounds
– alpha : largest value for Max across seen children (current lower
bound on MAX’s outcome).
– beta : lowest value for MIN across seen children (current upper
bound on MIN’s outcome).
↵—Ø pruning
• Strategy: Just like minimax, it performs a D F S .
• Parameters: Keep track of two bounds
– alpha : largest value for Max across seen children (current lower
bound on MAX’s outcome).
– beta : lowest value for MIN across seen children (current upper
bound on MIN’s outcome).

• Initialization: alpha = — 1 , b e t a = 1
↵—Ø pruning
• Strategy: Just like minimax, it performs a D F S .
• Parameters: Keep track of two bounds
– ↵: largest value for Max across seen children (current lower
bound on MAX’s outcome).
– Ø: lowest value for MIN across seen children (current upper
bound on MIN’s outcome).

• Initialization: alpha = — 1 , beta = 1

• Propagation: Send ↵, Øvalues down during the search to be
used for pruning.
– Update ↵, Ø values by propagating upwards values of ter-
minal nodes.
– Update ↵ only at M ax nodes and update Ø only at M in
nodes.
↵—Ø pruning
• Strateg y: Just like minimax, it performs a D F S .
• Parameters : Keep track of two bounds
– ↵: largest value for Max across seen children (current lower
bound on MAX’s outcome).
– Ø: lowest value for MIN across seen children (current upper
bound on MIN’s outcome).

• Initialization: ↵= — 1 , Ø= 1
• Propagation : Send ↵, Øvalues down during the search to be
used for pruning.
– Update ↵, Ø values by propagating upwards values of ter-
minal nodes.
– Update ↵ only at M ax nodes and update Ø only at M in
nodes.

• Pruning: Prune any remaining branches whenever ↵≤ Ø.

↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
↵—Ø pruning
Real-time decisions
• Minimax: generates the entire game search space

• alpha -beta algorithm: prune large chunks of the trees

• BUT alpha - beta still has to go all the way

to the leaves

• Impractical in real-time (moves has to be done in a

reasonable amount of time)

• Solution: bound the depth of search (cut off search)

and replace utility(s) w i t h eval(s), an evaluation
function to estim a t e value of current board
configurations
Real-time decisions
• eval(s) is a heuristic at state s
E.g., Othello: white pieces - black pieces
E.g., Chess: Value of all white
pieces Value of all black pieces turn non-
terminal nodes into terminal leaves!

• An ideal evaluation function would rank terminal

states in the same way as the true utility function;
but must be fast

• Typical to define features, make the function a

linear weighted sum of the features

• Use domain knowledge to craft the best and useful

features.
Real-time decisions

• How does it works?

– Select useful features f 1, . . .,f n e.g., C hess: # pieces on

board, value of pieces (1 for pawn, 3 for bishop,
etc.)
– Weighted linear function:

– Learn wi from the examples

– Deep blue uses about 6,000
features!
Deep Blue
Predictions were in Kasparov’s favor, many experts
predicted the champion to score at least four points out
of six. Kasparov was coming off a superb performance at
the Linares tournament and his rating was at an all-time
high of 2820. In a 2003 movie, he recalled his early
confidence: “I will beat the machine, whatever happens.
Look at Game One. It’s just a machine. Machines are
stupid.” Later Kasparov said that he sometimes saw
deep intelligence and creativity in the machine's moves
History of Chess playing machines
The Chess 4.5 program in the late 1970s first demonstrated that an
engineering approach emphasizing hardware speed might be more
fruitful.

Belle,2 a special purpose hardwired chess machine from Bell

Laboratories, became the first national master program in the early
1980s.

Cray Blitz3 running on a Cray supercomputer, and Hitech,4 another

specialpurpose chess machine, became the top programs in the mid-
1980s.

ChipTest (1986-1987), Deep Thought (1988-1991), and Deep

Thought II (1992-
1995)—claimed spots as the top chess programs in the world.

In 1988 the Deep Thought team won the second Fredkin

Intermediate Prize for Grandmaster-level performance for Deep
Thought’s 2650-plus rating on the USCF’s scale over 25 consecutive
games.

Deep Blue’s 1996 debut in the first Kasparov versus Deep Blue
Deep Blue
Winning a match against the human World Chess Champion
under regulation time control. The games had to play out no
faster than three minutes per move.

In 1996 Deep Blue was even with Kasparov after four games
in the first match. Kasparov pinpointed Deep Blue’s
weaknesses and won the remaining two games easily.
Computation speed alone apparently didn’t suffice.

What caused the problems for Cray Blitz and Deep Blue in the
1984 and 1996 matches? First, they played adaptable human
opponents.
Humans learn !

Make the weights associated with the positional features

individually adjustable from software.
System configuration

The 1997 version of Deep Blue included 480 chess chips. Since each
chess chip could search 2 to 2.5 million chess positions/s, the
“guaranteed not to exceed” system speed reached about one billion
chess positions per
second, or 40 tera operations.

For a program searching close to 40 billion positions for each

move—as the 1997 Deep Blue did—the alpha-beta algorithm
increases the search speed by up to 40 billion times.
Deep Blue evaluation functions
Deep Blue : computing resource
Stochastic games
• Include a random element (e.g., throwing a die).
• Include chance nodes.
• Backgammon: old board game combining skills and chance.
• The goal is that each player tries to move all of his pieces o↵ the
board before his opponent does.

Ptkfgs [Public domain], via Wikimedia Commons

Stochastic games

Partial g a m e t re e for B a c k g a m m o n .
Stochastic games
Algorithm Ex p e c t i mi n i ma x generalized Minimax to handle chance
nodes as follows:

• If state is a Max node then

return the highest Expectiminimax-Value of Successors(state)

• If state is a Min node then

return the lowest Expectiminimax-Value of Successors(state)

• If state is a chance node then

return average of Expectiminimax-Value of Successors(state)
Stochastic games
Example with coin-flipping:
Expectiminimax

For a state s:
Expectiminimax(s) =
8
>
< Utility(s) if Terminal-test(s)
maxa2Actions(s) E xpectiminimax( Result( s,a)) if Player(s) = Max
> min a2Actions(s) Expectiminimax(Result(s,a)) if Player(s) = Min
: P
r P (r) Expectiminimax(Result(s,r)) if Player(s) = Chance

Where r represents all chance events (e.g., dice roll), and Re-
sult(s,r) is the same state as s with the result of the chance event is
r.
Games: conclusion
• Games are modeled in AI as a search problem and use heuristic to
evaluate the game.

• Minimax algorithm choses the best most given an optimal play

from the opponent.

• Minimax goes all the way down the tree which is not practical give
game time constraints.

• A lpha-B eta pruning can reduce the game tree search which
allow to go deeper in the tree within the time constraints.

• Pruning, bookkeeping, evaluation heuristics, node re-ordering and

IDS are effective in practice.
Games: conclusion
• Games is an exciting and fun topic for AI.

• Devising adversarial search agents is challenging because of the

huge state space.

• We have just scratched the surface of this topic.

• F urther topics to explore include partially observable games

(card games such as bridge, pocker, etc.).

• Except for robot football (a.k.a. soccer), there was no much

interest from AI in physical games.
(see h t t p : / / w w w . r o b oc u p. o r g / ) .

• Interested in chess? check out the evaluation functions in

Claude Shannon’s paper.