String Edit Distance

(and intro to dynamic programming)

Lecture #4

Computational Linguistics
CMPSCI 591N, Spring 2006
University of Massachusetts Amherst

Andrew McCallum

Andrew McCallum, UMass Amherst,

including material from William Cohen

Dynamic Programming
(Not much to do with programming in the
CS sense.)
Dynamic programming is efficient in finding
optimal solutions for cases with lots of
overlapping sub-problems.
It solves problems by recombining solutions
to sub-problems, when the sub-problems
themselves may share sub-sub-problems.

Andrew McCallum, UMass Amherst,

including material from William Cohen

Fibonacci Numbers

1 1 2 3 5 8 13 21 34 ...
Andrew McCallum, UMass Amherst,
including material from William Cohen

Andrew McCallum, UMass Amherst,

including material from William Cohen

Calculating Fibonacci Numbers

F(n) = F(n-1) + F(n-2),
where F(0)=0, F(1)=1.

Non-Dynamic Programming implementation

def fib(n):
if n == 0 or n == 1:
return n
else:
return fib(n-1) + fib(n-2)
For fib(8), how many calls to function fib(n)?
Andrew McCallum, UMass Amherst,
including material from William Cohen

DP Example:
Calculating Fibonacci Numbers
Dynamic Programming: avoid repeated calls by remembering
function values already calculated.
table = {}
def fib(n):
global table
if table.has_key(n):
return table[n]
if n == 0 or n == 1:
table[n] = n
return n
else:
value = fib(n-1) + fib(n-2)
table[n] = value
return value
Andrew McCallum, UMass Amherst,
including material from William Cohen

DP Example:
Calculating Fibonacci Numbers
...or alternately, in a list instead of a dictionary...
def fib(n):
table = [0] * (n+1)
table[0] = 0
table[1] = 1
for i in range(2,n+1):
table[i] = table[i-2] + table[i-1]
return table[n]
We will see this pattern many more times in this course:
1. Create a table (of the right dimensions to describe our problem.
2. Fill the table, re-using solutions to previous sub-problems.
Andrew McCallum, UMass Amherst,
including material from William Cohen

String Edit Distance

Given two strings (sequences) return the distance
between the two strings as measured by...
...the minimum number of character edit operations
needed to turn one sequence into the other.

Andrew
Amdrewz

1. substitute m to n
2. delete the z
Distance = 2

Andrew McCallum, UMass Amherst,

including material from William Cohen

String distance metrics: Levenshtein

Given strings s and t
Distance is shortest sequence of edit
commands that transform s to t, (or equivalently s
to t).
Simple set of operations:

Copy character from s over to t

Delete a character in s
Insert a character in t
Substitute one character for another

This is Levenshtein distance

Andrew McCallum, UMass Amherst,
including material from William Cohen

(cost 0)
(cost 1)
(cost 1)
(cost 1)

Levenshtein distance - example

distance(William Cohen, Willliam Cohon)
s

W I

L L gap I

A M _ C O H E N

alignment
t
W I L L L I A M _ C O H O N
edit
C C C C I
C C C C C C C S C
op
cost 0 0 0 0 1 1 1 1 1 1 1 1 2 2
so far...
Andrew McCallum, UMass Amherst,
including material from William Cohen

Alignment is a little bit like a parse.

Finding the Minimum

What is the minimum number of operations for....?

Another fine day in the park

Anybody can see him pick the ball
Not so easy.... not so clear.
Not only are the strings, longer, but is isnt immediately
obvious where the alignments should happen.
What if we consider all possible alignments by brute force?
How many alignments are there?
Andrew McCallum, UMass Amherst,
including material from William Cohen

Dynamic Program Table for String Edit

PARK

Measure distance between strings

SPAKE
P

S
P
A
K
E
Andrew McCallum, UMass Amherst,
including material from William Cohen

cij

cij =

the number of edit

operations needed
to align PA with
SPA.

Dynamic Programming to the Rescue!

How to take our big problem and chop it into building-block pieces.

Given some partial solution, it isnt hard to figure

out what a good next immediate step is.
Partial solution =
This is the cost for aligning s up to position i
with t up to position j.
Next step =
In order to align up to positions x in s and y in
t, should the last operation be a substitute,
insert, or delete?
Andrew McCallum, UMass Amherst,
including material from William Cohen

Dynamic Program Table for String Edit

PARK

Measure distance between strings

SPAKE
Edit operations
for turning
SPAKE
into
PARK

delete

S
P
A
K
E

Andrew McCallum, UMass Amherst,

including material from William Cohen

insert

substitute

Dynamic Program Table for String Edit

PARK

Measure distance between strings

SPAKE
P

c00

c02 c03

c04

c05

c10

c11 c12

c13

c14

c20

c21 c22

c23

c24

c30

c31 ???

K
E
Andrew McCallum, UMass Amherst,
including material from William Cohen

Dynamic Program Table for String Edit

P
S
P
A

c00

c02 c03

c04

c05

c10

c11 c12

c13

c14

c23

c24

c20

subst

c30

insert

delete

c21 c22

c31 ???

K
E
D(i,j) = score of best alignment from s1..si to t1..tj
= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1), if si=tj
D(i-1,j-1)+1, if si!=tj
D(i-1,j)+1
D(i,j-1)+1

//copy
//substitute
//insert
//delete

Computing Levenshtein distance - 2

D(i,j) = score of best alignment from s1..si to t1..tj
= min

D(i-1,j-1) + d(si,tj) //subst/copy

D(i-1,j)+1
//insert
D(i,j-1)+1
//delete

(simplify by letting d(c,d)=0 if c=d, 1 else)

also let D(i,0)=i (for i inserts) and D(0,j)=j

Andrew McCallum, UMass Amherst,

including material from William Cohen

Dynamic Program Table Initialized

P
0
S

K
4

D(i,j) = score of best alignment from s1..si to t1..tj

= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1)+d(si,tj)
D(i-1,j)+1
D(i,j-1)+1

//substitute
//insert
//delete

Dynamic Program Table ... filling in

P
0

K
4

D(i,j) = score of best alignment from s1..si to t1..tj

= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1)+d(si,tj)
D(i-1,j)+1
D(i,j-1)+1

//substitute
//insert
//delete

Dynamic Program Table ... filling in

P

K
4
4

D(i,j) = score of best alignment from s1..si to t1..tj

= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1)+d(si,tj)
D(i-1,j)+1
D(i,j-1)+1

//substitute
//insert
//delete

Dynamic Program Table ... filling in

P

K
4
4

D(i,j) = score of best alignment from s1..si to t1..tj

= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1)+d(si,tj)
D(i-1,j)+1
D(i,j-1)+1

//substitute
//insert
//delete

Dynamic Program Table ... filling in

P

Final cost of
aligning all of
both strings.

D(i,j) = score of best alignment from s1..si to t1..tj

= min
Andrew McCallum, UMass Amherst,
including material from William Cohen

D(i-1,j-1)+d(si,tj)
D(i-1,j)+1
D(i,j-1)+1

//substitute
//insert
//delete

DP String Edit Distance

def stredit (s1,s2):
"Calculate Levenstein edit distance for strings s1 and s2."
len1 = len(s1) # vertically
len2 = len(s2) # horizontally
# Allocate the table
table = [None]*(len2+1)
for i in range(len2+1): table[i] = [0]*(len1+1)
# Initialize the table
for i in range(1, len2+1): table[i][0] = i
for i in range(1, len1+1): table[0][i] = i
# Do dynamic programming
for i in range(1,len2+1):
for j in range(1,len1+1):
if s1[j-1] == s2[i-1]:
d = 0
else:
d = 1
table[i][j] = min(table[i-1][j-1] + d,
table[i-1][j]+1,
table[i][j-1]+1)
Andrew McCallum, UMass Amherst,
including material from William Cohen

Remebering the Alignment (trace)

D(i,j) = min

D(i-1,j-1) + d(si,tj) //subst/copy

D(i-1,j)+1
//insert
D(i,j-1)+1
//delete
C

A trace indicates
where the min
value came from,
and can be used to
find edit
operations and/or
a best alignment

(may be more than 1)

Andrew McCallum, UMass Amherst,

including material from William Cohen

C
C

1
1
2

2
3

3
3

Three Enhanced Variants

Needleman-Munch
Variable costs

Smith-Waterman
Find longest soft matching subsequence

Affine Gap Distance

Make repeated deletions (insertions) cheaper

(Implement one for homework?)

Andrew McCallum, UMass Amherst,
including material from William Cohen

Needleman-Wunch distance
D(i,j) = min

D(i-1,j-1) + d(si,tj) //subst/copy

D(i-1,j) + G
//insert
D(i,j-1) + G
//delete
G = gap cost

d(c,d) is an arbitrary
distance function on
characters (e.g. related
to typo frequencies,
amino acid
substitutibility, etc)
Andrew McCallum, UMass Amherst,
including material from William Cohen

William Cohen
Wukkuan Cigeb

Smith-Waterman distance
Instead of looking at each sequence in its
entirety, this compares segments of all
possible lengths and chooses whichever
maximize the similarity measure.
For every cell the algorithm calculates all
possible paths leading to it. These paths can
be of any length and can contain insertions
and deletions.

Andrew McCallum, UMass Amherst,

including material from William Cohen

Smith-Waterman distance
D(i,j) = min

G=1
d(c,c) = -2
d(c,d) = +1

Andrew McCallum, UMass Amherst,

including material from William Cohen

0
//start over
D(i-1,j-1) + d(si,tj) //subst/copy
D(i-1,j) + G
//insert
D(i,j-1) + G
//delete
C

-2

-1

-2

-1

-1

-4

-3

-2

-1

-3

-6

-5

-3

-2

-5

-5

-7

Example output from Python

l
o
u
n
g
e

0
1
2
3
4
5
6

s
1
0
0
0
0
0
0

'
2
0
0
0
0
0
0

a
3
0
0
0
0
0
0

l
l
o
n
g
e
4
5
6
7
8
9
0 * 0
0
0
0
0
0
0 *-2 -1
0
0
0
0 *-1 -1
0
0
0
0
0 *-3 -2 -1
0
0
0 -2 *-5 -4
0
0
0 -1 -4 *-7

r
10
0
0
0
0
-3
-6

(My implementation of HW#2, task choice #2. -McCallum)

Andrew McCallum, UMass Amherst,
including material from William Cohen

Affine gap distances

Smith-Waterman fails on some pairs that
seem quite similar:
William W. Cohen
William W. Dont call me Dubya Cohen
Intuitively,single
a single
long
insertion
cheaper
Intuitively,
long
insertions
areis cheaper
thanaalot
lotof
ofshort
shortinsertions
insertions
than
Andrew McCallum, UMass Amherst,
including material from William Cohen

Affine gap distances - 2

Idea:
Current cost of a gap of n characters: nG
Make this cost: A + (n-1)B, where A is cost of
opening a gap, and B is cost of continuing a
gap.

Andrew McCallum, UMass Amherst,

including material from William Cohen

Affine gap distances - 3

D(i,j) = max

D(i-1,j-1)
D(i-1,j-1) +
+ d(si,tj)
d(si,tj) //subst/copy
D(i-1,j)-1
IS(I-1,j-1) + d(si,tj) //insert
D(i,j-1)-1
//delete
IT(I-1,j-1) + d(si,tj)

IS(i,j) = max

D(i-1,j) - A
IS(i-1,j) - B

Best score in which si

is aligned with a gap

D(i,j-1) - A
IT(i,j-1) - B

Best score in which tj

is aligned with a gap

IT(i,j) = max

Andrew McCallum, UMass Amherst,

including material from William Cohen

Affine gap distances as automata

IS

-B

IT

-B

-d(si,tj)
-A
-d(si,tj)

D
-d(si,tj)

Andrew McCallum, UMass Amherst,

including material from William Cohen

-A

Generative version of affine gap

automata (Bilenko&Mooney, TechReport 02)
HMM emits pairs: (c,d) in
state M, pairs (c,-) in state
D, and pairs (-,d) in state I.
For each state there is a
multinomial distribution
on pairs.
The HMM can trained with
EM from a sample of pairs
of matched strings (s,t)
E-step is forward-backward; M-step uses some ad hoc smoothing
Andrew McCallum, UMass Amherst,
including material from William Cohen

Affine gap edit-distance learning:

experiments results (Bilenko & Mooney)

Experimental method: parse records into fields; append a

few key fields together; sort by similarity; pick a
threshold T and call all pairs with distance(s,t) < T
duplicates; picking T to maximize F-measure.
Andrew McCallum, UMass Amherst,
including material from William Cohen

Affine gap edit-distance learning:

experiments results (Bilenko & Mooney)

Andrew McCallum, UMass Amherst,

including material from William Cohen

Affine gap edit-distance learning:

experiments results (Bilenko & Mooney)

Precision/recall for MAILING dataset duplicate detection

Andrew McCallum, UMass Amherst,
including material from William Cohen

Affine gap distances experiments

(from McCallum, Nigam,Ungar KDD2000)
Goal is to match data like this:

Andrew McCallum, UMass Amherst,

including material from William Cohen

Homework #2
The assignment
Start with my stredit.py code
Make some modifications
Write a little about your experiences

Some possible modifications

Implement Needleman-Wunch, Smith-Waterman, or Affine Gap
Distance.
Create a little spell-checker: if entered word isnt in the dictionary,
return the dictionary word that is closest.
Change implementation to operate on sequences of words rather
than characters... get an online translation dictionary, and find
alignments between English & French or English & Russian!
Try to learn the parameters of the function from data. (Tough.)

Andrew McCallum, UMass Amherst,

including material from William Cohen