COMP 412

FALL 2010

Introduction to Parsing

Comp 412

Copyright 2010, Keith D. Cooper & Linda Torczon, all rights reserved.
Students enrolled in Comp 412 at Rice University have explicit permission to make
copies of these materials for their personal use.
Faculty from other educational institutions may use these materials for nonprofit
educational purposes, provided this copyright notice is preserved.
The Front End

Source tokens IR
Scanner Parser
code

Errors

Parser
• Checks the stream of words and their parts of speech
(produced by the scanner) for grammatical correctness
• Determines if the input is syntactically well formed
• Guides checking at deeper levels than syntax
• Builds an IR representation of the code
Think of this chapter as the mathematics of diagramming
sentences

Comp 412, Fall 2010 2

The Study of Parsing
The process of discovering a derivation for some sentence
• Need a mathematical model of syntax — a grammar G
• Need an algorithm for testing membership in L(G)
• Need to keep in mind that our goal is building parsers,
not studying the mathematics of arbitrary languages

Roadmap for our study of parsing

1 Context-free grammars and derivations Today
2 Top-down parsing
— Generated LL(1) parsers & hand-coded recursive descent
parsers
3 Bottom-up parsing Lab 2
— Generated LR(1) parsers

We will define “context free” today. I am

Comp 412, Fall 2010 3
just deferring the definition for a couple of
slides.
Specifying Syntax with a Grammar
Context-free syntax is specified with a context-free grammar

SheepNoise  SheepNoise baa

| baa

This CFG defines the set of noises sheep normally make

It is written in a variant of Backus–Naur form

Formally, a grammar is a four tuple, G = (S,N,T,P)

• S is the start symbol (set of strings in L(G))
• N is a set of nonterminal symbols (syntactic variables)
• T is a set of terminal symbols (words)
• P is a set of productions or rewrite rules (P : N  (N  T)+ )
Example due to Dr. Scott K. Warren

Comp 412, Fall 2010 From Lecture 4

1
Deriving Syntax
We can use the SheepNoise grammar to create sentences
— use the productions as rewriting rules

And so on ...

While this example is cute, it quickly runs out of intellectual

steam ...
Comp 412, Fall 2010 5
Why Not Use Regular Languages & DFAs?
Not all languages are regular (RL’s  CFL’s  CSL’s)
You cannot construct DFA’s to recognize these languages
• L = { p k qk } (parenthesis
languages)
• L = { wcwr | w  *}
Neither of these is a regular language (nor an RE)

To recognize these features requires an arbitrary amount of

context (left or right …)
But, this issue is somewhat subtle. You can construct DFA’s
for
• Strings with alternating 0’s and 1’s
(  | 1 ) ( 01 )* (  | 0 )
• Strings with an even number of 0’s and 1’s
RE’s can count bounded sets and bounded differences
Comp 412, Fall 2010 6
Limits of Regular Languages
Advantages of Regular Expressions
• Simple & powerful notation for specifying patterns
• Automatic construction of fast recognizers
• Many kinds of syntax can be specified with REs
Example — a regular expression for arithmetic expressions
Term  [a-zA-Z] ([a-zA-Z] | [0-9])*
Op  +|-||/
Expr  ( Term Op )* Term
([a-zA-Z] ([a-zA-Z] | [0-9])* (+ | - |  | /))* [a-zA-Z] ([a-zA-Z] | [0-9])
Of course, this would generate a DFA …

If REs are so useful … Why not use them for everything?

 Cannot add parenthesis, brackets, begin-end pairs, …

Comp 412, Fall 2010 7

Context-free Grammars
What makes a grammar “context free”?

The SheepNoise grammar has a specific form:

SheepNoise  SheepNoise baa

| baa

Productions have a single nonterminal on the left hand side,

which makes it impossible to encode left or right context.
 The grammar is context free.
A context-sensitive grammar can have ≥ 1 nonterminal on
lhs.

Notice that L(SheepNoise) is actually a regular language: baa

+

Classic definition: any language that can be

Comp 412, Fall 2010 8
recognized by a push-down automaton is a
context-free language.
A More Useful Grammar Than Sheep Noise
To explore the uses of CFGs,we need a more complex
grammar
Rule Sentential Form
0 Expr  Expr Op Expr
— Expr
1 | number
0 Expr Op Expr
2 | id
2 <id,x> Op Expr
3 Op  + 4 <id,x> - Expr
4 | - 0 <id,x> - Expr Op Expr
5 | * 1 <id,x> - <num,2> Op
Expr
6 | /
5 <id,x> - <num,2> *
Expr
2 <id,x> - <num,2> *
<id,y>

• Such a sequence of rewrites is called a derivation

• Process of discovering a derivation is called parsing
We denote this derivation: Expr * id – num *
id
Comp 412, Fall 2010 9
Derivations
The point of parsing is to construct a derivation

• At each step, we choose a nonterminal to replace

• Different choices can lead to different derivations
Two derivations are of interest
• Leftmost derivation — replace leftmost NT at each step
• Rightmost derivation — replace rightmost NT at each step
These are the two systematic derivations
(We don’t care about randomly-ordered derivations!)

The example on the preceding slide was a leftmost

derivation
• Of course, there is also a rightmost derivation
• Interestingly, it turns out to be different
Comp 412, Fall 2010 10
Derivations
The point of parsing is to construct a derivation

A derivation consists of a series of rewrite steps

S  0  1  2  …  n–1  n  sentence

• Each i is a sentential form

— If  contains only terminal symbols,  is a sentence in L(G)
— If  contains 1 or more non-terminals,  is a sentential form
• To get i from i–1, expand some NT A  i–1 by using A 
— Replace the occurrence of A  i–1 with  to get i
— In a leftmost derivation, it would be the first NT A  i–1

A left-sentential form occurs in a leftmost derivation

A right-sentential form occurs in a rightmost derivation

Comp 412, Fall 2010 11

The Two Derivations for x – 2 * y

Rule Sentential Form Rule Sentential Form

— Expr — Expr
0 Expr Op Expr 0 Expr Op Expr
2 <id,x> Op Expr 2 Expr Op <id,y>
4 <id,x> - Expr 5 Expr * <id,y>
0 <id,x> - Expr Op Expr 0 Expr Op Expr * <id,y>
1 <id,x> - <num,2> Op 1 Expr Op <num,2> *
Expr <id,y>
5 <id,x> - <num,2> * 4 Expr - <num,2> *
Expr <id,y>
2 <id,x> - <num,2> * 2 <id,x> - <num,2> *
Leftmost
<id,y> derivation Rightmost
<id,y>
derivation
In both cases, Expr * id – num * id
• The two derivations produce different parse trees
• The parse trees imply different evaluation orders!
Comp 412, Fall 2010 12
Derivations and Parse Trees
Leftmost derivation
G
Rule Sentential Form
— Expr
0 Expr Op Expr
2 <id,x> Op Expr E
4 <id,x> - Expr
0 <id,x> - Expr Op Expr
1 <id,x> - <num,2> Op E Op E
Expr
5 <id,x> - <num,2> *
Expr x – E Op E
2 <id,x> - <num,2> *
<id,y>

This evaluates as x – ( 2 * 2 y
y) *

Comp 412, Fall 2010 13

Derivations and Parse Trees
Rightmost derivation
G
Rule Sentential Form
— Expr
0 Expr Op Expr
2 Expr Op <id,y> E
5 Expr * <id,y>
0 Expr Op Expr * <id,y>
1 Expr Op <num,2> * E Op E
<id,y>
4 Expr - <num,2> *
<id,y>
E Op E * y
2 <id,x> - <num,2> *
<id,y>

This evaluates as ( x – 2 ) * x – 2
y

This ambiguity is NOT good

Comp 412, Fall 2010 14
Derivations and Precedence

These two derivations point out a problem with the grammar:

It has no notion of precedence, or implied order of evaluation

To add precedence
• Create a nonterminal for each level of precedence
• Isolate the corresponding part of the grammar
• Force the parser to recognize high precedence
subexpressions first

For algebraic expressions

• Parentheses first (level 1 )
• Multiplication and division, next ( level
2)
• Subtraction and addition, last ( level 3)

Comp 412, Fall 2010 15

 Derivations and Precedence
Adding the standard algebraic precedence produces:
0 Goal  Expr This grammar is slightly larger
1 Expr  Expr + Term •Takes more rewriting to
level
2 | Expr - Term reach some of the terminal
3
3 | Term symbols

level
4 Term  Term * Factor •Encodes expected
5 | Term / Factor precedence
2
6 | Factor •Produces same parse tree
7 Factor  ( Expr ) under leftmost & rightmost
level
8 | number derivations
1
9 | id •Correctness trumps the speed
of the parser

Cannot handle Let’s see how

Introduced it parses xtoo
parentheses, -2*
precedence in an RE for y
(beyond power of an RE)
expressions
Comp 412, Fall 2010 One form of the “classic expression 16
grammar”
Derivations and Precedence
Rule Sentential Form G
— Goal
0 Expr E
2 Expr - Term
4 Expr - Term * Factor E – T
9 Expr - Term * <id,y>
6 Expr - Factor * <id,y> T T * F
8 Expr - <num,2> *
<id,y> F F <id,y
3 Term - <num,2> * >
<id,y>
6 Factor - <num,2> * <id,x <num,2>
<id,y> >
9 <id,x> - <num,2> * Its parse tree
The rightmost
<id,y>
derivation

It derives x – ( 2 * y ), along with an appropriate parse tree.

Both the leftmost and rightmost derivations give the same expression,
because the grammar directly and explicitly encodes the desired
precedence.
Comp 412, Fall 2010 17
Ambiguous Grammars
Let’s leap back to our original expression grammar.
It had other problems.
Rule Sentential Form
0 Expr  Expr Op Expr — Expr
1 | number 0 Expr Op Expr
2 | id 2 <id,x> Op Expr
3 Op  + 4 <id,x> - Expr
0 <id,x> - Expr Op Expr
4 | -
1 <id,x> - <num,2> Op
5 | * Expr
6 | / 5 <id,x> - <num,2> *
Expr
2 <id,x> - <num,2> *
<id,y>
• This grammar allows multiple leftmost derivations for x - 2 * y
• Hard to automate derivation if > 1 choice
Different choice
• The grammar is ambiguous than the first time
Comp 412, Fall 2010 18
Two Leftmost Derivations for x – 2 * y
The Difference:
 Different productions chosen on the second step
Rule Sentential Form Rule Sentential Form
— Expr — Expr
0 Expr Op Expr 0 Expr Op Expr
2 <id,x> Op Expr 0 Expr Op Expr Op Expr
4 <id,x> - Expr 2 <id,x> Op Expr Op
0 <id,x> - Expr Op Expr Expr
1 <id,x> - <num,2> Op 4 <id,x> - Expr Op Expr
Expr 1 <id,x> - <num,2> Op
5 <id,x> - <num,2> * Expr
Expr 5 <id,x> - <num,2> *
1 <id,x> - <num,2> * Expr
Original choice
<id,y> 2 New -choice
<id,x> <num,2> *
<id,y>

 Both derivations succeed in producing x - 2 * y

Comp 412, Fall 2010 19
Two Leftmost Derivations for x – 2 * y
The Difference:
 Different productions chosen on the second step
Rule Sentential Form Rule Sentential Form
— Expr — Expr
0 Expr Op Expr 0 Expr Op Expr
2 <id,x> Op Expr 0 Expr Op Expr Op Expr
4 <id,x> - Expr 2 <id,x> Op Expr Op
0 <id,x> - Expr Op Expr Expr
1 <id,x> - <num,2> Op 4 <id,x> - Expr Op Expr
Expr 1 <id,x> - <num,2> Op
5 <id,x> - <num,2> * Expr
Expr 5 <id,x> - <num,2> *
2 <id,x> - <num,2> * Expr
Original choice
<id,y> 2 New -choice
<id,x> <num,2> *
<id,y>
Different choices in same
situation, again
Remember
Comp 412, Fall 2010 nondeterminism? 20
Ambiguous Grammars
Definitions
• If a grammar has more than one leftmost derivation for
a single sentential form, the grammar is ambiguous
• If a grammar has more than one rightmost derivation
for a single sentential form, the grammar is ambiguous
• The leftmost and rightmost derivations for a sentential
form may differ, even in an unambiguous grammar
— However, they must have the same parse tree!

Classic example — the if-then-else problem

Stmt  if Expr then Stmt
| if Expr then Stmt else Stmt
| … other stmts …
This ambiguity is inherent in the grammar
Comp 412, Fall 2010 21
Ambiguity
This sentential form has two derivations
if Expr1 then if Expr2 then Stmt1 else Stmt2 Part of the problem
is that the structure
built by the parser
if if will determine the
interpretation of the
code, and these two
E1 then else E1 then forms have different
meanings!

if S2 if

E2 then E2 then else

S1 S1 S2

production 2, then production 1, then

production 1 production 2

Comp 412, Fall 2010 22

 The grammar forces the
Ambiguity structure to match the desired
meaning.
Removing the ambiguity
• Must rewrite the grammar to avoid generating the
problem
• Match each else to innermost unmatched if (common sense
0)
rule Stmt  if Expr then Stmt
1  if Expr then WithElse else Stmt
2  Other Statements
3 WithElse  if Expr then WithElse else WithElse
4  Other Statements

Intuition: once into WithElse, we cannot generate an unmatched

With
elsethis grammar, example has only one rightmost
derivation
… a final if without an else can only come through rule 2 …
Comp 412, Fall 2010 23
Ambiguity
if Expr1 then if Expr2 then Stmt1 else Stmt2

Rul Sentential Form

e
— Stmt
0 if Expr then Stmt
1 if Expr then if Expr then WithElse else Stmt
2 if Expr then if Expr then WithElse else S2
4 if Expr then if Expr then S1 else S2
? if Expr then if E2 then S1 else S2
? if E1 then if E2 then S1 else S2
Other productions to derive Expr
s
This grammar has only one rightmost derivation for the
example
Comp 412, Fall 2010 24
Deeper Ambiguity
Ambiguity usually refers to confusion in the CFG
Overloading can create deeper ambiguity
a = f(17)
In many Algol-like languages, f could be either a function
or a subscripted variable

Disambiguating this one requires context

• Need values of declarations
• Really an issue of type, not context-free syntax
• Requires an extra-grammatical solution (not in CFG)
• Must handle these with a different mechanism
— Step outside grammar rather than use a more complex
grammar

Comp 412, Fall 2010 25

Ambiguity - the Final Word
Ambiguity arises from two distinct sources
• Confusion in the context-free syntax (if-then-else)
• Confusion that requires context to resolve (overloading)

Resolving ambiguity
• To remove context-free ambiguity, rewrite the grammar
• To handle context-sensitive ambiguity takes cooperation
— Knowledge of declarations, types, …
— Accept a superset of L(G) & check it by other means†
— This is a language design problem

Sometimes, the compiler writer accepts an ambiguous

grammar
— Parsing techniques that “do the right thing”
— i.e., always select the same derivation

Comp 412, Fall 2010 †

See Chapter 4 26