NLP LECTURE NOTES UNIT2

NLP UNIT II (PART 1)

Grammars and Parsing
Grammars and Parsing- Top- Down and Bottom-Up Parsers, Transition Network Grammars, Feature Systems and Augmented Grammars,
Morphological Analysis and the Lexicon, Parsing with Features, Augmented Transition Networks, Bayes Rule, Shannon game, Entropy and Cross
Entropy.

The syntactic structure of a sentence can be computed with two things: the grammar, which is a formal
specification of the structures of the language, and the parsing technique, is the method of analyzing a
sentence to determine its structure according to the grammar.

Grammar and Sentence Structure

• The most common way of representing how a sentence is broken into its major subparts, and how
those subparts are broken up, is as a tree.
• The tree representation for the sentence John ate the cat is shown in Figure 3.1.

• This illustration can be read as follows: The sentence (S) consists of an initial noun phrase
(NP) and a verb phrase (VP).
• The initial noun phrase is made of the simple NAME John. The verb phrase is composed of a verb
(V) ate and an NP, which consists of an article (ART) the and a common noun (N) cat.

• In list notation this same structure could be represented as

(S (NP (NAME John))

(VP (V ate)
(NP (ART the)

(N cat) )))

• Since trees play such an important role , some terminology needs to be introduced.
• Trees are a special form of graph, which are structures consisting of labeled nodes (for example,
the nodes are labeled S, NP, and so on in Figure 3.1) connected by links.
• They are called trees because they resemble upside-down trees, and the terminology is derived
from this analogy with actual trees.
• The node at the top is called the root of the tree, while the nodes at the bottom are called the
leaves.

DR.P.GANGADHARA REDDY 1
NLP LECTURE NOTES UNIT2

• We say a link points from a parent node to a child node.

• The node S in Figure 3.1 is the parent node of the nodes labeled NP and VP, and the node NP is
in turn the parent node of the node labeled NAME.

• While every child node has a unique parent, a parent may point to many child nodes.An ancestor of
a node N is defined as N's parent, or the parent of its parent, and so on.

• A node is dominated by its ancestor nodes. The root node dominates all other nodes in the tree.

• To construct a tree structure for a sentence, must know what structures are legal for English.

• A set of rewrite rules describes what tree structures are allowable.These rules say that a certain symbol
may be expanded in the tree by a sequence of other symbols.
• A set of rules that would allow the tree structure in Figure 3.1 is shown as Grammar 3.2.

• Rule 1 :S may consist of an NP followed by a VP.

• Rule 2: VP may consist of a V followed by an NP.
• Rules 3 and 4:NP may consist of a NAME or may consist of an ART followed by an N.
• Rules 5 – 8: define possible words for the categories.

DR.P.GANGADHARA REDDY 2
NLP LECTURE NOTES UNIT2

• Context-Free Grammers is a formal grammer which is used to generate all possible strings in a
given formal language.

• Grammars consists of rules with a single symbol on the left-hand side, called the mother, are called
context-free grammars (CFGs).
• CFGs are a very important class of grammars for two reasons: The formalism is powerful to
describe most of the structure in natural languages, yet it is restricted enough so that efficient parsers
can be built to analyze sentences.
• Symbols that cannot be further decomposed in a grammar,are called terminal symbols.
• The other symbols, such as NP, VP, and S, are called nonterminal symbols.

• The grammatical symbols such as N and V that describe word categories are called lexical
symbols.Of course, many words will be listed under multiple categories.

• For example, can would be listed under V and N.

• Grammars have a special symbol called the start symbol.

• The start symbol will always be S. A grammar is said to derive a sentence if there is a sequence of
rules that allow you to rewrite the start symbol into the sentence.
• For instance, Grammar 3.2 derives the sentence John ate the cat.

• This shows the sequence rewrites starting from the S symbol, as follows:
S

=> NP VP (rewriting S)

=> NAME VP (rewriting NP)

=> John VP (rewriting NAME)

=> John V NP (rewriting VP)

=> John ate NP (rewriting V)

=> John ate ART N (rewriting NP)

=> John ate the N (rewriting ART)

=> John ate the cat (rewriting N)

DR.P.GANGADHARA REDDY 3
NLP LECTURE NOTES UNIT2

• Two important processes are based on derivations.The first is sentence generation, which uses
derivations to construct legal sentences.

• A simple generator could be implemented by randomly choosing rewrite rules, starting from the
S symbol to have a sequence of words.
• The preceding example shows that the sentence John ate the cat can be generated from the grammar.
• The second process based on parsing, which identifies the structure of sentences given a grammar.
• There are two basic methods of searching.

• A top-down strategy starts with the S symbol and then searches through different ways to rewrite
the symbols until the input sentence is generated, or until all possibilities have been explored.
• The preceding example demonstrates that John ate the cat is a legal sentence by showing the derivation
that could be found by this process.
• In a bottom-up strategy, start with the words in the sentence and use the rewrite rules backward
to reduce the sequence of symbols until it consists solely of S.The left-hand side of each rule is
used to rewrite the symbol on the right-hand side.

• A possible bottom-up parse of the sentence John ate the cat is

=> NAME ate the cat (rewriting John)

=> NAME V the cat (rewriting ate)

=> NAME V ART cat (rewriting the)

=> NAME V ART N (rewriting cat)

=> NP V ART N (rewriting NAME)

=> NP V NP (rewriting ART N)

=> NP VP (rewriting V NP)

=> S (rewriting NP VP)

• A tree representation in Figure 3.1, can be viewed as a record of the CFG rules for the structure of the
sentence.In other words, a record of the parsing process, working either top-down or bottom-up, it
would be something similar to the parse tree representation.

DR.P.GANGADHARA REDDY 4
NLP LECTURE NOTES UNIT2

A Top-Down Parser
• A parsing algorithm is a procedure that generates a tree that could be the structure of the input
sentence.
• A simple top-down parsing method relates this to work in artificial intelligence (Al) on search
procedures.
• A top-down parser starts with the S symbol and attempts to rewrite it into a sequence of terminal
symbols that matches the classes of the words in the input sentence.
• The state of the parse can be represented as a list of symbols, called the symbol list.

• For example, the parser starts in the state (S) and after applying the rule S -> NP VP the symbol list
will be (NP VP).If it then applies the rule NP ->ART N, the symbol list will be (ART N VP), and so
on.The parser could continue in this fashion until the state consisted entirely of terminal symbols,
and then it could check the input sentence to see if it matched.

• A better algorithm checks the input and a structure called the lexicon is used to efficiently store
the possible categories for each word.
• For now the lexicon will be very simple. A very small lexicon for use in the examples is
Cried :V
dogs : N, V
the : ART

• With a lexicon specified, a grammar, such as that shown as Grammar 3.4, need not contain any
lexical rules.
Grammer 
1.S->NP VP
2.NP->ART N
3.NP->ART ADJ N
4.VP->V
5.VP->V NP

• A state of the parse is defined by a pair: a symbol list similar to before and a number indicating the
current position in the sentence.Positions fall between the words, with 1 being the position before the
first word.For example, here is a sentence with its positions indicated:

1 The 2 dogs 3 cried 4

DR.P.GANGADHARA REDDY 5
NLP LECTURE NOTES UNIT2

• A typical parse state would be (N VP) 2) indicating that the parser needs to find an N followed by a VP,
starting at position two.

• New states are generated from old states depending on whether the first symbol is a lexical symbol
or not.
• If it is a lexical symbol, like N, and if the next word can belong to that lexical category, then update
the state by removing the first symbol and updating the position counter.
• In this case,the word dogs is listed as an N in the lexicon, the next parser state would be((VP) 3)which
means it needs to find a VP starting at position 3.

• If the first symbol is a nonterminal, like VP, then it is rewritten using a rule from the grammar.
• For example, using rule 4 in Grammar 3.4, the new state would be ((V) 3) which means it needs to find
a V starting at position 3.
• On the other hand, using rule 5, the new state would be ((V NP) 3)

• A parsing algorithm that is guaranteed to find a parse if there is one must systematically explore
every possible new state is called backtracking.

• Using this , rather than generating a single new state from the state ((VP) 3), you generate all
possible new states.One of these is picked to be the next state and the rest are saved as backup states.

• If the current state cannot lead to a solution, then pick a new current state from the list of backup
states.

A Simple Top-Down Parsing Algorithm

• The algorithm manipulates a list of possible states, called the possibilities list.

• The first element of this list is the current state, which consists of a symbol list - and a word position
In the sentence, and the remaining elements are the backup states, each indicating an alternate symbol-
list—word-position pair.
• For example, the possibilities list (((N) 2) ((NAME) 1) ((ADJ N) 1)) indicates that the current state
consists of the symbol list (N) at position 2, There are two possible backup states:
First one consisting of the symbol list (NAME) at position 1.

Second one consisting of the symbol list (ADJ N) at position 1.

DR.P.GANGADHARA REDDY 6
NLP LECTURE NOTES UNIT2

Step Current State Backup Sates Comment

1 ((S) 1) initial position
2 ((NP VP) 1) rewriting S by rule I
3 ((ART N VP) 1) rewriting NP by rules 2 & 3
((ART ADJN VP) I)
4 ((N VP) 2) matching ART with the
((ART ADJ N VP) 1)
5 ((VP) 3) matching N with dogs
6 ((V) 3) rewriting VP by rules 5—8
((V NP) 3)
((ART ADJ N VP) 1)
7 the parse succeeds as V is
matched to cried, leaving
an empty grammatical symbol
list with an empty sentence
Figure 3.5 Top-down depth-first parse of 1 The 2 dogs 3 cried 4

Top-Down Parser Algorithm:

The algorithm starts with the initial state ((S) 1) and no backup states.

1. Select the current state: Take the first state off the possibilities list and call it C. If the possibilities
list is empty, then the algorithm fails (that is, no successful parse is possible).
2. If C consists of an empty symbol list and the word position is at the end of the sentence, then the
algorithm succeeds.
3. Otherwise, generate the next possible states.

3.1 If the first symbol on the symbol list of C is a lexical symbol, and the next word in the sentence
can be in that class, then create a new state by removing the first symbol from the symbol list and
updating the word position, and add it to the possibilities list.
3.2 Otherwise, if the first symbol on the symbol list of C is a non-terminal, generate a new state
for each rule in the grammar that can rewrite that nonterminal symbol and add them all to the
possibilities list.
Consider an example.

• Using Grammar 3.4, Figure 3.5 shows a trace of the algorithm on the sentence

The dogs cried.

DR.P.GANGADHARA REDDY 7
NLP LECTURE NOTES UNIT2

• First, the initial S symbol is rewritten using rule 1 to produce a new current state of ((NP VP) 1) in
step 2.

• The NP is then rewritten in turn, but since there are two possible rules for NP in the grammar,

two possible states are generated:

• The new current state involves (ART N VP) at position 1, whereas the backup state involves
(ART ADJ N VP) at position 1.
• In step 4, a word in category ART is found at position I of the sentence, and the new current state
becomes (N VP).
• The backup state generated in step 3 remains untouched.

• The parse continues in this fashion to step 5, where two different rules can rewrite VP.

• The first rule generates the new current state, while the other rule is pushed onto the stack of backup
states.
• The parse completes successfully in step 7, since the current state is empty and all the words in the
input sentence have been accounted for.
• Consider the same algorithm and grammar operating on the sentence

1 The 2 old 3 man 4 cried 5

• In this case, assume that the word old is ambiguous between an ADJ and an N and that the word man
is ambiguous between an N and a V. (as in the sentence The sailors man the boats).
• Specifically, the lexicon is

the : ART
old : ADJ, N

man : N, V

cried :V
• The parse proceeds as follows (see Figure 3.6).

DR.P.GANGADHARA REDDY 8
NLP LECTURE NOTES UNIT2

Figure 3.6 A top-down parse of 1 The 2 old 3 man 4 cried 5

• The initial S symbol is rewritten by rule 1 to produce the new current state of ((NP VP) 1).

• The NP is rewritten in turn, giving the new state of ((ART N VP) 1) with a backup state of ((ART
ADJ N VP) 1).

• The parse continues, finding the as an ART to produce the state ((N VP) 2) and then old as an N to
obtain the state ((VP) 3).

• There are now two ways to rewrite the VP, giving us a current state of ((V) 3) and the backup states
of ((V NP) 3) and ((ART ADJ N) 1) from before.

• The word man can be parsed as a V. giving the state (04).

• Unfortunately, while the symbol list is empty, the word position is not at the end of the sentence, so no
new state can be generated and a backup state must be used. In the next cycle, step 8, ((V NP) 3) is

DR.P.GANGADHARA REDDY 9
NLP LECTURE NOTES UNIT2

attempted.
• Again man is taken as a V and the new state ((NP) 4) generated. None of the rewrites of NP yield a
successful parse.
• Finally, in step 12, the last backup state, ((ART ADJ N VP) 1), is tried and leads to a successful parse.
Parsing as a Search Procedure

• Parsing as a special case of a search problem as defined in Al.The top-down parser was described
in terms of the following generalized search procedure.
• The possibilities list is initially set to the start state of the parse.

• Then repeat the following steps until have success or failure:

1. Select the first state from the possibilities list (and remove it from the list).

2. Generate the new states by trying every possible option from the selected state (there may be none
if we are on a bad path).
3. Add the states generated in step 2 to the possibilities list.

• For a depth-first strategy, the possibilities list is a stack.

• In other words, step 1 always takes the first element off the list, and step 3 always puts the new states
on the front of the list, yielding a last-in first-out (LIFO) strategy.
• In contrast, in a breadth-first strategy the possibilities list is manipulated as a queue.

• Step 3 adds the new positions onto the end of the list, rather than the beginning, yielding a first-in
first-out (FIR)) strategy.
• We can compare these search strategies using a tree format, as in Figure 3.7, which shows the
entire space of parser states for the last example.

DR.P.GANGADHARA REDDY 10
NLP LECTURE NOTES UNIT2

Figure 3.7 Search tree for two parse strategies (depth-first strategy on left; breadth-first on right)

• Each node in the tree represents a parser state, and the sons of a node are the possible moves from
that state.
• The number beside each node records when the node was selected to be processed by the algorithm.
• On the left side is the order produced by the depth-first strategy, and on the right side is the order
produced by the breadth-first strategy. Remember, the sentence being parsed is
1 The 2 old 3 man 4 cried 5

DR.P.GANGADHARA REDDY 11
NLP LECTURE NOTES UNIT2

• The main difference between depth-first and breadth-first searches in this simple example is the order
in which the two possible interpretations of the first NP are examined.
• With the depth-first strategy, one interpretation is considered and expanded until it fails; only then is
the second one considered.
• With the breadth-first strategy, both interpretations are considered alternately, each being expanded
one step at a time.
• In this example, both depth-first and breadth-first searches found the solution but searched the space
in a different order.
• A depth-first search often moves quickly to a solution but in other cases may spend considerable time
pursuing futile paths.
• The breadth-first strategy explores each possible solution to a certain depth before moving on.
• In this particular example, the depth-first strategy found the solution in one less step than the breadth-
first. (The state in the bottom right hand side of Figure 3.7 was not explored by the depth-first parse.)
• In certain cases it is possible to put these simple search strategies into an infinite loop. For example,
consider a left-recursive rule that could be a first account of the possessive in English (as in the NP the
man ‘s coat): NP -> NP 's N
• With a naive depth-first strategy, a state starting with the nonterminal NP would be rewritten to a new
state beginning with NP s N.
• But this state also begins with an NP that could be rewritten in the same way.

• Unless an explicit check was incorporated into the parser, it would rewrite NPs forever!

• The breadth-first strategy does better with left-recursive rules, as it tries all other ways to rewrite the
original NP before coming to the newly generated state with the new NP.
• But with an ungrammatical sentence it would not terminate because it would rewrite the NP forever
while searching for a solution.
• For this reason, many systems prohibit left-recursive rules from the grammar.

• Many parsers built today use the depth-first strategy because it tends to minimize the number of backup
states needed and thus uses less memory and requires less bookkeeping.

DR.P.GANGADHARA REDDY 12
NLP LECTURE NOTES UNIT2

Bottom-Up Chart Parser

• The main difference between top-down and bottom-up parsers is the way the grammar rules are used.

• For example, consider the rule NP -> ART ADJ N

• In a top-down system you use the rule to find an NP by looking for the sequence ART ADJ N.

• In a bottom-up parser you use the rule to take a sequence ART ADJ N that you have found and
identify it as an NP.
• The basic operation in bottom-up parsing then is to take a sequence of symbols and match it
to the right-hand side of the rules.
• Bottom-up parser simply by formulating this matching process as a search process.
• The state would simply consist of a symbol list, starting with the words in the sentence.

• Successor states could be generated by exploring all possible ways to:

➢ rewrite a word by its possible lexical categories

➢ replace a sequence of symbols that matches the right-hand side of a grammar rule by its
left-hand side symbol

Grammar 3.8 A simple context-free grammar

• Unfortunately, such a simple implementation would be prohibitively expensive, as the parser would
tend to try the same matches again and again, thus duplicating much of its work unnecessarily.

• To avoid this problem, a data structure called a chart is introduced that allows the parser to store
the partial results of the matching it has done so far so that the work need not be reduplicated.

DR.P.GANGADHARA REDDY 13
NLP LECTURE NOTES UNIT2

• Matches are always considered from the point of view of one constituent called the key.
• For instance, consider Grammar 3.8.

• Assume you are parsing a sentence that starts with an ART. With this ART as the key,rules 2 and 3
are matched because they start with ART.
• To record this for analyzing the next key, you need to record that rules 2 and 3 could be continued at
the point after the ART.
• You denote this fact by writing the rule with a dot (o), so the modified rules are
2'. NP -> ART o ADJ N

3'. NP -> ART o N

• If the next input key is an ADJ, then rule 4 may be started, and the modified rule 2 may be extended
to give
2''. NP -> ART ADJ o N

• The chart maintains the record of all the constituents derived from the sentence so far in the parse.
• Chart also maintains the record of rules that have matched partially but are not complete.
These are called the active arcs.
• For example, after seeing an initial ART followed by an ADS in the preceding example, the chart
shown in Figure 3.9.

Figure 3.9 The chart after seeing an ADJ in position 2

• You should interpret this figure as follows.

DR.P.GANGADHARA REDDY 14
NLP LECTURE NOTES UNIT2

• There are two completed constituents on the chart:

ART1 from position 1 to 2

ADJ1 from position 2 to 3.

• There are four active arcs indicating possible constituents. These are indicated by the arrows and
are interpreted as follows (from top to bottom).
• There is a potential NP starting at position 1, which needs an ADJ starting at position 2.

• There is another potential NP starting at position 1, which needs an N starting at position 2.

• There is a potential NP starting at position 2 with an ADS, which needs N starting at position 3.
• Finally, there is a potential NP starting at position 1 with an ART and then an ADJ, which
needs an N starting at position 3.

• The basic operation of a chart-based parser involves combining an active arc with a
completed constituent.The result is either a new completed constituent or a new active arc that is an
extension of the original active arc. New completed constituents are maintained on a list called
the agenda until they themselves are added to the chart.This process is defined more precisely by the
arc extension algorithm shown in Figure 3.10.

Figure 3.10 The arc extension algorithm

• Given this algorithm, the bottom-up chart parsing algorithm is specified in Figure 3.11.

DR.P.GANGADHARA REDDY 15
NLP LECTURE NOTES UNIT2

Figure 3.11 A bottom-up chart parsing algorithm

• As with the top-down parsers, you may use a depth-first or breadth-first search strategy, depending
on whether the agenda is implemented as a stack or a queue.
• Also, for a full breadth-first strategy, you would need to read in the entire input and add the
interpretations of the words onto the agenda before starting the algorithm.
• Let us assume a depth-first search strategy for the following example.

• Consider using the algorithm on the sentence The large can can hold the water using Grammar
3.8 with the following lexicon:

the: ART

large : ADS
can : N,AUX,V

hold : N,V

water : N, V
• To best understand the example, draw the chart as it is extended at each step of the algorithm.
• The agenda is initially empty, so the word the is read and a constituent ART1 placed on the agenda.
Entering ART1: (the from 1 to 2)

Adds active arc NP -> ART o ADJ N from 1 to 2

DR.P.GANGADHARA REDDY 16
NLP LECTURE NOTES UNIT2

Adds active arc NP -> ART o N from 1 to 2

• Both these active arcs were added by step 3 of the parsing algorithm and were derived from rules
2 and 3 in the grammar, respectively. Next the word large is read and a constituent ADJ1 is created.
Entering ADJ1: (large from 2 to 3)

Adds arc NP -> ADS o N from 2 to 3

Adds arc NP -> ART ADJ o N from 1 to 3

• The first arc was added in step 3 of the algorithm.

• The second arc added here is an extension of the first active arc that was added when ART1 was
added to the chart using the arc extension algorithm (step 4).
• The chart at this point has already been shown in Figure 3.9.Notice that active arcs are never
removed from the chart.For example, when the arc NP → ART o ADJ N from 1 to 2 was extended,
producing the arc from 1 to 3, both arcs remained on the chart.This is necessary because the arcs
could be used again in a different way by another interpretation.
• The next word, can, has three constituents, N1, AUX1, and V1 are created as three interpretations.
• Entering N1 (can from 3 to 4)

• No active arcs are added in step 2, but two are completed in step 4 by the arc extension algorithm,
producing two NPs that are added to the agenda:
▪ The first, an NP from 1 to 4, is constructed from rule 2.

▪ The second, an NP from 2 to 4, is constructed from rule 4.

• These NPs are now at the top of the agenda.

Entering NP1: an NP (the large can from 1 to 4) Adding active arc S -> NP o VP from 1 to 4
Entering NP2: an NP (large can from 2 to 4) Adding arc S -> NP o VP from 2 to 4
Entering AUX1: (can from 3 to 4) Adding arc VP -> AUX o VP from 3 to 4
Entering V1: (can from 3 to 4) Adding arc VP —> V o NP from 3 to 4

• The chart is shown in Figure 3.12, which illustrates all the completed constituents (NP2, NP1,
ART1, ADJ1, N1, AUX1, V1) and all the uncompleted active arcs entered so far.

DR.P.GANGADHARA REDDY 17
NLP LECTURE NOTES UNIT2

Figure 3.12 After parsing the large can

• The next word is can again, and N2, AUX, and V2 are created.

Entering N2: (can from 4 to 5, the second can) Adds no active arcs
Entering AUX2: (can from 4 to 5) Adds arc VP -> AUX o VP from 4 to 5
Entering V2: (can from 4 to 5) Adds arc VP -> V o NP from 4 to 5

• The next word is hold, and N3 and V3 are created.

Entering N3: (hold from 5 to 6) Adds no active arcs

Entering V3: (hold from 5 to 6) Adds arc VP -> V o NP from 5 to 6

• The chart in Figure 3.13 shows all the completed constituents built so far, together with all the
active arcs, except for those used in the first NP.

Figure 3.13 The chart after adding hold, omitting arcs generated for the first NP

DR.P.GANGADHARA REDDY 18
NLP LECTURE NOTES UNIT2

Entering ART2: (the from 6 to 7) Adding arc NP -> ART o ADJ N from 6 to 7
Adding arc NP -> ART o N from 6 to 7
Entering N4: (water from 7 to 8) No active arcs added in step 3

An NP, NP3, from 6 to 8 is pushed onto the agenda, by completing arc NP -> ART o N from 6 to 7
Entering NP3: (the water from 6 to 8)

A VP, VP1, from 5 to 8 is pushed onto the agenda, by completing VP -> V o NP from 5 to 6
Adds arc S -> NP o VP from 6 to 8

• The chart at this stage is shown in Figure 3.14, but only the active arcs to be used in the remainder
of the parse are shown.

Figure 3.14 The chart after all the NPs are found, omitting all but the crucial active arcs

Entering VP1: (hold the water from 5 to 8)

A VP, VP2, from 4 to 8 is pushed onto the agenda, by completing VP -> AUX o VP from 4 to S
Entering VP2: (can hold the water from 4 to 8)
DR.P.GANGADHARA REDDY 19
NLP LECTURE NOTES UNIT2
An S, S1, is added from 1 to 8, by completing arcS -> NP o VP from 1 to 4
A VP, VP3, is added from 3 to 8, by completing arc VP -> AUX o VP from 3 to 4
An S, S2, is added from 2 to 8, by completing arc S -> NP o VP from 2 to 4
• you have derived an S covering the entire sentence, you can stop successfully.If you wanted to find
all possible interpretations for the sentence, you would continue parsing until the agenda became
empty.The chart would then contain as many S structures covering the entire set of positions as there
were different structural interpretations.In addition, this representation of the entire set of structures
would be more efficient than a list of interpretations, because the different S structures might share
common subparts represented in the chart only once.Figure 3.15 shows the final chart.

Figure 3.15 The final chart

Efficiency Considerations

• Chart-based parsers can be considerably more efficient than parsers that rely only on a search
because the same constituent is never constructed more than once.
• For instance, a pure top-down or bottom-up search strategy could require up to Cn operations to
parse a sentence of length n, where C is a constant that depends on the specific algorithm.
• Even if C is very small, this exponential complexity rapidly makes the algorithm unusable.
• A chart-based parser build every possible constituent between every possible pair of positions.
• This allows us to show that it has a worst-case complexity of K*n3, where n is the length of the sentence
DR.P.GANGADHARA REDDY 20
NLP LECTURE NOTES UNIT2
and K is a constant depending on the algorithm.A chart parser involves more work in each step, so K
will be larger than C.

• To contrast the two approaches, assume that C is 10 and that K is a hundred times worse, 1000. Given
a sentence of 12 words, the brute force search might take 1012 operations (that is, 1,000,000,000,000),
whereas the chart parser would take 1000 * 123 (that is, 1,728,000).
• Under these assumptions, the chart parser would be up to 500,000 times faster than the brute force
search on some examples!

Transition Network Grammars

• we have examined only one formalism for representing grammars, namely context-free rewrite
rules. Here another formalism that is useful in a wide range of applications. It is based on the notion of
a transition network consisting of nodes and labeled arcs. One of the nodes is specified as the initial
state, or start state.Consider the network named NP in Grammar 3.16, with the initial state
labeled NP and each arc labeled with a word category.

• Starting at the initial state, you can traverse an arc if the current word in the sentence is in the
category on the arc. If the arc is followed, the current word is updated to the next word. A phrase is
a legal NP if there is a path from the node NP to a pop arc (an arc labeled pop) that accounts for
every word in the phrase.
• This network recognizes the same set of sentences as the following context-free grammar:
NP -> ART NP1

NP1 -> ADJ NP1

NP1 -> N

• Consider parsing the NP a purple cow with this network. Starting at the node NP, you can follow the
arc labeled art, since the current word is an article— namely, a.

• From node NP1, you can follow the arc labeled adj using the adjective purple, and finally, again
from NP1, you can follow the arc labeled noun using the noun cow.

• Since you have reached a pop arc, a purple cow is a legal NP.

• Simple transition networks are often called finite state machines (FSMs).Finite state machines are
equivalent in expressive power to regular grammars (see Box 3.2), and thus are not powerful enough

DR.P.GANGADHARA REDDY 21
NLP LECTURE NOTES UNIT2
to describe all languages that can be described by a CFG.
• To get the descriptive power of CFGs, you need a notion of recursion in the network grammar.

• A recursive transition network (RTN) is like a simple transition network, except that it allows arc
labels to refer to other networks as well as word categories.

Thus, given the NP network in Grammar 3.16, a network for simple English sentences can be expressed
as shown in Grammar 3.17.

• Uppercase labels refer to networks.The arc from S to S1 can be followed only if the NP
network can be successfully traversed to a pop arc.RTNs allow true recursion—that is, a network
might have an arc labeled with its own name.

• Consider finding a path through the S network for the sentence The purple cow ate the grass.

• Starting at node S, to follow the arc labeled NP, you need to traverse the NP network.

• Starting at node NP, traverse the network as before for the input the purple cow.

• Following the pop arc in the NP network, return to the S network and traverse the arc to node S1.

• From node S1, you follow the arc labeled verb using the word ate.

• Finally, the arc labeled NP can be followed if you can traverse the NP network again.

• This time the remaining input consists of the words the grass.
DR.P.GANGADHARA REDDY 22
NLP LECTURE NOTES UNIT2
• You follow the arc labeled art and then the arc labeled noun in the NP network; then take the pop
arc from node NP2 and then another pop from node S3.
• Since you have traversed the network and used all the words in the sentence.

• The purple cow ate the grass is accepted as a legal sentence.

• In practice, RTN systems incorporate some additional arc types that are useful but not formally
necessary.
• Figure 3.18 summarizes the arc types, together with the notation that will be used to indicate
these arc types.

Arc Type Example How Used

succeeds only if current word is of the named category

CAT Noun
succeeds only if current word is identical to the label
WRD Of
succeeds only if named network can be successfully
PUSH NP
JUMP Jump always succeeds
succeeds and signals the successful end of the network
POP Pop
Figure 3.18 The arc labels for RTNs

• According to this terminology, arcs that are labeled with networks are called push arcs, and arcs
labeled with word categories are called cat arcs.
• In addition, an arc that can always be followed is called a jump arc.

Top-Down Parsing with Recursive Transition Networks

• An algorithm for parsing with RTNs can be developed along the same lines as the algorithms for
parsing CFGs.
• The state of the parse at any moment can be represented by the following:

▪ current position - a pointer to the next word to be parsed.

▪ current node - the node at which you are located in the network.

▪ return points - a stack of nodes in other networks where you will continue if you pop from the current
network.

DR.P.GANGADHARA REDDY 23
NLP LECTURE NOTES UNIT2
• First, consider an algorithm for searching an RTN that assumes that if you can follow an arc, it will be
the correct one in the final parse.
• Say you are in the middle of a parse and know the three pieces of information just cited.

• You can leave the current node and traverse an arc in the following cases:

Case 1: If arc names word category and next word in sentence is in that category,

Then (1) update current position to start at the next word;

(2) update current node to the destination of the arc.

Case 2: If arc is a push arc to a network N,

Then (1) add the destination of the arc onto return points;

(2) update current node to the starting node in network N.

Case 3: If arc is a pop arc and return points list is not empty,

Then (1) remove first return point and make it current node.

Case 4: If arc is a pop arc, return points list is empty and there are no words left,

Then (1) parse completes successfully.

• Grammar 3.19 shows a network grammar.

• The numbers on the arcs simply indicate the order in which arcs will be tried when more than
one arc leaves a node.

DR.P.GANGADHARA REDDY 24
NLP LECTURE NOTES UNIT2

• Figure 3.20 demonstrates that the grammar accepts the sentence 1 The 2 old 3 man 4 cried 5 by
showing the sequence of parse states that can be generated by the algorithm.

Figure 3.20 A trace of a top-down parse

• In the trace, each arc is identified by the name of the node that it leaves plus the number identifier.
• Thus arc S/1 is the arc labeled 1 leaving the S node.

• If you start at node 5, the only possible arc to follow is the push arc NP.

• As specified in case 2 of the algorithm, the new parse state is computed by setting the current node
to NP and putting node S1 on the return points list.
• From node NP, arc NP/1 is followed and, as specified in case 1 of the algorithm, the input is
checked for a word in category art.
• Since this check succeeds, the arc is followed and the current position is updated (step 3).

• The parse continues in this manner to step 5, when a pop arc is followed, causing the current node
to be reset to S1 (that is, the NP arc succeeded).
• The parse succeeds after finding a verb in step 6 and following the pop arc from the S network in
step 7.
• In this example, the parse succeeded because the first arc that succeeded was ultimately the correct
one in every case.

• However, with a sentence like The green faded, where green can be an adjective or a noun, this

DR.P.GANGADHARA REDDY 25
NLP LECTURE NOTES UNIT2
algorithm would fail because it would initially classify green as an adjective and then not find a
noun following.
• To be able to recover from such failures, we save all possible backup states as we go along, just as
we did with the CFG top down parsing algorithm.
• Consider this technique in operation on the following sentence:

1 One 2 saw 3 the 4 man 5

• The parser initially attempts to parse the sentence as beginning with the NP one saw, but after failing
to find a verb, it backtracks and finds a successful parse starting with the NP one.
• The trace of the parse is shown in Figure 3.21, where at each stage the current parse state is shown
in the form of a triple (current node, current position, return points), together with possible states for
backtracking. The figure also shows the arcs used to generate the new state and backup states.
Step Current State Arc to be Followed Backup States
1 (S, 1, NIL) S/1 NIL
NP/2 (& NP/3 for backup)
2 (NP, 1, (S1)) NIL
3 (NP1, 2, (S1)) NPl/2 (NP2, 2,(S1))
4 (NP2, 3, (S1)) NP2/l (NP2, 2,(S1))
no arc can be followed
5 (S1, 3, NIL) (NP2, 2,(S1))
6 (NP2, 2, (S1)) NP2/l NIL
7 (S1, 2, NIL) S1/l NIL
8 (S2, 3, NIL) S2/2 NIL
9 (NP, 3, (S2)) NP/1 NIL
10 (NP1, 4, (S2)) NP1/2 NIL
11 (NP2, 5, (S2)) NP2/1 NIL
12 (S2, 5, NIL) S2/1 NIL
13 parse succeeds NIL
Figure 3.21 A top-down RTN parse with backtracking

• This trace behaves identically to the previous example except in two places.

DR.P.GANGADHARA REDDY 26
NLP LECTURE NOTES UNIT2
• In step 2, two arcs leaving node NP could accept the word one.

• Arc NP/2 classifies one as a number and produces the next current state.

• Arc NP/3 classifies it as a pronoun and produces a backup state.

• This backup state is actually used later in step 6 when it is found that none of the arcs leaving node
S 1 can accept the input word the.
• Of course, in general, many more backup states are generated than in this simple example.
• In these cases, there will be a list of possible backup states.

• Depending on how this list is organized, you can produce different orderings on when the states are
examined.
• An RTN parser can be constructed to use a chart-like structure to gain the advantages of chart
parsing.
• In RTN systems, the chart is often called the well-formed substring table (WFST).

• Each time a pop is followed, the constituent is placed on the WFST, and every time a push is
found, the WFST is checked before the subnetwork is invoked.
• If the chart contains constituent(s) of the type being pushed for, these are used and the subnetwork
is not reinvoked.
• An RTN using a WFST has the same complexity as the chart parser described in the last section: K*n3,
where n is the length of the sentence.

Difference between Top-Down Parsing and Bottom-Up Parsing:

Top-Down Parsing and Bottom-Up Parsing are used for parsing a tree to reach the starting node of the
tree. Both the parsing techniques are different from each other. The most basic difference between the
two is that top-down parsing starts from top of the parse tree, while bottom-up parsing starts from
the lowest level of the parse tree.

Top-Down Parsing:Top-Down Parsing technique is a parsing technique which starts from the top level of
the parse tree, move downwards, evaluates rules of grammar. In other words, top-down parsing is a
parsing technique that looks at the highest level of the tree at start and then moves down to the parse
tree.The top-down parsing technique tries to identify the leftmost derivation for an input. It evaluates the
DR.P.GANGADHARA REDDY 27
NLP LECTURE NOTES UNIT2
rules of grammar while parsing. Consequently, each terminal symbol in the top-down parsing is produced
by multiple production of grammar rules.Since top-down parsing uses leftmost derivation, hence in this
parsing technique, the leftmost decision selects what production rule is used to construct the string.

Bottom-Up Parsing:Bottom-Up Parsing technique is again a parsing technique which starts from the
lowest level of the parse tree, move upwards and evaluates the rules of grammar. Therefore, the bottom up
parsing technique makes an attempt to decrease the input string to the start symbol of the grammar.
In bottom-up parsing, the parsing of a tree starts from the leaf node (bottom node) of the parse tree and
works towards the start node of the parse tree. Hence, it works in a bottom-up manner so its named.
The bottom-up parsing technique makes use of rightmost derivation. The main rightmost decision is to
select when a production rule is used to reduce the string to get the starting symbol of the parsing
tree.

Now, let us discuss the differences between top-down parsing and bottom-up parsing in detail.

Difference between Top-Down Parsing and Bottom-Up Parsing:

The following are some of the important differences between TopDown Parsing and BottomUp Parsing

Key Top Down Parsing Bottom Up Parsing

Strategy Top-down approach starts evaluating the Bottom-up approach starts evaluating the
parse tree from the top and move parse tree from the lowest level of the tree
downwards for parsing other nodes. and move upwards for parsing the node.
Attempt Top-down parsing attempts to find the Bottom-up parsing attempts to reduce the
left most derivation for a given string. input string to first symbol of the grammar.
Derivation Top-down parsing uses leftmost Bottom-up parsing uses the rightmost
Type derivation. derivation.
Objective Top-down parsing searches for a Bottom-up parsing searches for a production
production rule to be used to construct a rule to be used to reduce a string to get a
string. starting symbol of grammar.

Conclusion:The most significant difference that should note that top-down parsing uses leftmost
derivation, while bottom-up parsing uses the rightmost derivation.

DR.P.GANGADHARA REDDY 28