CS8602 – COMPILER DESIGN

UNIT I INTRODUCTION TO COMPILERS

Structure of a compiler – Lexical Analysis – Role of Lexical Analyzer – Input Buffering –

Specification of Tokens – Recognition of Tokens – Lex – Finite Automata – Regular
Expressions to Automata – Minimizing DFA.

INRODUCTION TO COMPILERS

Translator:
It is a program that translates one language to another.

source code Translator target code

Types of Translator:
1.Interpreter
2.Compiler
3.Assembler

1.Interpreter:
It is one of the translators that translate high level language to low level language.

high level language Interpreter low level language

During execution, it checks line by line for

errors. Example: Basic, Lower version of Pascal.

2.Assembler:
It translates assembly level language to machine code.

assembly language machine code

Assembler

Example: Microprocessor 8085, 8086.

3.Compiler:
It is a program that translates one language(source code) to another language (target
code).

source code target code

Compiler

It executes the whole program and then displays the

errors. Example: C, C++, COBOL, higher version of
Pascal.
 Compilation & Interpretation:

Difference between compiler and interpreter:

Compiler Interpreter
It is a translator that translates high level to It is a translator that translates high level to low
low level language level language
It displays the errors after the whole program
is It checks line by line for errors.
executed.
Examples: C, C++, Cobol, higher version of
Examples: Basic, lower version of Pascal.
Pascal.

LANGUAGE PROCESSORS:

Preprocessor
A preprocessor produce input to compilers. They may perform the following
functions.
1. Macro processing: A preprocessor may allow a user to define macros that
are short hands for longer constructs.
2. File inclusion: A preprocessor may include header files into the
program text.
3. Rational preprocessor: these preprocessors augment older languages with
more modern flow-of-control and data structuring facilities.
4. Language Extensions: These preprocessor attempts to add capabilities to the
language by certain amounts to build-in macro

COMPILER
Compiler is a translator program that translates a program written in (HLL) the
source program and translate it into an equivalent program in (MLL) the target
program. As an important part of a compiler is error showing to the programmer.

Source pgm m target pgm

Compiler
Err
or
ms
g
Executing a program written i n HLL programming language is basically of two parts.
The source program must first be compiled translated into a object program. Then the
results object program is loaded into a memory executed.

Source pgm obj pgm

Compiler

Obj pgm input Obj pgm opj pgm output

ASSEMBLER: programmers found it difficult to write or read programs in machine

language. They begin to use a mnemonic (symbols) for each machine instruction, which
they would subsequently translate into machine language. Such a mnemonic machine
language is now called an assembly language. Programs known as assembler were
written to automate the translation of assembly language in to machine language. The
input to an assembler program is called source program, the output is a machine language
translation (object program).

INTERPRETER: An interpreter is a program that appears to execute a source

program as if it were machine language.

Languages such as BASIC, SNOBOL, LISP can be translated using interpreters. JAVA also
uses interpreter. The process of interpretation can be carried out in following phases.
1. Lexical analysis
2. Synatx analysis
3. Semantic analysis
4. Direct Execution

Advantages:
Modification of user program can be easily made and implemented as execution
proceeds.
Type of object that denotes a various may change dynamically.
Debugging a program and finding errors is simplified task for a program used for
interpretation.
The interpreter for the language makes it machine independent.
 Disadvantages:

The execution of the program is slower.

Memory consumption is more.

Loader and Link-editor:

“A loader is a program that places programs into memory and prepares
them for execution.” It would be more efficient if subroutines could be translated
into object form the loader could “relocate” directly behind the user’s program. The
task of adjusting programs o they may be placed in arbitrary core locations is
called relocation. Relocation loaders perform four functions.

Software tools used in Analysis part:

1) Structure editor:
 Takes as input a sequence of commands to build a source program.
 The structure editor not only performs the text-creation and modification functions of
an ordinary text editor, but it also analyzes the program text, putting an appropriate
hierarchical structure on the source program.
 For example , it can supply key words automatically - while …. do and begin….. end.

2) Pretty printers :
 A pretty printer analyzes a program and prints it in such a way that the structure of
the program becomes clearly visible.
 For example, comments may appear in a special font.

3) Static checkers :
 A static checker reads a program, analyzes it, and attempts to discover potential
bugs without running the program.
 For example, a static checker may detect that parts of the source program can never
be executed.

4) Interpreters :
 Translates from high level language ( BASIC, FORTRAN, etc..) into machine language.
 An interpreter might build a syntax tree and then carry out the operations at the nodes
as it walks the tree.
 Interpreters are frequently used to execute command language since each operator
executed in a command language is usually an invocation of a complex routine such as
an
editor or complier.
ANALYSIS OF THE SOURCE PROGRAM

Analysis consists of 3 phases:

Linear/Lexical Analysis :

 It is also called scanning. It is the process of reading the characters from left to right
and grouping into tokens having a collective meaning.

 For example, in the assignment statement a=b+c*2, the characters would be grouped
into the following tokens:
i) The identifier1 ‘a’
ii) The assignment symbol (=)
iii) The identifier2 ‘b’
iv) The plus sign (+)
v) The identifier3 ‘c’
vi) The multiplication sign (*)
vii) The constant ‘2’

Syntax Analysis :

 It is called parsing or hierarchical analysis. It involves grouping the tokens of the

source program into grammatical phrases that are used by the compiler to synthesize
output.

 They are represented using a syntax tree as shown below:

a +

b *

c 2

 A syntax tree is the tree generated as a result of syntax analysis in which the
interior nodes are the operators and the exterior nodes are the operands.

 This analysis shows an error when the syntax is incorrect.

Semantic Analysis :

 It checks the source programs for semantic errors and gathers type information for the
subsequent code generation phase. It uses the syntax tree to identify the operators
and operands of statements.

 An important component of semantic analysis is type checking. Here the

compiler checks that each operator has operands that are permitted by the source
language specification.
THE PHASES OF COMPILER/ STRUCTURE OF COMPILER

A Compiler operates in phases, each of which transforms the source program from
one representation into another. The following are the phases of the compiler:

Main phases:
1) Lexical analysis
2) Syntax analysis
3) Semantic analysis
4) Intermediate code generation
5) Code optimization
6) Code generation

Sub-Phases:
1) Symbol table management
2) Error handling

LEXICAL ANALYSIS:

 It is the first phase of the compiler. It gets input from the source program and
produces tokens as output.
 It reads the characters one by one, starting from left to right and forms the tokens.
 Token : It represents a logically cohesive sequence of characters such as
keywords, operators, identifiers, special symbols etc.
Example: a + b = 20
Here, a,b,+,=,20 are all separate tokens.
Group of characters forming a token is called the Lexeme.
 The lexical analyser not only generates a token but also enters the lexeme into the
symbol table if it is not already there.
SYNTAX ANALYSIS:

 It is the second phase of the compiler. It is also known as parser.

 It gets the token stream as input from the lexical analyser of the compiler and
generates syntax tree as the output.
 Syntax tree:
It is a tree in which interior nodes are operators and exterior nodes are operands.
 Example: For a=b+c*2, syntax tree is

a +

b *

c 2

SEMANTIC ANALYSIS:

 It is the third phase of the compiler.

 It gets input from the syntax analysis as parse tree and checks whether the given syntax
is correct or not.
 It performs type conversion of all the data types into real data types.

INTERMEDIATE CODE GENERATION:

 It is the fourth phase of the compiler.

 It gets input from the semantic analysis and converts the input into output as
intermediate code such as three-address code.
 The three-address code consists of a sequence of instructions, each of which has
atmost three operands.
Example: t1=t2+t3

CODE OPTIMIZATION:

 It is the fifth phase of the compiler.

 It gets the intermediate code as input and produces optimized intermediate code
as output.
 This phase reduces the redundant code and attempts to improve the intermediate code
so that faster-running machine code will result.
 During the code optimization, the result of the program is not affected.
 To improve the code generation, the optimization involves
- deduction and removal of dead code (unreachable code).
- calculation of constants in expressions and terms.
- collapsing of repeated expression into temporary string.
- loop unrolling.
- moving code outside the loop.
- removal of unwanted temporary variables.
CODE GENERATION:

 It is the final phase of the compiler.

 It gets input from code optimization phase and produces the target code or object code
as result.
 Intermediate instructions are translated into a sequence of machine instructions
that perform the same task.
 The code generation involves
- allocation of register and memory
- generation of correct references
- generation of correct data types
- generation of missing code

SYMBOL TABLE MANAGEMENT:

 Symbol table is used to store all the information about identifiers used in the program.
 It is a data structure containing a record for each identifier, with fields for the attributes
of the identifier.
 It allows to find the record for each identifier quickly and to store or retrieve data
from that record.
 Whenever an identifier is detected in any of the phases, it is stored in the symbol table.

ERROR HANDLING:

 Each phase can encounter errors. After detecting an error, a phase must handle the
error so that compilation can proceed.
 In lexical analysis, errors occur in separation of tokens.
 In syntax analysis, errors occur during construction of syntax tree.
 In semantic analysis, errors occur when the compiler detects constructs with
right syntactic structure but no meaning and during type conversion.
 In code optimization, errors occur when the result is affected by the optimization.
 In code generation, it shows error when code is missing etc.

To illustrate the translation of source code through each phase, consider the statement
a=b+c*2. The figure shows the representation of this statement after each phase:
 a=b+c*2

Lexical analyser

id1=id2+id3*2

Syntax analyser
=
Symbol Table
id1 +
a id1
b id2 id2 *
c id3
id3 2
Semantic analyser =

id1 +

id2 *

id3 inttoreal

2
Intermediate code generator

temp1=inttoreal(2)
temp2=id3*temp1
temp3=id2+temp2
id1=temp3

Code optimizer

temp1=id3*2.0
id1=id2+temp1

Code generator

MOVF id3,R2
MULF #2.0,R2
MOVF id2,R1
ADDF R2,R1
MOVF R1,id1
GROUPING OF PHASES:

Compiler can be grouped into front and back ends:

- Front end: analysis (machine independent)

These normally include lexical and syntactic analysis, the creation of the symbol
table, semantic analysis and the generation of intermediate code. It also includes error
handling that goes along with each of these phases.

- Back end: synthesis (machine dependent)

It includes code optimization phase and code generation along with the necessary
error handling and symbol table operations.

Compiler passes

A collection of phases is done only once (single pass) or multiple times (multi pass)
 Single pass: usually requires everything to be defined before being used in
source program.
 Multi pass: compiler may have to keep entire program representation in memory.

Several phases can be grouped into one single pass and the activities of these phases
are interleaved during the pass. For example, lexical analysis, syntax analysis, semantic
analysis and intermediate code generation might be grouped into one pass.

COMPILER CONSTRUCTION TOOLS

These are specialized tools that have been developed for helping implement
various phases of a compiler. The following are the compiler construction tools:

1) Parser Generators:
-These produce syntax analyzers, normally from input that is based on a context-free
grammar.
-It consumes a large fraction of the running time of a compiler.
-Example-YACC (Yet Another Compiler-Compiler).

2) Scanner Generator:
-These generate lexical analyzers, normally from a specification based on regular
expressions.
-The basic organization of lexical analyzers is based on finite automation.

3) Syntax-Directed Translation:
-These produce routines that walk the parse tree and as a result generate intermediate code.
-Each translation is defined in terms of translations at its neighbour nodes in the tree.

4) Automatic Code Generators:

-It takes a collection of rules to translate intermediate language into machine language. The
rules must include sufficient details to handle different possible access methods for data.

5) Data-Flow Engines:
-It does code optimization using data-flow analysis, that is, the gathering of information
about
how values are transmitted from one part of a program to each other part.
ERRORS ENCOUNTERED IN DIFFERENT PHASES:

Compiler Errors

 Lexical errors (e.g. misspelled word)

 Syntax errors (e.g. unbalanced parentheses, missing semicolon)
 Semantic errors (e.g. type errors)
 Logical errors (e.g. infinite recursion)
Error Handling
 Report errors clearly and accurately
 Recover quickly if possible
 Poor error recover may lead to avalanche of errors

In Different Phases:

A parser should be able to detect and report any error in the program. It is expected that
when an error is encountered, the parser should be able to handle it and carry on parsing the rest
of the input. Mostly it is expected from the parser to check for errors but errors may be
encountered at various stages of the compilation process. A program may have the following
kinds of errors at various stages:

 Lexical : name of some identifier typed incorrectly

 Syntactical : missing semicolon or unbalanced parenthesis

 Semantical : incompatible value assignment

 Logical : code not reachable, infinite loop

There are four common error-recovery strategies that can be implemented in the parser to
deal with errors in the code.

Panic mode

When a parser encounters an error anywhere in the statement, it ignores the rest of the
statement by not processing input from erroneous input to delimiter, such as semi-colon. This is
the easiest way of error-recovery and also, it prevents the parser from developing infinite loops.

Statement mode

When a parser encounters an error, it tries to take corrective measures so that the rest of
inputs of statement allow the parser to parse ahead. For example, inserting a missing semicolon,
replacing comma with a semicolon etc. Parser designers have to be careful here because one
wrong correction may lead to an infinite loop.

Error productions

Some common errors are known to the compiler designers that may occur in the code. In
addition, the designers can create augmented grammar to be used, as productions that generate
erroneous constructs when these errors are encountered.
Global correction

The parser considers the program in hand as a whole and tries to figure out what the
program is intended to do and tries to find out a closest match for it, which is error-free. When
an erroneous input (statement) X is fed, it creates a parse tree for some closest error-free
statement Y. This may allow the parser to make minimal changes in the source code, but due to
the complexity (time and space) of this strategy, it has not been implemented in practice yet.

THE ROLE OF THE LEXICAL ANALYZER

Lexical analysis is the process of converting a sequence of characters into a sequence
of okens. A program or function which performs lexical analysis is called a lexical analyzer or
scanner. A lex often exists as a single function which is called by a parser or another function.
 The lexical analyzer is the first phase of a compiler.
 Its main task is to read the input characters and produce as output a sequence of
tokens that the parser uses for syntax analysis.

 Upon receiving a “get next token” command from the parser, the lexical analyzer reads
input characters until it can identify the next token.
ISSUES OF LEXICAL ANALYZER

There are three issues in lexical analysis:

 To make the design simpler.
 To improve the efficiency of the compiler.
 To enhance the computer portability.

TOKENS

A token is a string of characters, categorized according to the rules as a symbol (e.g.,

IDENTIFIER, NUMBER, COMMA). The process of forming tokens from an input stream
of characters is called tokenization.
A token can look like anything that is useful for processing an input text stream or
text file. Consider this expression in the C programming language: sum=3+2;

Lexeme Token type

Sum Identifier
= Assignment operator
3 Number
+ Addition operator
2 Number
; End of statement
LEXEME:

Collection or group of characters forming tokens is called Lexeme.

PATTERN:

A pattern is a description of the form that the lexemes of a token may take.

In the case of a keyword as a token, the pattern is just the sequence of characters that
form the keyword. For identifiers and some other tokens, the pattern is a more complex
structure that is matched by many strings.

Attributes for Tokens

Some tokens have attributes that can be passed back to the parser. The lexical
analyzer collects information about tokens into their associated attributes. The attributes
influence the translation of tokens.

i) Constant : value of the constant

ii) Identifiers: pointer to the corresponding symbol table entry.

Lexical Errors:

The following are the error-recovery actions in lexical analysis:

1) Deleting an extraneous character.

2) Inserting a missing character.

3) Replacing an incorrect character by a correct character.

4) Transforming two adjacent characters.

5) Panic mode recovery: Deletion of successive characters from the token until error
is resolved.

INPUT BUFFERING
We often have to look one or more characters beyond the next lexeme before we can
be sure we have the right lexeme. As characters are read from left to right, each character is
stored in the buffer to form a meaningful token as shown below:

Forward pointer
A = B + C

Beginning of the token Look ahead pointer

We introduce a two-buffer scheme that handles large look aheads safely. We then
consider an improvement involving "sentinels" that saves time checking for the ends of
buffers.
BUFFER PAIRS

 A buffer is divided into two N-character halves, as shown below

::E::=::M:* C:*::*:2:
eof
 lexeme_beginning
forwa
rd

 Each buffer is of the same size N, and N is usually the number of characters on one
disk block. E.g., 1024 or 4096 bytes.
 Using one system read command we can read N characters into a buffer.
 If fewer than N characters remain in the input file, then a special character,
represented by eof, marks the end of the source file.
 Two pointers to the input are maintained:
1. Pointer lexeme_beginning, marks the beginning of the current
lexeme, whose extent we are attempting to determine.
2. Pointer forward scans ahead until a pattern match is
found.
Once the next lexeme is determined, forward is set to the character at its
right end.
 The string of characters between the two pointers is the current lexeme.
After the lexeme is recorded as an attribute value of a token returned to the
parser, lexeme_beginning is set to the character immediately after the lexeme
just found.

Advancing forward pointer:

Advancing forward pointer requires that we first test whether we have reached the end
of one of the buffers, and if so, we must reload the other buffer from the input, and move
forward to the beginning of the newly loaded buffer. If the end of second buffer is reached, we
must again reload the first buffer with input and the pointer wraps to the beginning of the
buffer.

Code to advance forward pointer:

if forward at end of first half then

begin reload second half;
forward := forward + 1
end
else if forward at end of second half then
begin reload second half;
move forward to beginning of first half
end
else forward := forward + 1;

SENTINELS

 For each character read, we make two tests: one for the end of the buffer, and one to
determine what character is read. We can combine the buffer-end test with the test for
the current character if we extend each buffer to hold a sentinel character at the end.
 The sentinel is a special character that cannot be part of the source program, and a
natural choice is the character eof.
 The sentinel arrangement is as shown below:

: : E : : = : : M : * : eof C : * : : * : 2 : eof : : : eof

 lexeme_beginning
forwa
rd

Note that eof retains its use as a marker for the end of the entire input. Any eof
that appears other than at the end of a buffer means that the input is at an end.

Code to advance forward pointer:

forward : = forward + 1;
if forward ↑ = eof then begin
if forward at end of first half
then begin reload second
half;
forward := forward + 1
end
else if forward at end of second half
then begin reload first half;
move forward to beginning of
first half end
else /* eof within a buffer signifying end of input */
terminate lexical analysis
end

SPECIFICATION OF TOKENS

There are 3 specifications of tokens:

1) Strings
2) Language
3) Regular expression

Strings and Languages

An alphabet or character class is a finite set of symbols.

A string over an alphabet is a finite sequence of symbols drawn from that

alphabet. A language is any countable set of strings over some fixed alphabet.

In language theory, the terms "sentence" and "word" are often used as synonyms for
"string." The length of a string s, usually written |s|, is the number of occurrences of symbols
in s. For example, banana is a string of length six. The empty string, denoted ε, is the string of
length zero.

Operations on strings

The following string-related terms are commonly used:

1. A prefix of string s is any string obtained by removing zero or more symbols from the end
of string s.
For example, ban is a prefix of banana.

2. A suffix of string s is any string obtained by removing zero or more symbols from
 the beginning of s.
For example, nana is a suffix of banana.

3. A substring of s is obtained by deleting any prefix and any suffix from s.

For example, nan is a substring of banana.

4. The proper prefixes, suffixes, and substrings of a string s are those prefixes, suffixes,
and substrings, respectively of s that are not ε or not equal to s itself.

5. A subsequence of s is any string formed by deleting zero or more not necessarily

consecutive positions of s.
For example, baan is a subsequence of banana.

Operations on languages:

The following are the operations that can be applied to languages:

1.Union
2.Concatenation
3.Kleene closure
4.Positive closure

The following example shows the operations on

strings: Let L={0,1} and S={a,b,c}

1. Union : L U S={0,1,a,b,c}
2. Concatenation : L.S={0a,1a,0b,1b,0c,1c}
*
3. Kleene closure : L ={ ε,0,1,00….}
+
4. Positive closure : L ={0,1,00….}

Regular Expressions

Each regular expression r denotes a language L(r).

Here are the rules that define the regular expressions over some alphabet Σ and the
languages that those expressions denote:

1. ε is a regular expression, and L(ε) is { ε }, that is, the language whose sole member is
the empty string.

2. If ‘a’ is a symbol in Σ, then ‘a’ is a regular expression, and L(a) = {a}, that is, the
language with one string, of length one, with ‘a’ in its one position.

3. Suppose r and s are regular expressions denoting the languages L(r) and L(s).

Then, a) (r)|(s) is a regular expression denoting the language L(r) U L(s).

b) (r)(s) is a regular expression denoting the language
L(r)L(s). c) (r)* is a regular expression denoting (L(r))*.
d) (r) is a regular expression denoting L(r).
4. The unary operator * has highest precedence and is left associative.

5. Concatenation has second highest precedence and is left associative.

6. | has lowest precedence and is left associative.

Regular set
A language that can be defined by a regular expression is called a regular set.
If two regular expressions r and s denote the same regular set, we say they are equivalent and
write r = s.

There are a number of algebraic laws for regular expressions that can be used to
manipulate into equivalent forms.
For instance, r|s = s|r is commutative; r|(s|t)=(r|s)|t is associative.
Regular Definitions
Giving names to regular expressions is referred to as a Regular definition. If Σ is an
alphabet of basic symbols, then a regular definition is a sequence of definitions of the form

dl → r 1
d2 → r2
………
dn → rn

1. Each di is a distinct name.

2. Each ri is a regular expression over the alphabet Σ U {dl, d2,. . . , di-l}.

Example: Identifiers is the set of strings of letters and digits beginning with a letter. Regular
definition for this set:
letter → A | B | …. | Z | a | b | …. | z |
digit → 0 | 1 | …. | 9
id → letter ( letter | digit ) *
Shorthands
Certain constructs occur so frequently in regular expressions that it is convenient to
introduce notational shorthands for them.
1. One or more instances (+):

- The unary postfix operator + means “ one or more instances of” .

+
- If r is a regular expression that denotes the language L(r), then ( r ) is a regular expression
+
that denotes the language (L (r ))
+
- Thus the regular expression a denotes the set of all strings of one or more a’s.
+ *
- The operator has the same precedence and associativity as the operator .
2. Zero or one instance ( ?):

- The unary postfix operator ? means “zero or one instance of”.

- The notation r? is a shorthand for r | ε.

- If ‘r’ is a regular expression, then ( r )? is a regular expression that denotes the language
L( r ) U { ε
}.

3. Character Classes:

- The notation [abc] where a, b and c are alphabet symbols denotes the regular
expression a | b | c.

- Character class such as [a – z] denotes the regular expression a | b | c | d | ….|z.

 - We can describe identifiers as being strings generated by the regular
expression, [A–Za–z][A–Za–z0–9]*

Non-regular Set

A language which cannot be described by any regular expression is a non-regular set.

Example: The set of all strings of balanced parentheses and repeating strings cannot be
described by a regular expression. This set can be specified by a context-free grammar.

RECOGNITION OF TOKENS
Consider the following grammar fragment:

stmt → if expr then stmt

| if expr then stmt else stmt
|ε

expr → term relop term

| term

term → id
| num

where the terminals if , then, else, relop, id and num generate sets of strings given by
the following regular definitions:
if → if
then →
then else →
else
relop → id → num →
<|<=|=|<>|>|>=
*
letter(letter|digit)
+ + +
digit (.digit )?(E(+|-)?digit )?

For this language fragment the lexical analyzer will recognize the keywords if, then,
else, as well as the lexemes denoted by relop, id, and num. To simplify matters, we assume
keywords are reserved; that is, they cannot be used as identifiers.
Transition diagrams
It is a diagrammatic representation to depict the action that will take place when a
lexical analyzer is called by the parser to get the next token. It is used to keep track of
information about the characters that are seen as the forward pointer scans the input.
Transition diagram for Identifiers:

FINITE AUTOMATA

Finite Automata is one of the mathematical models that consist of a number of states
and edges. It is a transition diagram that recognizes a regular expression or grammar.

Types of Finite Automata

There are tow types of Finite Automata :

 Non-deterministic Finite Automata (NFA)
 Deterministic Finite Automata (DFA)

Non-deterministic Finite Automata

NFA is a mathematical model that consists of five tuples denoted by

M = {Qn, Ʃ, δ, q0, fn}
Qn – finite set of states
Ʃ – finite set of input symbols
δ – transition function that maps state-symbol pairs to set of states
q0 – starting
state fn – final
state

Deterministic Finite Automata

DFA is a special case of a NFA in

which i) no state has an ε-
transition.
ii) there is at most one transition from each state on any input.

DFA has five tuples denoted by

M = {Qd, Ʃ, δ, q0, fd}
Qd – finite set of states
Ʃ – finite set of input symbols
δ – transition function that maps state-symbol pairs to set of states
q0 – starting
state fd – final
state

Construction of DFA from regular expression

The following steps are involved in the construction of DFA from regular expression:
i) Convert RE to NFA using Thomson’s
rules ii) Convert NFA to DFA
iii) Construct minimized DFA

Constructing an NFA from a RE : Thompson’s Construction Algorithm

• Input: A regular expression r over an alphabet

• Output: An NFA N accepting L(r)
– First r is parsed into its constituent sub-expressions
– Then NFA is constructed for each of the basic symbols
– If same symbol occurs repeatedly, separate NFA is constructed for each
occurrence
 Example - ab*c | a(b|c*)
Parse Tree for this regular
expression:
where i is the start state
of N(s) (or new under the
alternative) and f is the
final state of N(t) (or
new). Overlap maps final
states of N(s) to start state
of N(t). 

a  b  c
r5 : r 3 r4


 

r12 : r11 a  c 
r10

Conversion of NFA to DFA

1. Algorithm Constructs a Transition Table for DFA from NFA

2. Each state in DFA corresponds to a set of states of the NFA
3. DFA state keeps track of set of all states NFA can be in after reading each input symbol
 4. After reading a1a2…an DFA is in a state that represents a subset T of NFA states
reachable form start states
a. Î moves
b. non-determinism
c. Handling Ambiguity
Input: NFA N; Output: DFA D
initialize Dstates to unmarked ε-closure(s0)
while there is an unmarked state T in Dstates
mark T
for each input symbol a
U := ε-closure(move(T,a))
if U is not in Dstates
add U as unmarked state to Dstates
Dtran[T,a] := U


2 a 3 

st 0  1 6  7 a 8 b 9
 b
5 
ar
4 b 1
t  0

b C b

sta a b b
rt A B D E
a
a
a

DFA Minimization

 Dead state for undefined transitions

 String w distinguishes state s from state t if
 o by starting from s and giving w – accepting state results;
o by starting from t and giving w – non- accepting state results;
 Find all groups of states that can be distinguished by some input string
 Merge all states that cannot be distinguished
 Partition states into two groups:
o Acceptance states and others
o From a given group transitions must be to same group – if not partition
o Continue until no further partitions are possible

1. Construct initial partition P of S with two groups: accepting/ non-accepting.

2. (Construct Pnew ) for each group G of P do begin
o Partition G into subgroups such that two states s, t of G are in the same subgroup
iff for all symbols a state’s s, t have transitions on a to states of the same group of
P.
o Replace G in Pnew by the set of all these subgroups.
3. Compare Pnew and P. If equal, Pfinal:= P then proceed to 4, else set P := Pnew and goto 2.
4. Aggregate states belonging in the groups of Pfinal
5. If M’ has a dead state d from M’. Also remove any states not reachable from the start state.
Any transitions to d from other states become undefined.

a
C b

A B D E

a
C b

A B D E
 A B Dab E

Converting Regular Expression to DFA

 A regular expression can be converted into a DFA (without creating a NFA first).
 First the given regular expression is augmented by concatenating it with a special
symbol #.
 r è (r)#
 augmented regular expression
 Then a syntax tree is created for this augmented regular expression.
 In this syntax tree, all alphabet symbols, # , and the empty string in the augmented
regular expression will be on the leaves, and
 All inner nodes will be the operators
 Then each alphabet symbol and # will be numbered (position numbers).
Types of Interior nodes
 Cat-node
 Star-node
 Or-node

Regular Expression è DFA

Followpos :

Followpos is defined for the positions

(positions assigned to leaves).
followpos(i) : is the set of positions which can follow

Syntax tree of (a
the position i in the strings generated by
the augmented regular expression.
For example, ( a | b) * a #
1 2 3 4
followpos(1) = {1,2,3}
 followpos(2) = {1,2,3} each symbol is at a le
followpos(3) = {4}
followpos(4) = {}
firstpos, lastpos, nullable
inner nodes are oper
To evaluate followpos, three more functions are to be defined for the nodes (not just for leaves)
of the syntax tree.
 firstpos(n) -- the set of the positions of the first symbols of strings generated by the
sub-expression rooted at n.
 lastpos(n) -- the set of the positions of the last symbols of strings generated by the
sub-expression rooted at n.
 nullable(n) -- true if the empty string is a member of strings generated by the sub-
expression rooted by n false otherwise

Evaluation of firstpos, lastpos, nullable

n nullable(n) firstpos(n) lastpos(n)

leaf labeled
true F F
e
leaf labeled
with position false {i} {i}
i
nullable(c1) or firstpos(c1) È
c1 + c2 lastpos(c1) È lastpos(c2)
nullable(c2) firstpos(c2)
if (nullable(c1)) if (nullable(c2))
nullable(c1) and firstpos(c1) È lastpos(c1) È
C1·C2 nullable(c2) firstpos(c2) lastpos(c2)
else firstpos(c1) else lastpos(c2)

C1* true firstpos(c1) lastpos(c1)

Evaluation of followpos

Two-rules define the function followpos:

1. If n is concatenation-node with left child c1 and right child c2,and i is a position in
lastpos(c1), then all positions in firstpos(c2) are in followpos(i).
2. If n is a star-node, and i is a position in lastpos(n), then all positions in firstpos(n) are in
followpos(i).
If firstpos and lastpos have been computed for each node, followpos of each position
can be computed by making one depth-first traversal of the syntax tree.

For cat node,

 followpos(i)
Cat
n =
{ firstpos(c
ilast 1C
-
C ) }
firstpo
2
For star node,
2
no
pos(c1 s(c2) de
)followpos(i) =
n Star-
firstp } ilastpos
{ firstpos(n)node
os(n) C1 (n)
Example -- ( a | b) * a #

{1,2,3} {4
}
red –
firstpo
{4} #{4}
{1,2,3} {3} 4
{1,2} {1 ,2} {3} a{3}
*
3

p o s (1 ) = { s1,2,3}
Algorithm (RE è DFA) {1,2}
follo w
3e}–
{1,2}
|
{2}
o s ( 2 ) =
wp lastpos, nullable { b
1 l
,2u ,
{1} a{1} {2} b
follofirstpos,
 Create the syntax tree of (r) #
r E xp r e ss
1
2
 Calculate the functions: followpos,
ll o w p o s (3
u c d 4
{lfa
) t=estate. }
o s
r o
thpe Rs e gu la
n ow be f o
 Put firstpos(root) into the states of DFA
co n
as
s
an
tr
unmarked

The DF A c a n wp os (4 ) {}
while (there is an unmarked state S in the states of DFA) do
=
o mark S
fo ll o
o for each input symbol a do
 let s1,...,sn are positions in S and symbols in those positions are a
 S’ ç followpos(s1) È ... È followpos(sn)
 move(S,a) ç S’
 if (S’ is not empty and not in the states of DFA)
 put S’ into the states of DFA as an unmarked state.
 the start state of DFA is firstpos(root)
 the accepting states of DFA are all states containing the position of #
Example -- ( a | b) * a #
 A LANGUAGE FOR SPECIFYING LEXICAL ANALYZER:

There is a wide range of tools for constructing lexical analyzers.

 Lex
 YACC

LEX
Lex is a computer program that generates lexical analyzers. Lex is commonly used with
the yacc parser generator.

Creating a lexical analyzer

 First, a specification of a lexical analyzer is prepared by creating a program lex.l in

the Lex language. Then, lex.l is run through the Lex compiler to produce a C
program lex.yy.c.
 Finally, lex.yy.c is run through the C compiler to produce an object program
a.out, which is the lexical analyzer that transforms an input stream into a sequence of tokens.

Lex Specification

A Lex program consists of three parts:

{ definitions }
%%
{ rules }
%%
{ user subroutines }

 Definitions include declarations of variables, constants, and regular definitions

 Rules are statements of the

 form p1 {action1}
p2 {action2}
…
pn {actionn}
where pi is regular expression and actioni describes what action the lexical
analyzer should take when pattern pi matches a lexeme. Actions are written in C
code.

 User subroutines are auxiliary procedures needed by the actions. These can be
compiled separately and loaded with the lexical analyzer.

Here is a list of Lex operators with examples:

Operator notation Example Meaning

* (astersk) a* Set of all strings of zero
    or more a's, i.e. . (empty a, aa,
    aaa ... )
| (or) a|b Either a or b
+ a+ One or more instances of
    a i.e. a, aa, aaa etc.
? a? Zero one of instance of a
[,] [a b c] a | b | c. An alphabetical
    character class such as [a-zl
    denotes the regular
    expression a | h | ... | z.

Here are few more examples of Lex operators:

 (a | b) (a | h) denotes (aa, ab. ba, bb), the set of an strings of a's and b's of length two.
Another regular expression for this same set is aa | ab | ba | bb

(a b)* denotes the set of all strings containing zero or more instances of an a or b, that is
the set of all strings of a's and b's.

X | X'YZ denotes the set containing the string X and all strings consisting of zero or
more X's followed by YZ.

Given a familiarity with Lex and C, a lexical analyzer for almost any programming language can
be written in a very short time. Lex does not have application solely in the generator of lexical
analysers. It can also be used to assist in the implementation of almost any text pattern matching
application, such as text editing, code conversion and so on.