Lexical Analysis

Module 5

Prepared By: Vaibhav Ambhire

Content
• Lex Compiler
• Structure of LEX compiler
• Design of Lexical Analyzer Generator
• Data Structures used in Lexical Analysis

Prepared By: Vaibhav Ambhire

 Lex Source
Program LEX
lex.yy.c
lex.l Compiler

LEX
C
Compiler lex.yy.c
Compiler
a.out

Input Sequence
Stream a.out
of
Tokens

Prepared By: Vaibhav Ambhire

 Lex program consists of three parts:-

%{

Structure Declaration

%}
of
%%
LEX
Rule Section
program
%%

Auxiliary Procedure

Prepared By: Vaibhav Ambhire

 Translation Rules are of the form:-

P1 {action1}

P2 {action2}
Structure
of ....

LEX Pi {action3}

program • where each Pi is a regular expression and

• each action i is a program fragment describing what action

the lexical analyzer should take when pattern pi matches a
lexeme

Prepared By: Vaibhav Ambhire

Structure Auxiliary Procedures

• This section holds whatever auxiliary functions are used

of in the actions

LEX • These functions can be compiled separately and loaded

with the lexical analyzer

program

Prepared By: Vaibhav Ambhire

 • When called by the parser, the lexical analyzer begins
reading its remaining input, one character at a time

Working of • It reads until it finds the longest prefix of the input that
matches one of the patterns

Lexical • It then executes action i which returns the control to the

parser
Analyzer • But if it does not then the lexical analyzer proceeds to find
additional lexemes, until one of the corresponding actions
with causes a return to the parser.

Parser • The lexical analyzer returns a single quantity i.e the token
to the parser

• To pass an attribute value with information about the

lexeme, a global variable is set known as yylval
Prepared By: Vaibhav Ambhire
 LEX program for Tokens
%{
/* Definitions of manifest constants

LT, LE, EQ, NE, GT, GE, IF, THEN, ELSE, ID, NUMBER, RELOP */

%}

/* Regular definitions */

delim [\t \n]

ws {delim}+
letter [A-Za-z]
digit [0-9]
id {letter}( {letter} | {digit} )*
number {digit}+(\ . {digit} +) ? (E [+-] ? {digit}+) ?

Prepared By: Vaibhav Ambhire

 LEX program for Tokens
%%
{ws} { /* No action and No Return */ }
if { return (IF); }
else { return (ELSE); }
then { return (THEN); }
{id} { yylval = (int) installID(); return (ID); }
{number} { yylval = (int) installNum(); return (NUMBER); }
“<” { yylval = LT; return(RELOP); }
“<=” { yylval = LE; return(RELOP); }
“=” { yylval = EQ; return(RELOP); }
“<>” { yylval = NE; return(RELOP); }
“>” { yylval = GT; return(RELOP); }
“>=” { yylval = GE; return(RELOP); }
%%
Prepared By: Vaibhav Ambhire
 LEX program for Tokens
int installID() {

/* Function to install the lexeme whose first character is pointed to by yytext, and whose length
is yyleng, into the symbol table and return a pointer to Symbol Table

*/ }

int installNum() {

/* Similar to installID but puts numerical constants into a separate table */ }

Prepared By: Vaibhav Ambhire

 Steps for installation of LEX and YACC

• sudo apt-get update

• sudo apt-get install flex

• sudo apt-get install bison

LEX Compilation steps:

Program • lex program.l

• gcc lex.yy.c -lfl

• ./a.out

Prepared By: Vaibhav Ambhire

 %{

%}

%%

LEX \n { printf(“\n Hello Good Morning”);

Program
%%

Example 1
void main()
{
yylex();
}

Prepared By: Vaibhav Ambhire

 %{
char name[10];
%}

%%
[\n] {printf("\n Hi........%s......Good Morning\n",name); return 1;}

LEX %%
void main()
Program {
char opt;
do {
Example 2 printf("\nWhat is your name?"); scanf("%s",name);
yylex();
printf("\nPress y to continue"); scanf("%c",&opt);
}
while(opt=='y');
}

Prepared By: Vaibhav Ambhire

 %{

int linecount = 0, charcount = 0;

%}

%%

LEX . { charcount++; }

Program \n { linecount++; charcount++; }

%%

Example 3 void main()

yylex();
printf("# of lines = %d, # of chars = %d", linecount, charcount);

Prepared By: Vaibhav Ambhire

 LEX Program Example 4

%{
void display(int flag,char *t)
void display(int, char *);
{
int flag;
%} if(flag==1)

%% printf("\nThe given character %s is vowel \n",t);

[a|e|i|o|u] { flag =1; display(flag,yytext); } }
. { flag =0; display(flag,yytext); }
else
%%
{

printf("\nThe given character %s is not vowel \n",t);

void main()
{ }
printf("\nEnter the word:"); yylex(); }
}

Prepared By: Vaibhav Ambhire

 Input Buffer Design of Lexical Analyzer
Lexeme
Generator

• A Lex program is turned into a transition table

and actions which are used by a finite
Forward automaton simulator
Lexeme begin
• The program that serves as the lexical analyzer
includes a fixed program that simulates an
Automaton automaton
Simulator
• The rest of the lexical analyzer consists of
following component

1. A transition table of the automaton

2. Those functions that are passed directly

through Lex to the output
Transition 3. The action from the input program which is to
Table be invoked at the appropriate time by
Lex Lex _________ Automaton Simulator
Program Compiler Action
Prepared By: Vaibhav Ambhire
 Input Buffer Design of Lexical Analyzer
Lexeme
Generator

• To construct the automaton take each regular

Forward expression pattern in the Lex program and
Lexeme begin
convert it using Algorithm to an NFA

Automaton • We need a single automaton that will recognize

Simulator
lexemes matching any of the patterns in the
program

• Hence combine all the NFA s into one by

introducing a new start state with transitions
Transition to each of the start states of the NFA s Ni for
Table
Lex Lex _________ pattern pi
Program Compiler Action
Prepared By: Vaibhav Ambhire
 • Symbol: Identifier used in the Source Program
• Examples: Names of variables, functions and
Procedures
Data • Symbol Table: To maintain information about
Structure attributes of symbol
used in • Operations on Symbol Table:
Lexical
1. Add a symbol and its attributes
Analyzer
2. Locate a symbol’s entry
3. Delete a symbol’s entry
4. Access a symbol’s entry

Prepared By: Vaibhav Ambhire

 • Design Goal of Symbol Table

1. The Table’s organization should facilitate efficient search

2. ii. Table should be compact (Less Memory)

Data
Structure • Time Space Trade off
used in
Lexical • To improve search efficiency, allocate more memory to
Analyzer symbol table

• Organization of Entries:

1. Linear Data Structure

2. Non-Linear Data Structure

Prepared By: Vaibhav Ambhire
 Symbol Table Entry Format

Data • Number of Fields to accommodate attributes of one symbol

Structure
used in • Symbol Field: The symbol to be stored

Lexical • Key Field: Basis for the search in table

Analyzer
• Entry of Symbol is called record

• Each entry can be of type fixed length, variable length or hybrid

Prepared By: Vaibhav Ambhire

 Symbol Class Attributes

Data Type, Length, number and bounds of

Variable
dimensions
Structure
used in Procedure
Number of parameters, address of
Lexical parameter list

Analyzer
Number of parameters, address of
Function parameter list, type and length of
returned value

Label Statement Number

Prepared By: Vaibhav Ambhire