Diving deep into twig

DIVING DEEP
INTO TWIG
or

TWIG INTERNALS

MATTHIAS NOBACK
Zeist, the Netherlands
Feature addition in March 2014
Started as a web designer (2003)
Was employed for 6 years at
several companies...

Now self employed: Noback's Office
Blog: php-and-symfony.matthiasnoback.nl
Twitter: @matthiasnoback

Tony Piper

Follow

@tonypiper

@matthiasnoback enjoyed reading your book should be
mandatory reading for any Symfony developer.
6:30 PM 7 Sep 2013

Damon Jones

Follow

@damon__jones

@matthiasnoback An excellent job, good sir. Your blog posts
and now your book are really showing us the way to be better
developers.
10:47 PM 5 Sep 2013

leanpub.com/a-yearwith-symfony

ARMIN RONACHER
Developed Jinja for Python (2006)
Ported Jinja to PHP, called it Twig
(2008)

FABIEN POTENCIER
Lead developer of the Symfony project
Was looking for a Django-like templating
engine for Symfony2
Found Twig and started "hacking" on it

OVERVIEW
Sample
Extensions
Lexer
Parser
Token parsers
Node visitors
Compiler

BLOCKS AND VARIABLES
{ frue i ues%
% o sr n sr }
<i{ ue.ae}<l>
l>{ srnm }/i
{ edo %
% nfr }

FUNCTIONS
Ia { rno('ap' 'a')}
m { adm[hpy, sd] }
{ stsesrne1 2)%
% e tp=ag(, 0 }
<icas"{cce[od,'vn] i }"../i
l ls={ yl('d' ee', ) }>.<l>

FILTERS
H,{ ue.aecptlz }!
i { srnm|aiaie }
{ ue.hnNmesfrt}
{ srpoeubr|is }
{ ps.asji(,' }
{ ottg|on' ) }

TESTS
{ i eprDt i dfnd%..
% f xiyae s eie }.
{ i uesi epy%..
% f sr s mt }.
{ i ii od%
% f
s d }

TAGS
{ i ue.nbe %..
% f sreald }.
{ boksdbr%..
% lc iea }.
{ icue'poiehm.wg %..
% nld _rfl.tlti' }.

EXTENSIONS
itraeTi_xesoItrae
nefc wgEtninnefc
{
pbi fnto gtucin(;
ulc ucin eFntos)
pbi fnto gtitr(;
ulc ucin eFles)
pbi fnto gtet(;
ulc ucin eTss)
}

..
.

$n-adxeso(etnin / i tewy
ev>dEtnin$xeso) / s h a

(Symfony2: create a service with a t i . x e s o tag)
wgetnin

FUNCTIONS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtucin(
ulc ucin eFntos)
{
rtr ary
eun ra(
nwwgSmlFnto(
e Ti_ipeucin
'yucin,
mFnto'
fnto (tig {
ucin $hn)
rtr srnf'hsi <>y/>%.,$hn)
eun pit(Ti s bm<b s' tig;
}
)
)
;
}
}
{ mFnto(cmue" }
{ yucin"optr) }

FILTERS
{
pbi fnto gtitr(
ulc ucin eFles)
{
rtr ary
eun ra(
nwwgSmlFle(
e Ti_ipeitr
'ie,
mn'
fnto (wa,$ie=tu){
ucin $ht mn
re
rtr srnf
eun pit(
's(hc % mn),
% wih s ie'
$ht
wa,
$ie?'s:i nt
mn
i''s o'
)
;
}
)
)
;
}
}
{ tigmn(as)}
{ hn|iefle }

TESTS
{
pbi fnto gtet(
ulc ucin eTss)
{
rtr ary
eun ra(
nwwgSmlTs(
e Ti_ipeet
'_ofrne,
acneec'
fnto(nm){
ucin$ae
rtr $ae=='yfnCn;
eun nm = Smoyo'
}
)
)
;
}
}
{ i "yfnCn i acneec %Itl yus{ edf%
% f Smoyo" s _ofrne } od o o% ni }

TAGS
{ cneec %
% ofrne }

Needs some explaining...

TWIG ENVIRONMENT
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-stodrnwwgLae_ieytm_DR_'tmlts);
ev>eLae(e Ti_odrFlsse(_I_./epae')
$otx =ary
cnet
ra(
..
.
)
;
eh $n-rne(idxhm.wg,$otx)
co ev>edr'ne.tlti' cnet;

WHAT IS A TEMPLATE?
casTi_niomn
ls wgEvrnet
{
..
.

}

pbi fnto rne(nm,ary$otx =ary)
ulc ucin edr$ae ra cnet
ra()
{
rtr $hs>odepae$ae-rne(cnet;
eun ti-laTmlt(nm)>edr$otx)
}

A template is a class that implements
wgTmltItrae
Ti_epaenefc
l a T m l t ( returns an instance of such a class
odepae)

COMPILED TEMPLATE CLASS

/ idxhm.wg*
* ne.tlti /
cas_TiTmlt_12098fe1d89ef1e6etnsTi_epae
ls _wgepaedd753ba3f93c0e59 xed wgTmlt
{
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
i (se(cnet"ae]){$nm_=$otx[nm";}es {$nm_=nl
f ist$otx[nm")
_ae
cnet"ae]
le
_ae
u
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh "i i afl"
co
s n
ie;
}

}

pbi fnto gtepaeae)
ulc ucin eTmltNm(
{
rtr "ne.tlti"
eun idxhm.wg;
}

BEFORE
<>{nm }

AFTER
eh 'p'
co <>;
i (se(cnet"ae]){
f ist$otx[nm")
$nm_=$otx[nm";
_ae
cnet"ae]
}es {
le
$nm_=nl;
_ae
ul
}
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh '/>;
co <p'

HOW TWIG CREATES A
TEMPLATE CLASS
1. Retrieve the source (written in "Twig") from the loader(s)
2. Compile the source to a PHP class

COMPILING A TEMPLATE

THE LEXER

LEXER
1. Matches the input string against known patterns
("lexemes")
2. Determines token types for these matches
3. Returns a stream of tokens

FINDING TWIG BLOCKS
The lexer first checks for the position of the main markers:
Start of block:
{
%
Start of variable: {
{
Start of comment: {
#
Then the lexer
1. iterates over the resulting positions, while
2. checking some basic syntax rules, and
3. collecting tokens on its way to E F
O

TOKEN TYPES
Tokens have:
a type
a value (optional)
a line number
BOKSAT
LC_TR
BOKED
LC_N
VRSAT
A_TR
VRED
A_N
TX
ET
NM
AE
NME
UBR
SRN
TIG
OEAO
PRTR
PNTAIN
UCUTO
..
.

{
%
%
}
{
{
}
}
raw template data
f r i , etc.
o, f
a number
" . "or ' . '
..
..
+ * ~ etc.
, , ,
| [ { etc.
, , ,
..
.

Take this template:
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }

/l
$ee =$n-gtee(;
lxr
ev>eLxr)
$epae=..
tmlt
.;
$oeSra =$ee-tknz(tmlt)
tkntem
lxr>oeie$epae;

TOKEN STREAM
eh $oeSra;
co tkntem

BOKSAT
LC_TR
NM(ni)
AEedf
BOKED
LC_N
TX(u>
ET<l)
BOKSAT
LC_TR
NM(o)
AEfr
NM(tm
AEie)
OEAO(n
PRTRi)
NM(tm)
AEies
BOKED
LC_N
..
.
EF
O

{
%
edf
ni
%
}
raw template
data
{
%
fr
o
ie
tm
i
n
ies
tm
%
}
..
.
end of input

{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }

/l

STATES
To keep track of what the lexer is doing.
DATA

lexing raw template data (start state)

BLOCK lexing a block
VAR

lexing a variable

STRING lexing a string

CONSECUTIVE STATES
DT
AA
BOK
LC
BOK
LC
DT
AA
BOK
LC
BOK
LC
BOK
LC
BOK
LC
BOK
LC
DT
AA
VR
A
..
.
DT
AA

template data
block e d f
ni
starts
block e d f
ni
ends
<l
u>
block f rstarts
o
name: i e
tm
name: i
n
name: i e s
tm
block f rends
o

variable starts,
name: i e
tm
...

/l

{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }

/l

SYNTAX VALIDATION
Each block and variable should be closed
{ fr{ i
% o % f

Brackets ( [should be closed symmetrically
{
{ [a }
{ '' }

Closing brackets ] )can not occur first
}
{ ]}
{
}

SYNTAX VALIDATION
(CONTINUED)
Expressions may not contain unexpected characters
{ }
{
}

Comments should be closed
{ cmet
# omn

FROM SYNTAX TO
SEMANTICS
The resulting list of tokens may be semantically incorrect.
In the Twig language, that is...
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }

/l


THE PARSER

PARSING THE TOKEN
STREAM
The parser
Processes the token stream
Builds an Abstract Syntax Tree for the template

CREATING THE ABSTRACT
SYNTAX TREE
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }

/l
$epae=..
tmlt
.;
$oeSra =$ee-tknz(tmlt)
tkntem
lxr>oeie$epae;
$asr=$n-gtasr)
pre
ev>ePre(;
$oere=$asr>as(tkntem;
ndTe
pre-pre$oeSra)
eh $oere
co ndTe;

EXCERPT OF THE ABSTRACT
SYNTAX TREE
Ti_oeMdl(
wgNd_oue
bd:Ti_oeBd(
oy wgNd_oy
0 Ti_oe
: wgNd(
0 Ti_oeTx(aa 'u>)
: wgNd_etdt: <l'
1 Ti_oe
: wgNd(
0 Ti_oeStep
: wgNd_eTm(
nm:'tm'
ae ies
)
1 Ti_oeFr
: wgNd_o(
vletre:Ti_oeEpeso_sinae
au_agt wgNd_xrsinAsgNm(
nm:'tm
ae ie'
)
sq Ti_oeEpeso_epae
e: wgNd_xrsinTmNm(
nm:'tm'
ae ies
)
bd:Ti_oe
oy wgNd(
0 Ti_oe
: wgNd(
0 Ti_oeTx(
: wgNd_et
dt:'l>
aa <i'
)
1 Ti_oe
: wgNd(
0 Ti_oeStep
: wgNd_eTm(
nm:'tm
ae ie'
)
1 Ti_oePit
: wgNd_rn(
ep:Ti_oeEpeso_itr
xr wgNd_xrsinFle(
nd:Ti_oeEpeso_itr
oe wgNd_xrsinFle(

THE ROOT NODE
The parsing process results in a root node containing
the body of the template,
or a collection of blocks,
the link to a parent template,
...

THE MAIN TOKEN TYPES
The parser collects nodes based on the token at the current
position in the token stream.
TX
ET

template text

create a T x node with the value of the
et
token

VRSAT
A_TR

variable

parse the expression that follows and
expect V R E D
A_N

B O K S A T block with a tag expect a name, which is the name of the
LC_TR
tag (i.e. for, if, etc.) and call a subparser

SUBPARSER == TOKEN
PARSER
Each token parser defines its own rules for the tokens that
should follow the tag:
{ frie i ies%
% o tm n tm }
{ icue'epaehm'wt {fo:'a' %
% nld tmlt.tl ih 'o' br} }
{ stfo br='o' 'a'%
% e o, a
fo, br }

FROM TOKENS TO NODES
The token parser returns nodes based on the tokens it finds.
Returned nodes are inserted in the Abstract Syntax Tree

A CUSTOM TOKEN PARSER
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto prewgTkn$oe)
ulc ucin as(Ti_oe tkn
{
$hs>asr>eSra(-epc(Ti_oe:BOKED;
ti-pre-gttem)>xetwgTkn:LC_N)
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()
}

}

rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()

pbi fnto gta(
ulc ucin eTg)
{
rtr 'ofrne;
eun cneec'
}

{ cneec %
% ofrne }

Teew g,{ cneec %!
hr e o % ofrne }
$epae=..
tmlt
.;
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-adoePre(e CneecTknasr);
ev>dTknasrnw ofrneoePre()
eh $n-pre$n-tknz(tmlt);
co ev>as(ev>oeie$epae)

(Better: register them using the g t o e P r e s )of
eTknasr(
your Twig extension class)

EXCERPT OF THE ABSTRACT
SYNTAX TREE
Ti_oeMdl(
wgNd_oue
bd:Ti_oeBd(
oy wgNd_oy
0 Ti_oe
: wgNd(
0 Ti_oeTx(
: wgNd_et
dt:'hr w g,'
aa Tee e o
)
1 Ti_oePit
: wgNd_rn(
ep:Ti_oeEpeso_osat
xr wgNd_xrsinCntn(
vle 'yfnCn
au: Smoyo'
)
)
2 Ti_oeTx(
: wgNd_et
dt:''
aa !
)
)
)
)

EXPRESSIONS
{ 5+ae*4}
{
g
}
$epae=..
tmlt
.;
$oueoe=$n-pre$n-tknz(tmlt);
mdlNd
ev>as(ev>oeie$epae)
eh $oueoe
co mdlNd;

Expressions are parsed by a specialized expression parser.

EXPRESSIONS
..
.
ep:Ti_oeEpeso_iayAd
xr wgNd_xrsinBnr_d(
lf:Ti_oeEpeso_osatvle 5
et wgNd_xrsinCntn(au: )
rgt Ti_oeEpeso_iayMl
ih: wgNd_xrsinBnr_u(
lf:Ti_oeEpeso_epaenm:'g'
et wgNd_xrsinTmNm(ae ae)
rgt Ti_oeEpeso_osatvle 4
ih: wgNd_xrsinCntn(au: )
)
)

ASSOCIATIVITY
Most operators are left associative,
which means that
a+b+c

is to be read as
(a+b +c
(
)
)

and not as
( +( +c)
a
b
)

PRECEDENCE
Operators have a number indicating
their precedence, so
a+b*c

will always be interpreted as
( +( *c)
a
b
)

instead of
( +b *c
a
)
)

ALSO: NODE VISITORS
Allowed to revisit the entire node tree and change anything.
E.g. auto-escaping


THE COMPILER

THE ROOT NODE
The node tree contains nodes generated by:
the parser
the expression parser
token parsers
node visitors

THE COMPILE STEP
{ frie i ies%
% o tm n tm }
{ ie|aiaie}<r
{ tmcptlz }b>
{ edo %
% nfr }
$epae=..
tmlt
.;
eh $n-cmieore$epae;
co ev>oplSuc(tmlt)

EXCERPT OF THE COMPILED
TEMPLATE

cas_TiTmlt_4dc9f0249098c82eetnsTi_epae
ls _wgepaed18d80b0e809ef47 xed wgTmlt
{
ra()
{
/ ln 1
/ ie
i (se(cnet"tm") {$ies =$otx[ies] }es {$ies =
f ist$otx[ies])
_tm_
cnet"tm";
le
_tm_
$otx[_aet]=(ra)$otx;
cnet'prn'
ary cnet
$otx[_e' =ti_nuetaesbe$ies)
cnet'sq]
wgesr_rvral(_tm_;
frah(cnet'sq]a $otx[_e" = $otx[ie"){
oec $otx[_e' s cnet"ky] > cnet"tm]
/ ln 2
/ ie
eh ""
co
;
i (se(cnet"tm]){$ie_=$otx[ie";}es {$ie_=
f ist$otx[ie")
_tm
cnet"tm]
le
_tm
eh ti_saefle(ti-ev ti_aiaiesrn_itr$hs>n,
co wgecp_itr$hs>n, wgcptlz_tigfle(ti-ev
eh "b>;
co <r"
}
$prn =$otx[_aet]
_aet
cnet'prn';
ust$otx[_e',$otx[_trtd] $otx[_e',$otx[ie'
ne(cnet'sq] cnet'ieae', cnet'ky] cnet'tm]
$otx =arymre$prn,aryitretky$otx,$prn);
cnet
ra_eg(_aet ra_nesc_e(cnet _aet)
}
}

RECURSIVE COMPILING
The compiler just calls the c m i e )
opl(
method of the root node.
Which calls the c m i e )of child
opl(
nodes, etc, etc.

CUSTOM TOKEN PARSER
REVISITED
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto preTi_oe $oe)
ulc ucin as(wgTkn tkn
{
..
.
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()

}

}

rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()

COMPILING A PRINT NODE
casTi_oePit
ls wgNd_rn
{
pbi fnto cmieTi_oplr$oplr
ulc ucin opl(wgCmie cmie)
{
$oplr
cmie
-adeuIf(ti)
>dDbgno$hs
-wie'co'
>rt(eh )
-sbopl(ti-gtoe'xr)
>ucmie$hs>eNd(ep')
-rw""
>a(;n)
;
}
}

The result can be found in the compiled template:
ra()
{
/ ln 1
/ ie
..
.
eh "yfnCn;
co Smoyo"
..
.
}

SOME REFLECTIONS
1. You can put any PHP code you want inside a template
2. You can do (heavy) calculations at compile time (just once)
You only have to create your own node type and
implement its compile method.

TESTING AND DEBUGGING
Writing parsers and nodes can be quite difficult, so
1. read the compiled templates in your cache directory and
2. write unit tests for your custom parser and node type
(extend from w g T s _ o e e t a e
Ti_etNdTsCs)

THANK YOU
AND GOOD BYE

joind.in/10368
nobacksoffice.nl
php-and-symfony.matthiasnoback.nl
@matthiasnoback

REFERENCES
Armin's blog
Jinja documentation
Templating engines in PHP
Templating engines in PHP - Follow-up
Article about node visitors

IMAGES
http://www.stockfreeimages.com/
http://twig.sensiolabs.org/

Diving deep into twig

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Diving deep into twig (20)

More from Matthias Noback (20)

Recently uploaded (20)

Diving deep into twig