1/10/12

Understanding Memor

Understanding Memor
O c e f ha d a e. If e ea ch. hi di c i i he AICT Li C ha e a c e e i ab @ a be a.ca. e , hich he a e ia 64-bi GNU/Li AMD O e e e ed, ea e e d a e

Contents
I d ci P ga a d P ce e S age C a a d Sc e P g a Si e Me Ma Ca S ac Page Tab e Lib a ie Me Li i Me A ca i I e e a i De ai Refe e ce

Introduction
U de Li , a ga i a virtual memor e i e . If a C ga e i he a e f a i e ( e e ece a i ac ice), he e i be a i a e add e . I F a , a h gh i e ae a a da d fea e, i a e add e i g i i ici i e e a iab e efe e ce a d b i e ca . Of c e, a g a ' c de a d da a ac a e ide i real ph sical memor . The ef e, each i a e add e i a ed b he e a i g e a h ica e add e i a c e a he page table ( ee Fig e 1). S , f e a e, e ie e da e he a e f a a iab e, a ga fi ac i e he a iab e' ea e add e b i g he a cia ed i a e add e i he age ab e. F a e , hi e i ha d ed a ae b Li . O i i ed c de i he Li e e a d ecia i ed ci c i i he CPU c bi e a e hi a ea ab efficie e a i . Ne e he e , a de h d i a a .

Fig. 1 Ma

i g i

ea

gh he age ab e.

I a ge e a e i- e c i ge i e ch a Li , a ga ha e he fi i e h ica e ha i a ai ab e. I he ab e ce f i a e , each ga d ha e be a a e f he ac i i ie f i eighb .F e a e, c ide i de e de ga i h di ec e acce i g c ai he a e a ea f f ee e a he a e i e. T a id a c f ic , he ga d ha e a ici a e i a ch i a i che e, eadi g e c e c de. I ead, i h i a e ,a -e e e a age e f c i a e de ega ed he e a i g e . Acc di g , he Li e e ai ai a i a e age ab e f each g a gi i g i he i i ha i i a e he c e . Whe c c e ga efe e ce he a e i a e add e , he e e e e ha each e e e a diffe e ea e add e a h i Fig e 2.
.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html 1/15

1/10/12

Understanding Memor

Fig. 2 C

ga

. ac i a e ha he e a i g e e f hi a e i h ha i g be a a e f he g a cia ed i h i a e i c e , he i ci e de c ibed a he

I a a he hi f i a e a a ab i a e he e ha d a e. P g a i de ai fa ecific e i e e ai . We i bec e ac ai ed i h he c ec i ha f . A h gh f c he c i ge i e .
g

ce a d e i i be he AICT Li

P og am and P oce
A c

e
e . O he c e e a iab e ), a d i c de i c e . T ge he , he

g a ' age ab e i ec e f i e ec i c i g di ec , i f e fi e , e i e (i fe i i e ha i a a p oce a k.

Of e , " g a " a d " ce " a e ed i e cha geab . The e i a diffe e ce h e e . The e ga i a a cia ed i h ce c de i e i a a ic a ga i g a g age. We ea faF a ga aC ga f e a e. I a efe he c i ed ce c de e ec ab e fi e di . A ce , he he ha d, i he e a i g e ' c ce f a i g ga . The e e a ig each ce a i e ide ifica i be , a p oce ID ( id), a d e hi i de he a i da a c e ha ei f a i ab he ce . A g a ca e ie e i id a d a he ce - e a ed a ib e h gh a ga i g i e face. Thi i e face i a da d i Ca di ed a a e e i i F a . F e a e, g i ( e e d) he e ID ( id) f he ca i g ce . E ec i g a g a i e ac i e a he he ' c a d i ac a ce . The e ce i i e a a ed, f ed (af e he f r ( ok) e f he he ce . I hi a , a ce hie a ch i e ab i hed, i h he he e ce a chi d. Na a , he chi d i he i a f he a ib e f i ae i g di ec a de i e . Sig ifica , e e ce i i a e a ae chi d. M e hi a e .
g

c ea e a e ca ha i i ed), a a e a d he , i e he c e a ed d f

S o age Cla

and Scope
2/15

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

1/10/12

Understanding Memor

Programs comprise executable statements and data declarations. Each datum has a property known as storage class that reflects its lifespan during program execution. A related property called scope characterizes the extent of a datum's visibility. Storage class and scope are assumed from the location of a datum's declaration, and determine its placement within virtual memory. In C, data declared outside the body of any function have global scope and static (permanent) duration. Although initial values may be assigned, global data are usually uninitialized. Meanwhile, data declared inside the body of a function including m i ( have local scope and automatic (temporary) duration. an) A local datum can be made permanent by qualifying its declaration with the s a i keyword so that it ttc retains its value between function invocations. In Fortran, all data are local in scope unless they are declared in a module (Fortran 90/95) or appear in a named common block. Furthermore, by assigning the S V attribute to module variables and by AE referencing common blocks in a S V statement (or carefully locating common block specifications), AE they effectively acquire global scope and static duration. In the case of the Portland Group (PGI) Fortran compilers pgf77 and pgf95, local explicit-shaped arrays (those for which fixed bounds are explicitly specified for each dimension) have the static, rather than automatic, storage class. However, the contents of such arrays are invalidated between subroutine invocations, unless they are declared with the S V attribute or they appear in a S V statement. AE AE Be a are that the treatment of e plicit-shaped arra s differs among compilers. In contrast to pgf95, the IBM compiler lf95, for e ample, considers them automatic. If necessar , these semantics can be altered ith compiler options.
goto top

Program Si e
Compilers translate a program's executable statements into CPU instructions, and declarations of static data are translated into machine-specific data specifications. To create an executable file, the system linker aggregates the instructions and the data into distinct segments. All of the instructions go into one segment traditionally called te t. Unfortunately, that name conveys the impression that the segment contains source code, which it does not. Meanwhile, the data are arranged in two segments. One is called data, for the initialized static data and literal constants, and the other, bss, for the uninitialized static data. Bss once stood for "block started from symbol," which was a mainframe assembly language instruction, but the term carries no meaning today. Consider the following simple C program, and the equivalent Fortran 90/95 version, in which the major data component is an uninitialized static array of 200 million bytes. /* * *sml. ipec * / #nld <ti.> icue sdoh #nld <tlbh icue sdi.> #eieNIE20000 dfn SZ 0000 ca xNIE; hr [SZ] it n mi (od an vi) { fr(n i0 iNIE i+ o it =; <SZ; +) xi ='' [] x; pit (dn\"; rnf "oen) ei (XTSCES; xt EI_UCS)

$pc -9 - sml sml. gc cx o ipe ipec

1/10/12

Understanding Memor

$s esml i ipe tx et dt aa 12 26 50 6

bs s 20002 0003

dc e 20088 0011

hx e bb9a ec1

flnm ieae sml ipe

$l - sml s l ipe -wrxrx-rx 1eubruf 71 Nv1 1:2sml sma oa 14 o 5 41 ipe ! !sml.9 ipef0 ! mdl goas oue lbl ipii nn mlct oe itgr prmtr: NA =20000 nee, aaee : MX 0000 caatr1,sv : xNA) hrce() ae : (MX edmdl goas n oue lbl pormsml rga ipe uegoas s lbl ipii nn mlct oe itgr: i nee : d i=1 NA o , MX xi ='' () x edd n o pit,"oe rn* dn" so tp edpormsml n rga ipe $pf5- sml sml.9 g9 o ipe ipef0 $s esml i ipe tx et dt aa bs s 777 72 187220072 087 0035

dc e 21721 0105

hx e bddb f94

flnm ieae sml ipe

$l - sml s l ipe -wrxrx-rx 1eubruf 1064Nv1 1:2sml sma oa 219 o 5 41 ipe $fl sml ie ipe sml:EF6-i LBeeual,ADx66,.. ipe L 4bt S xctbe M 8-4 . Compiling (and implici l linking) a ill a ed abo e p od ce an e ec able p og am file in he ELF (E ec able and Linking Fo ma ) fo ma . R nning he i e command e ac he magni de of he e , da a, and b egmen f om he ELF file. In bo h ca e , he b egmen i indeed 200 million (pl ome admini a i e o e head). The p og am' con ib ion o he da a egmen incl de onl o ing li e al and one n me ic con an , hich doe no acco n fo he i e epo ed. Appa en l , he compile a e e pon ible fo he di c epanc . F he mo e, beca e he ELF file con ain all of he p og am in c ion and all of he ini iali ed da a, he m of he e and da a egmen ho ld app oach, b ne e e ceed, he i e of he file on di k. Re e ing pace fo he b egmen in he file i nnece a ince he e a e no al e o o e. Thi i confi med b he e ample . Be aware that the data and bss segments are frequentl referred to collectivel as just data, occassionall leading to confusion.
go o op

Memor Map
When he ELF file i e ec ed, he e and he o da a egmen a e loaded in o epa a e a ea of i al memo . B con en ion, e occ pie he lo e add e e , i h da a abo e. App op ia e
.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html 4/15

1/10/12

Understanding Memor

.A

.I , memor map

F 3.

Fig. 3 P V 512GB AMD64 A .A .O 512GB

. L . .T

si e ( + + ) , ml c) al ( C ALCTBE LOAAL F 77 heap segment. S F 4.

F .T

90/95. A

Fig. 4 M D A T C , , , ( + ),

. . , . . .

Call S ack
P .I .I automatic segment , stack. S F 5.
5/15

( ,

.S

stack

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

1/10/12

Understanding Memor

F g. 5 M T C MI AN .T F .U

. , ' mi( an) " " F 6.

1. OS 2. 3. 1 4. 2 5. 1 6. 3 7. 1 8. 9. 4 10. F g. 6 T B ,

1 2 1 3 1 4 ( ) OS . " " ag rc , , . I C, , F . mi( an) MI AN .T .T , 1, 1 , 1. ack f ame. S .T

ag r

IRC AG

GTR EAG A " T " , '

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

6/15

1/10/12

Understanding Memor

Fig. 7 E ol ion of he

ack egmen co e ponding o he call g aph ho n in Fig e 6.

B con en ion, he c en l ac i e b o ine can onl efe ence he a g men i a pa ed and i o n local a oma ic and a ic a iable (pl an globall acce ible da a). Fo e ample, hile f nc2 i e ec ing, i can no acce f nc1' local a iable , nle of co e f nc1 pa e efe ence o i local a iable in he a g men li o f nc2.
go o op

Page Tab e
Fig e 8 ill a e he ela ion hip be een he memo map, page able, and eal memo . The page able e pand o con ac , mapping mo e o le eal memo , a bo h he ack and heap egmen change i e.
.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html 7/15

1/10/12

Understanding Memor

Fig. 8 Me

a , age ab e, a d ea

S e each age ab e e c i e e 64-bi be e e e i g a i a e add e a d a he 64-bi be e e e i g a ea e add e , f a a f 16 b e e e .T a each b e f a 200MB ce , f i a ce, d e i e a 3200MB age ab e. Ob i i ac ica . I ead f a i g i di id a b e , he age ab e a a ge ch f i a e ca ed page (he ce he a e). The c e di g i c e e f ea e i ca ed a page f ame. Page i e i a chi ec e- ecific a d f e c O e ce i he AICT Li c e d be 800KB. fig ab e. I i a a e e f e 4KB. Acc di g , he age ab e f . The AMD a 200MB ce

O e 128 i i age a he 512GB i a e add e a ge. The a c ec i e i h a i a age be (VPN). Each age ab e e a a ce e a a age f a e i ea e , ha i , f a VPN a (PFN). The VPN i ca c a ed f he i a e add e b di idi g i b he add e he igh ). The ecific b e i ca ed a a ff e f he a
g

e be ed i a age f he age f a e be he age i e (bi - hif i g f he age.

Lib a ie
Li i g a a ic ib a i a g a fi e. A a e , e . See Fig e 9. g a i eg a e he ib a ' e a d da a eg e i he ELF ga i ed i h he a e a ic ib a a b h a he ib a i ea

Fig. 9 T

ga

ed i h he a e

a ic ib a .
8/15

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

1/10/12

Understanding Memor

S T

ELF d namic shared ELF F 10.

. ( a .T , .T . o( s ,

) ) .F

.T ,

ELF . .S ,

.T

Fig. 10 T W ' , .W

. , .T , , .H ' ). F 10 , , .

, .

'

, COW (

Memor Limits
I P , , A , "S T , uii - 240 lmt s 08 ( mnts) a ch hard limit. O AICT .I ( , " hl uii) ep lmt 20MB. S . .F lmtsakie240 ii tcs 08 " ," , , , P soft limit , , , , .I .W , , .I , AICT , 10MB. T ,

. , ,

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

9/15

1/10/12

Understanding Memor

. Be aware that the soft and hard limits on shell resources var greatl among s stems. M AICT 10GB RAM .O , , swap space ( , , ). ( F 11. AICT 1000MB .A , 9800MB ( 1GB ). T

RAM ), .T T ) , , SWAP (

F g. 11 S T .M S SWAP , , , A , .W "S ml c) al ( ALCT LOAE F ( " NL UL + .T F . AICT , + ) ,C 90/95 " ," .T , .M " " SWAP.

. .M ,

, ( .W , . ). T

SA TT F 12.

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

10/15

1/10/12

Understanding Memor

Fig. 12 Stack size and virtual memory limits imposed on a batch job process. Because the page table represents the virtual memory footprint of a process, the virtual memory soft limit is depicted as a bounding box surrounding the page table. Note that under certain conditions, the virtual memor limit can be violated b stack-based data even if the stack does not e ceed the stack si e limit. A job's pvmem specification translates as the amount of SWAP that is dedicated to the process by the batch job system. The process may actually use less. Nevertheless, when considering a node for scheduling another job, the batch system accounts for the total SWAP dedicated to not used by all the other jobs currently running on the node. This could lead to waste through unused resources unless the pvmem estimates are accurate.
goto top

Memor Allocation
Thus far, our graphical representations of the page table have implied that virtual memory pages are always mapped to real page frames. However, because some pages may never be used, it is more efficient for the operating system to defer the mapping until the page is actually referenced, a technique known as demand paging. For example, a common practice in Fortran 77 is to compile a program with the largest anticipated static arrays and use the same executable to perform calculations with varied array extents. In this case, a sufficient number of virtual pages are always reserved in the page table to completely accommodate the large arrays. However, only those pages containing array elements actually referenced by the program are allocated page frames in real memory. Those pages that are never referenced are not allocated page frames. The sum of all the allocated page frames is called the resident set si e (RSS). As shown in Figure 13, RSS will be less than or equal to the process's virtual memory footprint.

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

11/15

1/10/12

Understanding Memor

Fig. 13 A W

12 , page fault

. .L O TLB

' .I , 512 CPU' translation lookaside buffer (TLB). R ' . F RAM. H .T ' F ( .A .B .E .A , , ), .I , .T s apped in , , , s apped out , , , RAM

, ,

"

"

, paged out.

, , .T SWAP , .

Implemen a ion De ail

/* * *mpc a. * / #nld <ti.> icue sdoh #nld <tlbh icue sdi.> #nld < stpsh icue s / e.>
.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html 12/15

1/10/12

Understanding Memor

#nld <nsdh icue uit.> #eieNIE dfn SZ (*1<0) 5(<2) #eieSEP dfn LET 1 0 ln g[SZ] og xNIE; it n mi (n ag,ca *rv] an it rc hr ag[) { ca cNIE; hr [SZ] it *x=mlo (SZ*iefit) n p alc NIEszo(n); fr(n i0 iNIE i+ { o it =; <SZ; +) g[]=(ogi xi ln); p[]=i xi ; ci [] ='' c; pit (adeso g[]=%1pn,&x0) rnf "drs f x0 02\" g[]; pit (adeso p[]=%1pn,&x0) rnf "drs f x0 02\" p[]; pit (adeso rnf "drs f c0 =%1pn,&[]; [] 02\" c0) pit (mmr mpfl:/rc%/asn,gti() rnf "eoy a ie po/dmp\" epd); pit (sepn %..,SEP) rnf "leig d." LET; fls (UL; fuh NL) sep(LET; le SEP) fe (x; re p) pit (\dn\"; rnf "noen) ei (XTSCES; xt EI_UCS)

A label has been added to each line of the maps file output in the terminal displa below to make subsequent references more convenient. $pc -9 - mpmpc gc cx o a a. $l - mp s l a -wrxrx-rx 1eubruf 87 Dc 10:4mp sma oa 16 e 92 a $sz mp ie a tx et dt aa bs s dc e hx e flnm ieae 16 78 74 2 4937 1402 4956 1454 209c mp 80d a $.mp /a adeso g[]=00050c drs f x0 x000b0 adeso p[]=0295a1 drs f x0 xa5700 adeso drs f c0 =07baf6 [] xfff40 mmr mpfl:/rc214mp eoy a ie po/31/as sepn 1..Z leig 0.^ [] Sopd 2+ tpe .mp /a $ct/rc214mp a po/31/as ()0400-0000rx ../cac/sma/a a 00000410 -p . srtheubrmp ()0500-0000r- ../cac/sma/a b 00000510 wp . srtheubrmp ()0510-2000rx .. c 00000d10 wp . ()29560-a5500r- .. d a550029570 wp . ()29580-a6700r- .. e a5700299c0 wp . ()30300-3c100rx ../i6/d234s f 3c00030350 -p . lb4l-...o ()30440-3c100r- ../i6/d234s g 3c10030460 wp . lb4l-...o ()30500-3c200rx ../i6/l/ic234s h 3c000306a0 -p . lb4tslb-...o ()306a0-3c200-- ../i6/l/ic234s i 3c20030790 -p . lb4tslb-...o ()30790-3c200r- ../i6/l/ic234s j 3c200307c0 -p . lb4tslb-...o ()307c0-3c200r- ../i6/l/ic234s k 3c200307f0 wp . lb4tslb-...o ()307f0-3c300r- .. l 3c20030730 wp . ()30800-3c800rx ../i6/l/im234s m 3c00030850 -p . lb4tslb-...o ()30850-3c800-- ../i6/l/im234s n 3c80030940 -p . lb4tslb-...o
.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html 13/15

1/10/12

Understanding Memor

()30940-3c800r- ../i6/l/im234s o 3c80030960 p . lb4tslb-...o ()7bae0-f0000r p.. p fff007c000 . ()fffff600-fffff000-- .. q fffff000fffffe00 -p . $f g .mp /a dn oe M 10, 100016 , U , , ELF 0 400000 (4MB) . I R ( ) . (L g , ( ), N (20MB+16KB), p . R () ( ) , . $ldmp d a lb.o6= /i6/l/ics. ( 00030500) ics. > lb4tslb.o6 0 0003c000 lb.o6= /i6/l/ims. ( 00030800) ims. > lb4tslb.o6 0 0003c000 /i6/dln -8-4s. ( 00030300) lb4l-iu 66.o2 0 0003c000 P .C , A , , . F , 0 7 c .H , . " , " .R ( ) (8MB) , ( ) 0 502000 (5MB+8KB). O .A , ( ) ( ) ( ) ' ( ), ' , , C , P . , , , . () , .M , pit( rnf) , , .I .A C( . ) ( . ) .T , " " smt) ha( , 0 500 0 2800000 (40MB). ,( ) ( ). T .R ( ), ( ), 0 ( ) .I r p ma( mp) .W 0 500000 (5MB). H , .) J ( ) 0 1404000 , , .B .P " ." 0 401000 (4MB+4KB), .T 1768 .A .T .F , 0 0 1000. N 9 A (10) ( F (15). A . " ," " ," ( ) .N .H , ) , , 4KB 16

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

14/15

1/10/12

Understanding Memor

Refe ence
1. Understanding the Linu Kernel, by Daniel P. Bovet and Marco Cesati, 2001 O'Reilly 2. Linkers and Loaders, by John R. Levine, 2000 Morgan Kaufmann 3. S stem V Application Binar Interface: AMD64 Architecture Processor Supplement, Draft Version 0.96, by Michael Matz, Jan Hubi ka, Andreas Jaeger, and Mark Mitchell
goto top $Id: mem.html 299 2010-05-12 16:16:29Z esumbar $

2008 University of Alberta

.ualberta.ca/CNS/RESEARCH/Linu Clusters/mem.html

15/15