DTrace by Example:

Solving a Real-World Problem

Paul van den Bogaard
January 2007
Sun Microsystems, Inc.
Copyright 2007 Sun Microsystems, Inc., !"#0 $et%or& Circle, Santa Clara, Cali'ornia (#0#!, ).S.*. *ll rights
reserved.
).S. +overnment ,ights - Commercial so't%are. +overnment users are su./ect to the Sun Microsystems, Inc.
standard license agreement and applica.le provisions o' the 0*, and its supplements. )se is su./ect to license terms.
1his distri.ution may include materials developed .y third parties.
Parts o' the product may .e derived 'rom Ber&eley BS2 systems, licensed 'rom the )niversity o' Cali'ornia. )$I3 is
a registered trademar& in the ).S. and in other countries, e4clusively licensed through 356pen Company, 7td.
356pen is a registered trademar& o' 356pen Company, 7td.
Sun, Sun Microsystems, the Sun logo, Solaris, and 6penSolaris are trademar&s or registered trademar&s o' Sun
Microsystems, Inc. in the ).S. and other countries.
1his product is covered and controlled .y ).S. 84port Control la%s and may .e su./ect to the e4port or import la%s
in other countries. $uclear, missile, chemical .iological %eapons or nuclear maritime end uses or end users, %hether
direct or indirect, are strictly prohi.ited. 84port or ree4port to countries su./ect to ).S. em.argo or to entities
identi'ied on ).S. e4port e4clusion lists, including, .ut not limited to, the denied persons and specially designated
nationals lists is strictly prohi.ited.
26C)M8$1*1I6$ IS P,69I282 :*S IS: *$2 *77 83P,8SS 6, IMP7I82 C6$2I1I6$S,
,8P,8S8$1*1I6$S *$2 ;*,,*$1I8S, I$C7)2I$+ *$< IMP7I82 ;*,,*$1< 60 M8,C=*$1*BI7I1<,
0I1$8SS 06, * P*,1IC)7*, P),P6S8 6, $6$-I$0,I$+8M8$1, *,8 2ISC7*IM82, 83C8P1 16 1=8
8318$1 1=*1 S)C= 2ISC7*IM8,S *,8 =872 16 B8 78+*77< I$9*7I2.
DTrace by Example: Solving a Real-World Problem 2
Table of Contents
Introduction................................................................................................................................................
T!e "pplication Domain............................................................................................................................
" S!ort Introduction to DTrace.................................................................................................................
Example............................................................................................................................................#
T!e Initial Situation...................................................................................................................................#
Watc!ing t!e $!ildren o% t!e Proce&&........................................................................................................'
doorfs: W!at I& It(......................................................................................................................)*
+ame Service $ac!e Daemon..................................................................................................................22
Re%erence& ...............................................................................................................................................2,

DTrace by Example: Solving a Real-World Problem ,
Introdction
I ran into a &ecure &!ell -SS./ i&&ue 0!en preparing &ome demon&tration&. "t login1 more t!an )2
&econd& pa&&ed be%ore I 0a& prompted %or a pa&&0ord. Since t!e demon&tration 0ould include 0atc!ing
&tati&tic& 0!ile &ome program& 0ere running1 I needed multiple &!ell&. 3ecau&e c!anging t!e&e &etting&
over and over mandated %re4uent1 repeated login&1 t!e&e mandated pau&e& 0ere more t!an a mere
nui&ance.
In t!i& article you 0ill learn !o0 to u&e Dynamic Tracing -DTrace/ in t!e Solari&
T5
6perating Sy&tem
to re&olve t!e i&&ue o% t!e&e pau&e&. T!e article de&criber& t!e &tep& ta7en and t!e rea&oning be!ind
t!em. Eac! &tep re&ult& in a DTrace &cript and eac! &cript give& more in&ig!t into t!e problem. Eac!
ne0 in&ig!t rai&e& more 4ue&tion&1 0!ile al&o narro0ing t!e %ocu& o% attention. T!e re&ult 0ill be
gaining a better under&tanding o% t!e &ituation. Toget!er 0it! &ome application-&peci%ic domain
7no0ledge t!e %inal 4ue&tion& can be an&0ered and t!e problem re&olved.
I a&&ume t!e reader !a& no &peci%ic bac7ground in DTrace and it& D programming language1 nor o% t!e
application domain. I do a&&ume t!e reader can read a $-li7e programming language &ource. 8or a
complete overvie0 o% DTrace1 it& language1 and &tandard provider&1 plea&e c!ec7 t!e Solaris
1M
2ynamic 1racing +uide
)
. 8or a complete de&cription o% t!e dtrace command1 &ee t!e manual page
dtrace-)5/. T!e 6penSolari&
T5
DTrace community
2
i& a valuable re&ource %or all 7ind& o% DTrace-
related in%ormation1 example &cript&1 documentation1 and &o on.
T!e "pplication Domain
T!e SS. application i& u&ed to &a%ely log in to a remote &y&tem. During t!e connection &etup1 &en&itive
data li7e u&er name and pa&&0ord i& &ent in an encrypted %orm. 6nce t!e connection i& &et up1 all data
t!at i& &ent %rom eit!er mac!ine to t!e ot!er i& al&o encrypted.
T!i& SS. client contact& t!e sshd daemon on t!e !o&t mac!ine. T!e daemon u&e& port 22 a& t!e
de%ault port on t!e &erver to &ee t!e incoming re4ue&t. 6t!er con%iguration item& t!at are sshd-
&peci%ic can be %ound in t!e sshd_config %ile1 0!ic! i& in t!e directory /etc/ssh. 8or a complete
de&cription o% t!e sshd daemon1 t!e con%iguration %ile1 and t!e SS. client1 &ee t!e corre&ponding
manual page& %or sshd-)5/1 sshd_config-/1 and ssh-)/.
" S!ort Introdction to DTrace
DTrace i& a compre!en&ive dynamic tracing %rame0or7 %or t!e Solari& 6S. DTrace provide& a po0er%ul
in%ra&tructure to permit admini&trator&1 developer&1 and &ervice per&onnel to conci&ely an&0er arbitrary
4ue&tion& about t!e be!avior o% t!e operating &y&tem and u&er program&. DTrace con&i&t& o% multiple
element&1 0!ic! are built into t!e Solari& )2 7ernel: t!e D language1 t!e dtrace command1 t!e
provider&1 &ome DTrace-related librarie&1 and t!e DTrace %rame0or7.
T!e &cript& are 0ritten in t!e D language. T!e&e &cript& are &tarted 0it! t!e dtrace command. T!i&
command 0ill c!ec7 and compile t!e &cript into an intermediate language t!at i& interpreted by t!e
dtrace virtual mac!ine embedded in&ide t!e 7ernel. 6n a de%ault Solari& )2 in&tallation1 you need to
!ave root acce&& in order to execute DTrace &cript&. It i& advi&able to u&e t!e ne0 privilege rig!t&
management %eature o% t!e Solari& )2 6S to enable non-root u&er& to u&e DTrace.
DTrace by Example: Solving a Real-World Problem
" &cript contain& one or more probe-predicate-action triplet& called clau&e&.
DTrace probe& are point& o% in&trumentation in&ide t!e 7ernel. Related probe& are grouped into DTrace
provider&1 eac! o% 0!ic! per%orm& a particular 7ind o% in&trumentation. Speci%ying a probe in your
&cript enable& it to collect data 0!enever a probe i& %ired.
I% you enable a probe it 0ill %ire 0!en t!e corre&ponding event occur&. 8or example1 t!e
syscall::write:entry probe %ire& 0!en any proce&& on t!e &y&tem call& t!e write &y&tem call.
T!e predicate part o% a clau&e evaluate& to a boolean value o% true9%al&e and t!e probe i& only activated
0!en t!e predicate become& true. T!e predicate part t!ere%ore enable& you to &et t!e appropriate
context in 0!ic! t!e action &!ould be executed. T!e action part i& t!e actual code t!at run& 0!en t!e
probe %ire& and t!e predicate !old& -evaluate& to true/.
" probe i& %ully &peci%ied by t!e %our tuple& provider name1 module name1 function name1
and name. In a probe &peci%ication t!e&e element& are &eparated by a colon. "ny o% t!e&e can be
un&peci%ied. T!e %ormat i& provider:module:function:name. Re%erence to t!e&e can be done
by t!e u&e o% t!e prede%ined variable name&: probeprov1 probemod1 probefunc1 and
probename.
I% a tuple element i& un&peci%ied t!e probe 0ill matc! all po&&ible entrie& %or t!at tuple. :n&peci%ied
matc!e& everyt!ing1 li7e a 0ildcard.
Example
syscall::write:entry
/pid == 1/
{
printf("Fired\");
}
In t!i& &cript1 t!e probe &peci%ie& t!e entry o% all 0rite %unction& in any module -not &peci%ied/ %rom t!e
syscall provider. In ot!er 0ord&1 t!i& probe %ire& 0!en a write &y&tem call i& made. T!e predicate
/pid == 1/ place& a re&triction: only i% t!i& write call i& done by t!at proce&& 0it! proce&& ID
-PID/ o% 1.
I% proce&& 1 doe& a write call1 t!e corre&ponding action i& executed: Fired i& printed rig!t a%ter t!i&
write 0a& called1 but ;u&t be%ore it i& executed.
T!e Initial Sitation
T!e current &tatu& i& t!at it ta7e& 4uite &ome time 0!en trying to log on to t!e &y&tem t!roug! SS.. 6n
t!at &y&tem1 a daemon -sshd/ i& running in t!e bac7ground1 0aiting %or re4ue&t& to arrive on port 22.
W!en a re4ue&t come& in1 it 0ill %or7 a c!ild. T!i& c!ild t!en !andle& t!e re4ue&t and !elp& t!e client to
gain acce&& to t!e &y&tem. It 0ill c!ec7 t!e credential& and only 0!en t!ey are %ound to be o7ay 0ill it
&tart a &!ell. 6nly at t!at time 0ill t!ere be a 0or7ing environment.
T!e ob&ervation <It ta7e& 4uite &ome time1< !a& been made. T!e 4ue&tion t!at %ollo0& i&1 <W!at
con&ume& time(< During t!e li%etime o% a proce&&1 t!e proce&& eit!er u&e& $P: or 0ait& %or a re&ource.
$P: con&umption can ea&ily be determined by a utility li7e vmstat1 mpstat1 or prstat. Since t!e
proce&& o% intere&t i& 7no0n1 t!e prstat command i& t!e obviou& c!oice. W!en u&ed 0it! t!e -m
-enable micro&tate accounting/ and -L -give in%ormation %or all t!read&/ option&1 a 4uic7 impre&&ion
can be %ormed about t!e $P: con&umption o% a proce&&. 8or %urt!er in%ormation on prstat &ee
DTrace by Example: Solving a Real-World Problem #
prstat-)5/. 6ne 0ay to determine t!e proce&& ID o% t!e sshd daemon i& to log on to t!e &y&tem.
+o01 %rom t!e &!ell1 give t!e ptree command a& %ollo0&:
$ ptree $$
462 /usr/lib/ssh/sshd
2457 /usr/lib/ssh/sshd
2460 /usr/lib/ssh/sshd
2462 -bash
2478 ptree 2462
T!e ptree command &!o0& t!e !ierarc!y o% proce&&e& t!at end& in your &!ell. "& explained above1
t!i& tree &tart& 0it! t!e sshd daemon o% intere&t. T!e PID t!at need& to be u&ed i& =2.
:&ing prstat no0 &!o0&:
$ prstat -m -L -p 462
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
462 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 sshd/1
Total: 1 processes, 1 lwps, load averages: 0.05, 0.04, 0.04
T!i& 0ay o% monitoring &!o0& t!e percentage o% time &pent in t!e &tate& USR -u&er time/1 SYS -&y&tem
time/1 TRP -trap !andling/1 TFL -text page %ault !andling/1 DFL -data page %ault !andling/1 LCK -0ait
%or u&er loc7&/1 SLP -&leeping/1 and LAT -0aiting %or $P:/. T!i& al&o &!o0& t!at proce&& =2 i& )22>
in SLP &tate? t!at i&1 &leeping and 0aiting %or &ome re&ource. @etting t!e prstat command run %or a
0!ile 0ill update t!i& in%ormation every %ive &econd&. I% t!ere i& no &udden bur&t o% connection re4ue&t&
t!i& output 0ill not c!ange. In ot!er 0ord&1 $P:& are barely con&umed.
T!e only 0ay to obtain a re&ource i& t!roug! a &y&tem call. I% a proce&& i& 0aiting it mu&t be in a &y&tem
call. 3y &!o0ing 0!ic! &y&tem call& are con&uming time1 you can determine probable candidate& %or
%urt!er in&pection.
T!e original 4ue&tion1 <W!at con&ume& time(< i& no0 re%ined to1 <W!ic! &y&tem call-&/ con&ume-&/
time(< T!i& 4ue&tion can be an&0ered by proper u&e o% DTrace.
DTrace !a& t!ree variable& to depict time. I% elap&ed time i& o% intere&t1 t!e timestamp variable i& t!e
rig!t one to u&e. T!i& one !old& t!e number o% nano&econd& a%ter &ome moment in t!e pa&t. T!e value
!a& no relation 0it! 0all cloc7 time. T!ere%ore it i& be&t u&ed to mea&ure interval&.
To mea&ure t!e elap&ed time during a &y&tem call1 you need to get one time&tamp %or 0!en t!e &y&tem
call begin& and a &econd time&tamp %or 0!en t!e &y&tem call return&. .ere i& a code &nippet:
syscall:::entry
{
ts = timestamp;
}
W!en a &y&tem call i& entered -syscall:::entry/ a time&tamp i& ta7en and &tored in t!e variable
called ts. To get an interval1 t!e &econd clau&e i& needed:
syscall:::return
{
ela = timestamp - ts;
}
I% a &y&tem call return& -syscall:::return/1 t!e elap&ed time i& calculated by ta7ing a ne0
time&tamp and &ubtracting t!e value in ts %rom it. T!e re&ult i& &tored in anot!er variable called ela.
DTrace by Example: Solving a Real-World Problem =
.o0 are t!e&e t0o clau&e& related1 &ince t!e PID -proce&& identi%ier/1 TID -t!read identi%ier/1 and
&y&tem call in%ormation are not &peci%ied(
T!e t0o code &nippet& above de%ine t0o variable&. I% not!ing el&e i& &peci%ied1 t!e&e t0o variable&
!ave global &cope -all proce&&e&1 all t!read&1 every probe in t!i& &cript/. Surely t!at i& not 0!at i&
needed. 8ir&t1 to en&ure t!at every t!read running on t!e &y&tem !a& it& o0n variable t!e self->
con&truct mu&t be u&ed.
Second1 t!e probe syscall:::entry %ire& %or every &y&tem call. T!i& mean& all &y&tem call& done
by every proce&& running on t!i& &y&tem are trac7ed. 6ur intere&t i& only %or t!o&e &y&tem call& made by
t!at one sshd proce&&.
3y including a predicate t!at depict& t!i&1 t!e action ts = timestamp; 0ill only be executed %or
&y&tem call& &tarted by t!at particular sshd proce&&. T!e predicate %or t!i& i& /pid == $target/.
$target i& a prede%ined variable. T!i& variable !a& t!at value &peci%ied on t!e command line by t!e
-p option. 8or example1 dtrace -p 12345 myscript.d en&ure& t!at t!e variable $target
0ill !ave t!e value 12345 0!en myscript.d i& executed and t!ere i& a proce&& 0it! a pid o%
12345. It 0ill al&o !old t!e pid o% t!e proce&& &tarted by dtrace i% dtrace 0a& &tarted 0it! t!e -c
option.
T!e complete code %or t!i& clau&e become&:
syscall:::entry
/pid == $target/
{
self->ts = timestamp;
}
DTrace !a& &ome prede%ined variable& li7e pid and tid. tid i& t!e t!read identi%ier o% t!e t!read t!at
triggered t!i& probe. pid i& t!e proce&& identi%ier o% t!e proce&& to 0!ic! t!i& t!read belong&. T!i&
mean& t!at %or eac! probe t!at i& triggered1 you 7no0 0!at triggered it. T!e con&truct /pid ==
$target/ i& a predicate t!at i& true only i% t!e probe %ired due to your ($target) proce&& o%
intere&t.
6nce t!e &y&tem call return& t!e probe1 syscall:::return %ire&. I% t!i& i& due to your proce&& o%
intere&t1 t!en dtrace need& to execute t!e corre&ponding action. .o0ever it can !appen t!at a &y&tem
call i& in progre&& 0!en you &tart your DTrace &cript. In t!at ca&e1 you did not &ee t!e
syscall:::entry probe %ire. T!ere%ore1 you do not !ave a valid value in t!e self->ts variable.
To circumvent t!i& problem t!e appropriate predicate &!ould become /self->ts/. 6nly i% t!ere i& a
value in t!i& variable 0ill 0e be able to determine elap&ed time. T!e self->ts belong& A and i&
vi&ible A to a &ingle t!read. We are intere&ted in t!at t!read. T!ere%ore1 t!e part pid == $target i&
not needed any more. T!e complete code %or t!i& %unctionality i& no0:
syscall:::entry
/pid == $target/
{
self->ts = timestamp;
}
syscall:::return
/self->ts/
{
ela = timestamp - self->ts;
}
DTrace by Example: Solving a Real-World Problem B
T!e original idea 0a& to mea&ure t!e total elap&ed time %or all &y&tem call&. T!e name o% t!e &y&tem call
i& t!e 7ey u&ed to index an array. T!e array& provided by DTrace are a&&ociative array&. "& in $1 t!e
&4uare brac7et& are u&ed a& t!e notation %or array&. T!e %ollo0ing example de%ine& and u&e& t!e array
total:
dtrace -n 'syscall:::return { total[probename] = 1; }'
" &pecial 7ind o% DTrace array i& t!e aggregation. "n aggregation i& t!e only type t!at can be u&ed 0it!
aggregation %unction& -count1 min1 max1 and &o on/. It i& t!i& data type t!at mu&t be u&ed to
determine total elap&ed time by &y&tem call. "dding t!e line @tot[probefunc] = sum(ela); to
t!e syscall:::return probe de%ine& and enable& u&e o% an aggregation t!at !old& t!e total amount
o% elap&ed time per &y&tem call. To under&tand t!i& con&truct1 note t!at t!i& 0ould loo7 li7e
tot[probefunc] = tot[probefunc] + ela; i% done in $.
syscall:::return
/self->ts/
{
ela = timestamp - self->ts;
@tot[probefunc] = sum(ela);
}
I% t!e DTrace &cript %ini&!ed eit!er becau&e it did an exit() call or becau&e it 0a& interrupted1 t!e
END probe %ire&. It i& t!i& probe t!at mu&t be u&ed to print t!e content o% aggregated data. I% no explicit
printa call& are done in t!e END clau&e1 or i% t!ere i& no explicit END clau&e1 t!en dtrace 0ill
print t!e content o% every aggregation.
timestamp return& a value mea&ured in nano&econd&. T!e %inal code 0ill be u&ed to %ind elap&ed time
in t!e &econd& range. To ma7e t!e calculated number& more readable1 t!e normalize %unction i&
u&ed. T!e normalize(@tot, 1000); code line en&ure& all t!e value& available in t!i& a&&ociative
array are divided by )222. To dump t!e&e value& to &tandard output t!e printa %unction i& provided.
T!i& 0ill print t!e aggregation 7ey and t!e corre&ponding value according to t!e %ormat &peci%ication i%
provided. In printa("%12s %@12u\n", @tot); t!e %ormat &peci%ier i& %12s %@12u\n1
0!ic! mean& %ir&t print t!e 7ey u&ing a minimum o% )2 c!aracter&1 and t!en print t!e value %or t!i& 7ey
a& an un&igned integer u&ing a minimum o% )2 c!aracter&.
T!e complete DTrace &cript i&:
syscall:::entry
/pid == $target/
{
self->ts = timestamp;
}
syscall:::return
/self->ts/
{
ela = timestamp - self->ts;
@tot[probefunc] = sum(ela);
}
END
{
normalize(@tot, 1000);
printa("%12s %@12u\n", @tot);
}
DTrace by Example: Solving a Real-World Problem *
Executing t!i& &cript produced t!e %ollo0ing output:
./sshd1.d -p 8143
sigaction 11
lwp_sigmask 17
gtime 25
setcontext 31
fcntl 39
waitsys 42
pipe 49
ioctl 59
getpid 61
open64 169
accept 170
close 190
fstat 209
putmsg 215
doorfs 234
open 708
fork1 9445
pollsys 12957382
8rom t!i& output it &eem& obviou& t!at pollsys i& t!e culprit1 0it! a total accumulated time o% )2.'#B
&econd&.
.o0ever1 !old onC Recall 0!at pollsys doe&( $ould t!i& be normal be!avior( Indeed1 t!e natural
be!avior o% t!e sshd proce&& i& to 0ait %or a re4ue&t on port 22. It i& normal t!at &o muc! time i& &pent
in t!i& call. Waiting %or a re4ue&t on a port i& normally implemented by u&ing t!e select libc
routine1 0!ic! call& t!e poll &y&tem call. I% t!i& 0a& not t!e ca&e1 t!at 0ould be abnormal be!avior.
So time &pent in pollsys 0ill not explain t!e )2 &econd&. T!i& &cript ran %or approximately ),
&econd&1 o% 0!ic! pollsys explain& )2.'#B. Since loo7ing at t!i& proce&& cannot explain t!e time
lo&t1 anot!er proce&& mu&t &!o0 t!i& be!avior. 6bviou&ly t!i& i& one o% t!e -grand/c!ildren o% t!e
proce&& t!at 0a& ;u&t di&cu&&ed. W!ic! c!ild i& it t!at &!ould get t!e attention( .o0 do 0e ma7e t!i&
determination( T!i& call& %or a ne0 &cript.
Watc!ing t!e C!ildren of t!e Process
In order to under&tand t!i& !ierarc!y o% sshd t!e proc provider mu&t be u&ed. T!e create probe
provided by t!i& provider %ire& once a proce&& i& created. T!e initial &cript can be a& %ollo0&:
#!/usr/sbin/dtrace -s
#pragma D option quiet
proc:::create
/pid == $target/
{
}
T!i& probe %ire& i% a proce&& i& created. T!e PID cau&ing t!i& can be t!e proce&& o% intere&t a& &peci%ied
0it! t!e -p option. .o0ever t!i& doe& not cover t!e creation o% proce&&e& by c!ildren. T!e&e c!ildren
!ave a di%%erent PID t!at cannot be 7no0n up%rontC
T!e creation o% c!ildren need& to be care%ully monitored in order to under&tand t!e !ierarc!y t!at &tart&
%rom t!at initial sshd proce&&. T!e&e a&&ociation& are &tored in an a&&ociate array.
DTrace by Example: Solving a Real-World Problem '
T!e create probe !a& one parameter o% type psinfo_t -&ee $!apter 2# on proc provider in t!e
Solaris 2ynamic 1racing +uide/. 6ne element o% t!i& &tructure i& pr_ppid t!at depict& t!e parent
pid. "not!er i& t!e pr_pid: t!e proce&& pid o% t!e c!ild. :&ing an a&&ociative array called
forked1 t!e %ollo0ing code can be u&ed to combine t!e in%ormation:
/* If "our" sshd process is creating a child */
proc:::create
/pid == $target/
{
/* remember the time the child is created */
forked[args[0]->pr_pid ] = timestamp;
}
/* If we know the triggering process as a child, we now know one of its
children.
*/
proc:::create
/forked[ pid ]/
{
forked[args[0]->pr_pid ] = timestamp;
}
T!i& &!o0& t!at t!e probe &peci%ication and t!e action are bot! e4ual. T!ere%ore t!e&e t0o clau&e& can
be combined into one clau&e. 6n a general :+ID
E
&y&tem t!ere 0ill be many c!ildren created. Doing a
loo7up in an a&&ociative array %or eac! &uc! event can be co&tly. :nder&tanding SS. -and loo7ing at
t!e ptree output above/ it i& clear t!at all t!e proce&&e& o% intere&t !ave t!e name sshd. T!i& can be
u&ed to reduce t!e proce&&ing re4uired1 by ma7ing t!e predicate a little more complex:
proc:::create
/execname == "sshd" && forked[pid]/
{
...
}
+o0 letF& combine t!e&e t0o clau&e&:
proc:::create
/pid == $target || (execname == "sshd" && forked[pid])/
{
forked[args[0]->pr_pid ] = timestamp;
printf("%5d created %5d\n", pid, args[0]->pr_pid);
}
T!i& clau&e 0ill catc! all %or7& o% sshd -grand/c!ildren t!at are called sshd. T!e array forked 0ill
contain t!e &tart time -&ince &ome arbitrary moment in t!e pa&t/ o% t!at c!ild. 6ne more piece o% data i&
needed to get in%ormation out o% t!i&: t!e moment in time t!at t!e&e c!ildren die? or1 in ot!er 0ord&1 t!e
in%ormation about !o0 long a proce&& lived.
proc:::exit
/forked[pid]/
{
printf("%d lived for %10d\n", pid, (timestamp - forked[pid]));
forked[pid] = 0;
}
DTrace by Example: Solving a Real-World Problem )2
Executing t!i& &cript in one 0indo01 &tarting an SS. client to log on1 and executing a ptree in
anot!er 0indo01 command1 and exiting give& t!e %ollo0ing re&ult&:
# dtrace -qp 6767 -s t.d
6767 created 9526
9526 created 9527
9527 lived for 17996197 nsec
9526 created 9529
9529 lived for 15227079 nsec
9526 created 9531
9531 lived for 14698576 nsec
9526 created 9533
9533 created 9534
9534 lived for 11462312 nsec
9533 created 9535
9535 lived for 2319426491 nsec
9533 lived for 2407494647 nsec
9526 lived for 17243049061 nsec
.ereF& t!e output o% t!e ot!er 0indo0:
Password:
Last login: Tue Oct 31 15:39:48 2006 from 172.19.3.1
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
bash-3.00# ptree $$
6767 /usr/lib/ssh/sshd
9526 /usr/lib/ssh/sshd
9533 /usr/lib/ssh/sshd
9535 bash -o vi
9539 ptree 9535
bash-3.00# exit
T!e proce&& 0it! pid 6767 i& t!e sshd daemon to be 0atc!ed. It &pa0n& one proce&& -pid:
9526/ t!at 0ill !andle our SS. connection. T!i& proce&& al&o &pa0n& a couple o% proce&&e& t!at live
%or a &!ort time -pids 95271 95291 and 9531/. T!e %inal sshd proce&& -pid 9533/ i& created
a%ter t!e pa&&0ord 0a& entered. T!i& can be deduced %rom t!e ptree li&ting and t!e time&: 9535 i&
t!e &!ell %rom 0!ic! t!e ptree command 0a& &tarted. T!i& &!ell lived %or about 2.# &econd&. T!e
actual sshd proce&& o% intere&t lived %or )B.2 &econd&. $learly t!i& one i& t!e only candidate to
in&pect &ince all ot!er& do not live long enoug! to explain )2 &econd& o% time lo&t. In conclu&ion1 t!e
%ir&t proce&& being created by t!e sshd daemon i& t!e proce&& o% intere&t1 %or 0!ic! t!e &y&tem call&
mu&t be in&pected.
T!ere i& one %inal note to ma7e about t!e preceding code: 5onitoring t!e sshd daemon li7e t!i& 0ill
al&o &!o0 ot!er c!ildren created becau&e ot!er people 0ere trying to log onto t!i& &y&tem 0!ile t!e
sshd proce&& 0a& being monitored. During t!i& particular te&t I 7ne0 t!i& mac!ine 0a& dedicated to
me. T!ere%ore given t!i& context1 t!e above procedure 0a& correct.
.o0ever1 i% multiple &imultaneou& logon& are po&&ible1 &ome extra code i& needed. W!en
proc:::create %ire&1 t!ere i& exactly one argument. "& di&cu&&ed previou&ly1 t!i& argument i& o%
type psinfo_t. T!e psinfo_t &tructure contain& an additional element t!at de&cribe& t!e uid
-real and e%%ective/.
" re0rite o% t!e previou&ly explained DTrace &cript i&:
proc:::create
/pid == $target || (execname == "sshd" && forked[pid])/
{
forked[args[0]->pr_pid] = timestamp;
DTrace by Example: Solving a Real-World Problem ))
printf("%5d created %5d by %d\n", pid,args[0]->pr_pid,args[0]->pr_uid);
}
proc:::exit
/forked[pid]/
{
printf("%5d lived for %d nsec\n", pid, (timestamp - forked[pid]));
forked[pid] = 0;
}
.ere t!e printf %unction i& c!anged &o it al&o di&play& t!e u&er ID -UID/. Running t!i& &cript and
0atc!ing it& output &!o0&:
# dtrace -qp 6767 -s t.d
6767 created 9572 by 0
9572 created 9573 by 0
9573 lived for 17585966 nsec
9572 created 9575 by 0
9575 lived for 15074605 nsec
9572 created 9577 by 0
9577 lived for 14347357 nsec
9572 created 9579 by 0
9579 created 9580 by 100
9580 lived for 11359409 nsec
9579 created 9581 by 100
9581 lived for 11674291567 nsec
9579 lived for 11752588808 nsec
9572 lived for 26118751753 nsec
+o0 it become& clear 0!y t!ere i& <one extra< sshd proce&&. T!e %ir&t proce&& &pa0ned by t!e daemon
being 0atc!ed run& 0it! UID 0 -root/. T!i& one &pa0n& a ne0 sshd proce&&1 0!ic! !a& uid 100
-an ordinary u&er on t!i& &y&tem/. T!e %inal code mu&t be a little more complex to !andle t!i& &ituation.
W!en t0o di%%erent u&er& are trying to acce&& t!i& &y&tem at almo&t t!e &ame time1 t!e output -indented
to &!o0 parent-c!ild relation&!ip/ loo7& li7e t!e %ollo0ing example. -+ote t!at t!e elap&ed time i& no0
divided by )222222 and t!ere%ore repre&ent& milli&econd&./
dtrace -qp 6767 -s t.d
6767 created 9667 by 0 the first process creation for the first
request
9667 created 9668 by 0
9668 lived for 17
9667 created 9670 by 0
9670 lived for 14
9667 created 9672 by 0
9672 lived for 15
6767 created 9674 by 0 the process to handle the second request
since entering passwd and exiting again
was done before continuing the first conn
request; the setup for this second request
follows.
9674 created 9675 by 0
9675 lived for 15
9674 created 9677 by 0
9677 lived for 14
9674 created 9679 by 0
9679 lived for 15
9674 created 9681 by 0
9681 created 9682 by 100 Changing to the UID of the first
9682 lived for 14
DTrace by Example: Solving a Real-World Problem )2
9681 created 9683 by 100
9683 lived for 5983
9681 lived for 6063
9674 lived for 18429 The first one took 18 seconds. Now finished
9667 created 9688 by 0
9688 created 9689 by 111 Changing to the UID of the second
9689 lived for 11
9688 created 9690 by 111
9690 lived for 4366
9688 lived for 4446
9667 lived for 28808 This includes two serialized login sessions
at least doubling the wait time into the
20+ seconds. Here 18.4 seconds is for the
first session plus 10 seconds for the second
session.
W!at i& t!e pattern t!at can be u&ed to ma7e t!e &cript multi-u&er login a0are( T!e proce&& o% intere&t
i& t!at proce&& created by t!e sshd proce&& t!at i& being 0atc!ed. T!e proce&& o% intere&t i& al&o t!at
proce&& t!at create& a c!ild t!at no0 !a& a UID e4ual to t!at o% your&el%.
In order to do t!i&1 t!e c!ild o% a fork &y&tem call could call t!e setuid &y&tem call to c!ange it&
:ID ;u&t be%ore executing anot!er proce&& -t!e &!ell/. Expanding t!e previou& &cript to t!e one li&ted
belo0 0ill &!o0 0!en a setuid i& re4ue&ted:
syscall::setuid:entry
/forked[pid]/
{
printf("%s(%d) chown-ed to %d\n", execname, pid, arg0 );
}
Every time t!e setuid i& called1 t!i& clau&e print& t!e name o% t!e executable t!at did t!i&1 toget!er
0it! t!e pid and t!e %ir&t argument: t!e UID to c!ange to. T!i& re&ult& in t!e %ollo0ing output:
6767 created 10211 by 0
10211 created 10212 by 0
10212 lived for 17 nsec
10211 created 10214 by 0
10214 lived for 15 nsec
10211 created 10216 by 0
10216 lived for 14 nsec
sshd(10211) chown-ed to 0
sshd(10211) chown-ed to 0
sshd(10211) chown-ed to 0
sshd(10211) chown-ed to 0
10211 created 10218 by 0
sshd(10218) chown-ed to 100 <<---
10218 created 10219 by 100
10219 lived for 11 nsec
10218 created 10220 by 100
10220 lived for 3188 nsec
10218 lived for 3266 nsec
10211 lived for 17807 nsec
@oo7ing care%ully at t!i& output it become& clear t!at t!e proce&& o% intere&t -t!e one created by t!e
proce&& being 0atc!ed1 a& 0e learned previou&ly/ %or7& a c!ild. T!i& c!ild 0ill call t!e setuid
&y&tem call to obtain t!e proper u&er ID. It i& t!ere%ore t!e parent o% t!e proce&& t!at call& setuid t!at
i& t!e one o% intere&t.
DTrace by Example: Solving a Real-World Problem ),
T!e code t!at 0ill give t!e proce&& o% intere&t i& a& %ollo0&:
1 BEGIN
2 {
3 pidOfInterest = -1;
4 uidOfInterest = $1;
5 printf("UID: %d\n", uidOfInterest);
6 }
7
8 /* No need to remember target, only those processes spawned by it are
9 potentially interesting
10 */
11 proc:::create
12 /pid == $target/
13 {
14 forked[args[0]->pr_pid] = timestamp;
15 printf("Target %5d created %5d\n", pid,args[0]->pr_pid);
16 }
17
18 /* If a process spawned by $target now creates a child of its own it
19 should be remembered, since one of these will do the setuid we are
20 looking for
21 */
22 proc:::create
23 /execname == "sshd" && forked[pid]/
24 {
25 tree[args[0]->pr_pid] = pid;
26 }
27
28 proc:::exit
29 /forked[pid]/
30 {
31 forked[pid] = 0;
32 tree[pid] = -1;
33 }
34
35 proc:::exit
36 /pid == pidOfInterest/
37 {
38 printf("%5d lived for %d nsec\n"
39 ,pid, (timestamp-forked[pid])/1000000);
40 }
41
42 syscall::setuid:entry
43 /(tree[pid] > -1) && (arg0 == uidOfInterest)/
44 {
45 pidOfInterest = tree[pid];
46 printf("%s(%d) chown-ed to %d\n";, execname, pid, arg0 );
47 }
.ere i& t!e output %rom t!e DTrace &cript:
$ dtrace -q -p 6767 -s t.d 100
UID: 100
Target 6767 created 10334
Target 6767 created 10341
sshd(10348) chown-ed to 100
10334 lived for 1914186181 nsec
The output of our ptree command:
ptree $$
6767 /usr/lib/ssh/sshd
10334 /usr/lib/ssh/sshd
10348 /usr/lib/ssh/sshd
DTrace by Example: Solving a Real-World Problem )
10350 -bash
10354 ptree 10350
T!i& code o%%er& proo% t!at t!i& proce&& 0or7&. In t!eory it can !appen t!at multiple u&er& log on u&ing
t!e &ame u&er ID. In t!at ca&e1 t!i& &cript i& not able to determine t!e di%%erence. It i& le%t a& an exerci&e
to c!ange t!e &cript &o it can !andle t!at &ituation.
Some ne0 con&truct& need to be introduced. T!e&e 0ill be di&cu&&ed by line number a& &!o0n in t!e
previou& code li&ting. @ine& )-= introduce t!e BEGIN clau&e &peci%ication. T!e action in t!i& clau&e i&
executed be%ore any o% t!e ot!er probe& &peci%ied can be triggered. T!e $1 notation on line depict&
t!e %ir&t parameter %or t!e DTrace &cript1 a& &peci%ied on t!e command line. In t!i& example $1
repre&ent& t!e user id o% intere&t. "lt!oug! it i& not nece&&ary to &tore t!i& $1 macro variable in
anot!er one1 it ma7e& t!e code more under&tandable.
"l&o t!e variable pidOfInterest i& initialiGed to -1. T!i& value i& c!o&en &ince t!ere 0ill never be
a pid on a &y&tem 0it! t!i& value. -1 t!ere%ore i& a nice candidate %or <no pid !a& been %ound up to
no0.<
T!e original proc:::create probe i& no0 &plit over t0o di%%erent clau&e&. T!e %ir&t clau&e -line ))/
!a& a predicate t!at enable& it to mea&ure time. T!e &econd clau&e -line 22/ i& needed becau&e t!e
intere&t i& no0 &!i%ted to &omet!ing el&e. pid i& a potential candidate o% intere&t. T!e proce&& 0it! t!i&
pid i& in t!e %or7ed array. T!ere%ore it 0a& &pa0ned by our target. T!e c!ild !a& a pid t!at i& u&ed
a& t!e 7ey into t!i& a&&ociative array. T!i& enable& t!e loo7up o% t!e parent in t!e clau&e &peci%ied on
line 2C
W!en t!e syscall::setuid:entry probe %ire&1 t!e pid t!at triggered t!e probe i& &tored in t!e
array tree1 and t!i& pid i& a potentially intere&ting pid. T!i& &y&tem call i& u&ed to c!ange t!e e%%ective
u&er ID into t!e one o% intere&t1 &o t!e parent proce&& o% t!e proce&& t!at triggered t!e probe i& t!e
proce&& %or 0!ic! t!e &y&tem call& need to be timed.
8inally1 in t!i& &cript t!ere are al&o t0o e4ual probe& proc:::exit. 6nce more1 t!e action ta7en i&
determined by t!e predicate. I% t!e exit i& o% a proce&& o% intere&t -clau&e at line 2*/ t!en t!e in%ormation
related to it i& no longer important: T!e action part clean& it. T!e ot!er clau&e at line ,# i& a dummy
!elping to &!o0 t!e proo%.
6ne more &tep i& needed to %ind t!e in%ormation t!at 0ill o%%er more in&ig!t into t!e problem. T!e
previou&ly li&ted &cript need& to be en!anced to li&t t!e elap&ed time per &y&tem call o% t!e proce&& o%
intere&t. T!e extra code needed to realiGe t!i& %ollo0&:
syscall:::entry
/pid == pidOfInterest/
{
self->ts = timestamp;
}
syscall:::return
/pid == pidOfInterest && self->ts/
{
@elapro!efunc" = sum#timestamp - self->ts$;
}
%&'
{
normali(e#@ela) *+++$;
DTrace by Example: Solving a Real-World Problem )#
printa#,-*.s -@*.u/n,) @ela$;
}
W!en t!i& &cript i& run1 t!e output i&:
UID: 100
Target 6767 created 10413
sshd(10420) chown-ed to 100
10413 lived for 1918517727 nsec
^C
c2audit 13
shutdown 16
getpeername 16
nfs 19
getgid 20
waitsys 25
lseek 27
dup 32
setcontext 32
ioctl 39
pipe 47
llseek 64
getuid 108
sigaction 113
gtime 115
lwp_sigmask 139
fstat64 166
fcntl 185
open64 198
read 199
stat64 222
getpid 257
write 493
putmsg 518
close 605
fstat 825
doorfs 1157
open 1556
munmap 1925
pollsys 3910441
T!i& re&ult doe& not meet expectation&. +ot even clo&e.
T!e time-con&uming &y&tem call& o% intere&t !appened be%ore t!e setuid call 0a& made. 6nce it i&
7no0n 0!ic! proce&& i& intere&ting1 t!e be!avior t!at ma7e& t!i& proce&& intere&ting !a& already
!appened. We need to be able to loo7 bac7 in t!e pa&t. DTrace provide& &peculation& to enable loo7ing
bac7 once an event !appen& A %or example1 to determine 0!at led to an error. :n%ortunately t!i& %eature
cannot !andle aggregation&.
In order to include data %rom t!e pa&t1 every &y&tem call done by any o% t!e proce&&e& o% intere&t need&
to be recorded. 8ortunately DTrace &upport& compound 7ey& %or a&&ociative array&. 3y prepending t!e
pid to t!e &y&tem call1 all data collected can be connected to a certain pid. I% t!e pid i& 7no0n an
external &cript can be 0ritten to %ilter out all data t!at doe& not relate to t!i& pid. .o0ever1 you do
need to u&e external &cripting to accompli&! t!i&.
DTrace by Example: Solving a Real-World Problem )=
T!e re&ulting DTrace &cript doe& not need t!e pidOfInterest variable any more. T!e re&ulting
updated clau&e& %or t!e syscall:::entry/return probe& are:
syscall:::entry
/forked[pid]/
{
self->ts = timestamp;
}
syscall:::return
/self->ts/
{
@ela[pid,probefunc] = sum(timestamp - self->ts);
}
T!e only proce&&e& t!at are examined are t!o&e t!at are potentially o% intere&t. "l&o1 t!e aggregation
no0 u&e& a compound 7ey pid, probefunc. In order to get t!e correct output1 t!e printa call in
t!e END clau&e need& a di%%erent %ormat &tring: %6d %12s %@12u\n. " -0d i& added. T!i& one
contain& t!e %ormat in&truction %or t!e %ir&t element o% t!e 7ey -t!e pid/. T!e -*.s i& t!e in&truction
%or t!e &econd part o% t!e 7ey. T!e %inal %@12u i& &till t!e %ormat in&truction %or t!e data a&&ociated
0it! t!e compound 7ey.
Since t!e pid o% intere&t &!ould be 7no0n to enable an external program to %ilter out t!e relevant data1
t!e syscall::setuid:entry no0 become&:
syscall::setuid:entry
/(tree[pid] > -1) && (arg0 == uidOfInterest)/
{
printf("pidOfInterest: %d\n", tree[pid]);
}
"ll ot!er in&tance& o% printf are removed %rom t!e &cript. Running t!i& &cript and u&ing t0o di%%erent
SS. client& to &imultaneou&ly log in re&ult& in t!e %ollo0ing output:
pidOfInterest: 10694
^C
10694 lwp_self 0
10701 lwp_self 0
10701 shutdown 0
... lots of calls taking 0 milliseconds. Followed by some using a small
... amount of milliseconds finally followed by the "huge consumers"
10701 write 9
10694 write 12
10701 open 17
10694 open 17
10701 read 35
10694 read 35
10694 pollsys 7132
10701 doorfs 7229
10701 pollsys 9018
10694 doorfs 10123
DTrace by Example: Solving a Real-World Problem )B
6ur pid o% intere&t i& 10694. T!e la&t line in t!e output &!o0& t!at t!i& proce&& u&ed )2.)2, &econd&
in t!e &y&tem call doorfs. T!i& number indeed matc!e& t!e time t!at 0a& &oug!t. 8inally1 t!i& &cript i&
ready and &!o0& t!e in%ormation t!at 0a& needed. T!e doorfs call con&ume& t!e <mi&&ing< time.
doorfs: What Is It?
Since t!i& in%ormation come& %rom 0atc!ing t!e syscall provider1 doorfs mu&t be a &y&tem call.
In order to under&tand 0!at a &y&tem call &!ould do1 it& manual page& mu&t be read. Sy&tem call& are
documented in &ection 2. .o0ever1 man -s 2 doorfs doe& not provide a document.
In t!e Solari& 6S1 t!e door file system %acility i& available %or %a&t and e%%icient inter-proce&&
communication -IP$/. It can only be u&ed %or communication bet0een proce&&e& running in t!e &ame
Solari& environment. -In t!e current day& o% virtualiGation t!ere can be multiple& o% t!e&e environment&
running on one &ingle &erver./
Hno0ing t!at t!e abbreviation fs i& commonly u&ed %or %ile &y&tem1 t!e doorfs call &eem& to be
related to t!i& IP$ mec!ani&m. Indeed it i& an undocumented call u&ed by t!e door li!rary
%unction&. See man -s 1'OO2 door_create. " door i& created by a &erver. T!e 0ell-7no0n
name o% t!i& door1 be!ind 0!ic! a &ervice i& provided1 i& vi&ible in t!e %ile &y&tem. 3y opening t!i&
%ile1 a %ile de&criptor i& created1 0!ic! i& al&o 7no0n a& t!e door de&criptor in t!i& context. T!i& door
de&criptor i& t!en u&ed a& t!e identi%ier during t!e u&e o% t!e actual door.
@et u& no0 %ocu& on t!i& &y&tem call to get a better under&tanding o% 0!at i& !appening. T!e
accumulation o% time in t!i& doorfs call could be due to &ome call& ta7ing a !uge amount o% time or
to a !uge number o% call& t!at all return almo&t immediately. To &ee t!e di&tribution o% time1 t!e
%ollo0ing DTrace &cript i& u&ed. .ere t!e lquantize aggregation %unction i& u&ed to create a linear
!i&togram. T!e upper bound i& 7no0n to be a little above )2 &econd&. T!e lo0er bound i& clearly 21
&ince elap&ed time i& mea&ured. T!e lquantize(this->ela, 0, 12, 1) %it& t!e value this-
>ela in a !i&togram. In t!i& !i&togram1 t!e cumulation o% value& &tart& at +1 and end& at *.1 0it! an
interval lengt! o% 1. T!u&1 [0,1), [1,2), [2,3), .., [11,12) are t!e range& covered by
t!e element& o% t!e !i&togram.
T!e previou& &cript !a& it& syscall:::entry and return c!anged1 a& &!o0n in t!e %ollo0ing
code. 3ecau&e t!e aggregation i& no0 u&ed 0it! a &ingle 7ey1 t!e %ormat &tring in t!e printa &!ould
al&o be c!anged.
T!e this-> con&truct i& u&ed to create and u&e a local variable. T!i& one i& valid in t!e &cope o% t!e
bloc7 in 0!ic! it i& de%ined: t!e action.
syscall::doorfs:entry
/forked[pid]/
{
self->ts = timestamp;
}
syscall::doorfs:return
/self->ts/
{
this->delta = (timestamp - self->ts) / 1000000000;
@ela[pid] = lquantize(this->delta, 0, 12, 1);
}
END
{
DTrace by Example: Solving a Real-World Problem )*
printa("%6d %@12u\n", @ela);
}
T!e output i& no0:
pidOfInterest: 10799
10799
value ------------- Distribution ------------- count
< 0 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 303
1 | 0
2 | 0
3 | 0
4 | 0
5 | 2
6 | 0
.ere 10799 0a& t!e pid o% intere&t. 5o&t call& too7 bet0een Gero and one &econd. Exactly t0o call&
too7 %ive to &ix &econd&. T!e total time needed i& )2 &econd&1 4uite clo&e to t0o time& %ive. Iiven t!at
%ive i& a <!uman number1< t!e cau&e o% t!e problem may turn out to be due to !uman error.
To under&tand more1 it i& nece&&ary to learn 0!at t!e&e t0o door call& do to t!e door identi%ier. Since
a door identi%ier i& returned by a call to open1 t!i& one can be a&&ociated 0it! a %ile name. -6ne
complication in doing t!i& i& t!at %ile de&criptor& can be re-u&ed a%ter being clo&ed./
To %ind out more about t!i& undocumented1 probably catc!all1 doorfs &y&tem call1 0e need to vi&it
!ttp:99&rc.open&olari&.org9&ource9. @oo7ing at t!e $ &ource code o% t!e door &y&tem call &!o0& extra
in%ormation on doorfs -&ee
!ttp:99cv&.open&olari&.org9&ource9xre%9on9u&r9&rc9ut&9common9%&9door%&9doorJ&y&.c/:
static int
doorfs(long arg1, long arg2, ..., long arg5, long subcode)
{
switch (subcode) {
....
}
}
T!e &ixt! parameter i& called su!code and drive& t!e deci&ion in t!e &0itc!. T!e %ollo0ing c!ange o%
t!e syscall::doorfs:entry and syscall::doorfs:return probe& &!o0 u& t!e value o%
t!i& and t!e ot!er parameter& %or t!o&e call& t!at ta7e longer t!an %ive &econd&:
syscall::doorfs:entry
/forked[pid]/
{
self->ts = timestamp;
self->arg0 = arg0;
self->arg1 = arg1;
self->arg2 = arg2;
self->arg3 = arg3;
self->arg4 = arg4;
self->arg5 = arg5;
self->arg6 = arg6;
}
syscall::doorfs:return
/self->ts > 0/
{
this->delta = (timestamp - self->ts) / 1000000000;
DTrace by Example: Solving a Real-World Problem )'
@ela[pid] = lquantize(this->delta, 0, 12, 1);
if (this->delta > 5) {
printf("%ld %ld %ld %ld %ld %ld \n", self->arg0, self->arg1,
self->arg2, self->arg3, self->arg4, self->arg5, self->arg6);
}
}
:n%ortunately t!e if () {} con&truct i& not &upported in DTrace. T!i& can be done 0it! proper u&e
o% predicate&. T!e syscall::doorfs:return clau&e 0ill no0 be replaced by t0o di%%erent
clau&e& t!at !ave t!e &ame probe:
syscall::doorfs:return
/self->ts > 0/
{
self->delta = (timestamp - self->ts) / 1000000000;
@ela[pid] = lquantize(self->delta, 0, 12, 1);
}
syscall::doorfs:return
/self->delta >= 5/
{
printf("%ld %ld %ld %ld %ld %ld \n", self->arg0, self->arg1,
self->arg2, self->arg3, self->arg4, self->arg5);
}
T!e output o% t!i& &cript i&:
3 4290756968 4278122472 4294967295 4281999360 3
3 4290756840 4278122472 4294967295 4281999360 3
pidOfInterest: 10933
^C
10933
value ------------- Distribution ------------- count
< 0 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 303
1 | 0
2 | 0
3 | 0
4 | 0
5 | 2
6 | 0
T!i& &!o0& t!at t!e &ubcode !a& t!e value 3 %or bot! doorfs call& t!at ta7e approximately %ive
&econd&. @oo7ing at t!e &ource code1 it i& obviou& t!at t!i& i& a clean $ program u&ing macro name&
in&tead o% !ard-coded value&. 8ortunately1 eac! Solari& )2 &y&tem &!ould !ave t!e %ile
/usr/include/sys/door.h. @oo7ing at t!i& !eader %ile it become& clear t!at t!e macro called
'OO2_3455 !a& t!e value 3. 8rom t!e $ &ource it %ollo0& t!at t!e %unction door_call i& called:
switch (subcode) {
case DOOR_CALL :
return door_call(arg1, (void*) arg2));
DTrace by Example: Solving a Real-World Problem 22
Since all parameter& 0ere printed1 it can be deduced t!at arg1 al&o !a& t!e value 3 and arg2 i&
&ome !ig! number1 li7ely a pointer. " little %urt!er into t!i& &ource %ile1 t!e &peci%ication %or
door_call can be %ound:
int door_call(int did, void *args)
{
...
}
.ere t!e %ir&t parameter i& called did. 5o&t li7ely t!i& &tand& %or doorid. T!e circle can no0 be
clo&ed. Indeed t!ere i& a li!door %unction called door_call1 0!ic! ta7e& a door id a& it& %ir&t
parameter and a pointer a& it& %inal &econd parameter. 3y &toring t!e %ile name a& a value in an
a&&ociative array 0!ere t!e 7ey i& t!e %ile de&criptor1 t!i& %ile name can be retrieved once t!e door
de&criptor i& &een. T!e code needed to &tore t!e %ile name& become&:
syscall::open*:entry
/forked[pid]/
{
self->filename = copyinstr(arg0);
self->hasFile = 1;
}
syscall::open*:return
/self->hasFile == 1/
{
filenames[arg0] = self->filename;
self->hasFile = 0;
}
T!ere are t0o type& o% open call&: open() and open64(). T!e open* notation u&e& a 0ildcard to
!ave t!i& probe &peci%ication matc! t!e&e t0o probe&. T!e copyinstr() i& a DTrace %unction. T!e
%ile name re&ide& in u&er &pace -t!e executable being 0atc!ed/. T!i& DTrace &cript run& in t!e 7ernel.
Since t!i& &tring 0ill be re%erenced later a private copy need& to made to en&ure it can be addre&&ed
later on.
6nly i% t!e &y&tem call return& i& it& return value 7no0n. T!e return value o% t!e open call& bot! depict
t!e %ile de&criptor. "t t!at time t!e %ilename 0ill be &tored 0it! t!e %ile de&criptor a& it& 7ey %or later
retrieval.
T!e t!read local variable self->hasFile i& u&ed to &!o0 t!e validity o% self->filename. "
&pecial value "" could be u&ed to do t!i&? alt!oug! not advi&able1 t!i& value i& a legal value %or a
%ilename.
3ot! t!e syscall::doorfs:entry and return probe mu&t be c!anged in order to loo7 up t!e
%ilename t!at i& u&ed:
syscall::doorfs:entry
/forked[pid]/
{
self->subcode = arg5;
self->delta = 0;
}
syscall::doorfs:entry
/self->subcode == 3/
DTrace by Example: Solving a Real-World Problem 2)
{
self->ts = timestamp;
self->did = arg0;
}
syscall::doorfs:return
/self->subcode == 3/
{
self->delta = (timestamp - self->ts) / 1000000000;
}
syscall::doorfs:return
/self->delta > 4/
{
printf("door call to %s used %d seconds\n",
filenames[self->did], self->delta);
}
T!e output o% t!i& &cript i&:
door call to /var/run/name_service_door used 5 seconds
door call to /var/run/name_service_door used 5 seconds
name_service_door i& called t0ice. In bot! ca&e& it ta7e& about %ive &econd&. :nder&tanding 0!at
name_service_door i& u&ed %or 0ill very li7ely lead to t!e &olution to t!e problem.
#ame Service Cac!e Daemon
.ere end& t!e u&e o% DTrace. To complete t!i& &tory1 a &mall explanation %ollo0&1 &!o0ing t!at t!e )2
&econd& are indeed due to a name loo7up con%iguration error.
Wit! t!e Solari& 6S1 t!e /etc/nsswitch.conf %ile i& u&ed to determine t!e order in 0!ic!
re&ource& &!ould be in&pected to determine a name. 8or example1 to c!ec7 t!e passwd t!at come&
0it! a u&er name1 to %ind t!e IP addre&& t!at map& to a hostname1 and &o on.
T!e sshd daemon i& a utility t!at ta7e& care o% u&er name and pa&&0ord mapping1 and provide& a
&ervice %or non-local u&er&. T!e time lo&t 0a& be%ore t!e daemon a&7ed %or a pa&&0ord. T!ere%ore it i&
unli7ely t!at t!i& mapping 0a& t!e cau&e. T!ere%ore1 t!e %ir&t candidate 0a& t!e name &erver. T!e
/etc/nsswitch %ile &!o0ed t!at %ir&t t!e /etc/hosts %ile A and t!en1 i% t!at 0a& not &ucce&&%ul1
t!e domain loo7up &ervice -D+S/ A 0ould be u&ed to %ind hostname or IP addre&&. In&pecting
/etc/hosts %ile &!o0ed t!i& %ile 0a& empty. T!ere%ore a &ervice %rom D+S 0a& in place. T!e
/etc/resolv.conf %ile i& t!e con%iguration %ile %or D+S.
6ne t!ing t!i& tell& u& i& t!e IP addre&& t!at provide& t!i& &ervice. In&pecting t!i& %ile &!o0ed t!at t!i& IP
addre&& !ad a typo. W!en t!i& 0a& corrected1 t!e SS. client returned immediately. $learly t!e problem
0a& &olved. .o0ever1 t!e an&0er to t!e 4ue&tion o% t!o&e )2 &econd& 0a& not yet %ound.
"ll t!e&e name loo7up &c!eme& can u&e a cac!e daemon1 called t!e name &ervice cac!e daemon
-nscd/. T!i& i& t!e daemon t!at adverti&e& t!e door t!at 0a& %ound -name_service_door/. @i7e
almo&t all daemon&1 t!i& one !a& a con%iguration %ile: /etc/nscd.conf.
In&pecting t!i& one &!o0& t!at nscd de%ault& to a timeout value o% %ive &econd& 0!ile 0aiting %or a
D+S loo7up. So !ere i& t!e an&0er: T!e %ir&t attempt to re&olve &omet!ing 0it! D+S timed out.
T!ere%ore t!e sshd daemon tried again and once more &a0 an error a%ter %ive &econd&. Since t!e
pa&&0ord prompt %inally appeared1 &urely t!e data t!at needed to be re&olved 0a& not critical.
DTrace by Example: Solving a Real-World Problem 22
In t!e end t!e problem 0a& &olved. T!i& 0ould !ave been almo&t impo&&ible to do i% t!e DTrace tool7it
!ad not been available.
References
)/ Solaris 2ynamic 1racing +uide
!ttp:99doc&.&un.com9app9doc&9doc9*)B-=22,
2/ 6penSolari& $ommunity: DTrace
!ttp:99000.open&olari&.org9o&9community9dtrace
$icensing Information
)nless other%ise speci'ied, the use o' this so't%are is authori>ed pursuant to the terms o' the license 'ound at
http?55developers.sun.com5.er&[email protected]
DTrace by Example: Solving a Real-World Problem 2,