Modelling Variance Heterogeneity in Normal Regression Using GLIM

Author(s): Murray Aitkin

Source: Journal of the Royal Statistical Society. Series C (Applied Statistics), Vol. 36, No. 3
(1987), pp. 332-339
Published by: Wiley for the Royal Statistical Society
Stable URL: http://www.jstor.org/stable/2347792 .
Accessed: 04/11/2014 19:30

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact [email protected].

Wiley and Royal Statistical Society are collaborating with JSTOR to digitize, preserve and extend access to
Journal of the Royal Statistical Society. Series C (Applied Statistics).

http://www.jstor.org

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
Appl. Statist. (1987)
36, No. 3, pp. 332-339

in Normal
ModellingVarianceHeterogeneity
RegressionUsing GLIM
By MURRAY AITKINt
Universityof Lancaster,UK

[Received August 1985. Revised January1987]

SUMMARY
This paper describes and presentssimple GLIM macros forthe modellingof variance heterogeneity
in normalregressionanalysis, using a log-linearregressionmodel forthe variance. The procedureis
illustratedwithtwo examples.
Keywords: Variance heterogeneity;Score test; GLIM; maximumlikelihood

1. Introduction
Homogeneityof varianceis one of the standardassumptionsof normalregressionanalysis.
The failureofthisassumptioncan sometimesbe rectified by a Box-Cox (1964) transformation
of the responsevariable,whichmay providethe added benefitsof closerapproximationsto
addivityand normality.This does not necessarilyoccur,however,and it is oftenof interest
is to findand
of variancein the analysis.The difficulty
to allow explicitlyforheterogeneity
fita satisfactorymodel forthe heterogeneity.
Modellingapproachesusingthe score testare proposedand used by Cook and Weisberg
(1983). For the normalregressionmodel
y= 3'xi + ei (1)
withei- N(O,a2) underhomogeneity,
theypropose the model forheterogeneity
var(ei) = a' = exp(X'zi), (2)
wherezi may containsome or all of the variablesin xi and othervariablesnot includedin
xi; zi is assumedto containa constant1. The log-linearformensuresthatoi remainspositive.
The resultsofChen (1983) suggestthatthescoretestused by Cook and Weisbergis not very
sensitiveto thefunctionalformoftherelationshipbetweenoQ and X'zi. The advantageofthe
score testis thatthe model (2) does not have to be explicitlyfitted;the disadvantageis that
the curvatureof the likelihoodat the null hypothesismay be verydifferent fromthatat the
maximumlikelihoodestimateofA,leadingto poor agreementbetweenthescoreand likelihood
ratio tests.
The purpose of this note is to point out that the model (2) can be fittedby maximum
likelihoodin GLIM3, (Bakerand Nelder 1978)and thatthemean regressionparameter/Bcan
be estimatedsimultaneously, allowingfor the model (2) for the dispersionusing a simple
macro. This resultprovides a powerfulgeneral regressionmodellingapproach in which
varianceheterogeneity is explicitlymodelledratherthanbeing(hopefully) transformed away.

Researchand Services,EducationalTesting
t Addressforcorrespondence:DivisionofStatisticaland Psychometric
Service,PrincetonN.J.08541, U.S.A.
? 1987 Royal StatisticalSociety 0035-9254/87/36332 $2.00

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
 VARIANCE HETEROGENEITY 333
Generalizationsofthisapproachto theexponentialfamilyare suggestedby Pregibon(1984),
and a veryfulldiscussionis givenby Smyth(1985).
Section2 givesdetailsofthejoint maximumlikelihoodestimationof/Band A by successive
relaxation.Section 3 illustratesthe proceduresusingtwo well-knownexampleswithstrong
dispersionheterogeneity. Section4 givesa discussionof the macrosand detailsof necessary
arguments, usingGLIM 3.77.
2. Maximum LikelihoodEstimation
Given observations(yi,xi, zi), i= 1, ..., n followingmodel (2), thekernelof thelikelihood
functionis
n 1
L(/B,A) = H -exp _
Xi)2/

with
log Ui = zj.
The log-likelihoodis
1(/,A)=1 E log UCi-4 E (yi-_#/xi)2/a1
and the firstand second derivativeswithrespectto the parametersare

a:_ -
a21~~~~~~~
ZXi(yi-#'xi)lui
=#F _-E xjx'/Ui

01 E Zi + 2 E (yi-
- 'Xi)2Zi-a2
OA 2E e21i- i
_
=- E (e 2/U2 )Zi
a21

a21
a/3x=-E - (ei/ua)xiz,

where
ei = yi- 3'xi
The observedinformation forthecomplete
matrix parameter A)is
vector(/B,
rX'Wl1X X'Wl2Z
[Z'W12X Z'W22Zj

where
X = Xl... Xn] Z' = [z1, ** Zn,
W, 1 = diag(I/&2)
H12 = diag(e&/&i
)
= 4 diag(ei2/&i2)
ei = y-3- Xi.

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
334 AITKIN

matrixis
The estimatedexpectedinformation
>=X'WjjX O
L ? ~2Z'Zj
sinceE(e2) = a2. Thus a Fisherscoringalgorithmforthe simultaneousmaximumlikelihood
estimationof /Band A reducesto two separatealgorithmsfor/3and A. Since howeverW1
dependson A and ej depends on ,B,the completescoringalgorithmcan be formulatedas a
successiverelaxationalgorithm.
For giveno2, , is a weightedleast squares estimatewithweights1/2, and forgiven/3, X
is themaximumlikelihoodestimateobtainedfroma gamma distribution of e2, withdegrees
offreedomv = 1 and henceGLIM "scale" parameter(squaredcoefficient ofvariation)2/v= 2,
and a log linkfunctionforthe gamma mean 2.
The algorithmcan conveniently begin withan initialunweightednormalregressionof y
on x, takingu =- 2. The deviance(-2 log Lmax)is calculatedforthismodel,to providea
global testof homogeneityof variance.The squared residualsfromthe least squares fitare
definedas a new responsevariablewitha gamma distribution withscale parameter2. The
linearpredictorA'z is thenfittedusing a log link function,and the deviancecalculatedfor
theinitialestimateof(/3, A).A weightednormalregressionof y on x is now fitted, withscale
parameter1and weightsgivenbythereciprocalsofthefitted valuesfromthegammamodel.The
squared residualsare again fittedusingthegammaerrormodel,and thenormaland gamma
models are alternateduntilthe devianceconverges.At this point the standarderrorsfrom
bothmodelsare correct(thoseforthegammamodelin GLIM are based on theFisherscoring
algorithmwhich uses the expected informationmatrix).This analysis assumes that the
parameters/3 and A are functionally
independent, whichwill generallybe the case.
We will illustratewithtwo examples.
3. Examples
We considerfirsta 3 x 4 factorialdesignwithfourreplicates,the well-knownexamplein
Box and Cox (1964). The responseis the survivaltimeof rats afterpoisoning,classifiedby
threetypesof poison and fourtreatments. Box and Cox noted thatforthe untransformed
survivaltime,withintreatment normality is acceptablebutcommondispersionand additivity
are not. If a model withcommondispersionis fitted,however,the interactionappears only
marginallysignificant. The Box-Cox transformation pointsstronglyto the reciprocalscale,
and we proceedto model on thisscale, using rate= 1/timeas responsevariable.
The generalprocedurefollowedis to fita saturatedmodelforthemean,and use thisto find
an appropriatemodel for the dispersion,beginningwith the saturatedmodel and using
backwardelimination.Whena finalmodelforthedispersionhas been found,themean model
is thensimplifiedin a similarway.
Fittingthesaturatedmodelsforbothmeanand dispersionwefindthatnoneoftheinteraction
parametersin thedispersionmodel is large,and omittingtheinteraction, i.e. fitting
the main
effectsmodelforthedispersion,givesa devianceincreaseof5.91 on 6 df.The further omission
of the treatmentmain effectgives a devianceincreaseof 3.19 on 3 df.Omittingthe poison
typemain effectgivesan increaseof 5.68 on 2 df,and thisis almost all concentratedin the
contrastofType 2 withtheothertwo types:equatingtheType 1 and 3 dispersionsincreases
thedevianceby only0.12 on 1 df,thecontrastof Type 2 withthe othertwo increasingit by
5.56 on 1 df.
The finalmodel forthe dispersionon the rate scale thus specifiesa commonvariancefor
Types1 and 3 buta largervarianceforType2 (theGLIM codeis $CALC TYP2= %EQ(TYPE,2)
$FIT TYP2 $).
The parameterestimatesforthe saturatedmean model do not depend on the dispersion
model fitted,but theirstandarderrorsdo depend on thismodel. The largestinteractionof
-0.9137 is forTYPE(3). TREA(4), and its standarderroris 0.490in thecommondispersion

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
 VARIANCE HETEROGENEITY 335
model,0.367 in the saturateddispersionmodel,and 0.340 in the finaldispersionmodel with
a Type 2 term.The standarderrorsare smallerin the last two models because the variance
estimateis based on the smallercell and Type 3 variances.
Fittingthe main effectsmodel for the mean with the Type 2 dispersionmodel gives a
devianceincreaseof 10.60on 6 df.Addinga parameterfortheType 3, Treatment4 cell using
a dummyvariable D34 (the GLIM code is $CALC D34= %EQ(TYPE,3)* %EQ(TREA,4) $FIT
TYPE + TREA + D34 $D E$) reducesthe devianceby 4.65 on 1 df,the remaining
fiveinteraction
parametersaccountingfora devianceof 5.95. The importanceof thisinteractiontermis not
evidentin thecommondispersionmodel,wherethedeviancechangeforD34 is 3.48,and that
forthe remainingfiveinteractionparametersis 4.54.
Table 1 showsthecell meansand variancefortherateresponsevariable,and fittedvalues
for the mean from the main effects/common dispersionmodel and for the final main
+ D34 mean model withthe Type 2 dispersionmodel,and forthe variancefromthe
effects
Type 2 model.
The secondexampleis theMINITAB treedata (Ryan,Joinerand Ryan 1976)discussedin
Atkinson(1982) and Cook and Weisberg(1983). The responsevariableis the volume V of
usable wood in 31 cherrytrees,withexplanatoryvariablesthe heightH and diameterD of
thetrees.One oftheplausiblephysicalmodelsforthetreevolumeregressesV1/3on H and D.
Cook and Weisbergnote increasingvariancewithH of the least squares residualsfrom
thismodel,but the score testforthe dispersionusingH is only 3.24 on 1 df.We beginwith
modelsforbothmean and dispersionusingH and D. The respectiveparameterestimatesand

TABLE I
Cell means for rate, with (fittedvalues) from main effects
commonvariancemodel,and [fittedvaluesjfrommaineffects +
D34, Type2 variancemodel

Treatment
(note order)
1 3 4 2
3 4.803 4.265 3.092 3.029
(4.694) (4.122) (3.336) (3.037)
[4.7581 [4.181] [3.092]t [3.160]
TYPE
(note order) 2 3.268 2.714 1.702 1.393
(3.166) (2.594) (1.808) (1.509)
[3.072] [2.495] [2.039] [1.474]

1 2.487 1.863 1.690 1.163

(2.698) (2.126) (1.340) (1.041)
[2.603] [2.026] [1.570] [1.005]

Cell variances+forrate,with[fittedvalues] fromType2 model.

Treatment
1 3 4 2
3 0.2104 0.0413 0.0447 0.1332
[0.1287] [0.1287] [0.1287] [0.1287]
TYPE
2 0.5072 0.1307 0.3695 0.2295
[0.3610] [0.3610] [0.3610] [0.3610]

1 0.1850 0.1796 0.0998 0.0299

[0.1287] [0.1287] [0.1287] [0.1287]

t The D34 dummyvariablereproducesthe observedmean in thiscell.

+ Within-cellsum of squares dividedby 4.

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
336 AITKIN

(standarderrors)for the (loglinear)dispersionmodel are 0.109 (0.047) and 0.032 (0.096).

OmittingD fromthedispersionmodelincreasesthedeviancebyonly0.07,and theestimatefor
H is now 0.114 (0.041). Fittingthe null dispersionmodel increasesthe devianceby 5.11, so
theheteogeneity withH is clearlyestablishedby the likelihoodratio test,thoughnot by the
score test.As noted in Section 1, this resultcan be expectedwhen the likelihoodin the
dispersionmodel parametersis stronglyskewed,so that the curvatureat i = A0 does not
correspondto the curvature(or generalshape of the likelihood)at A= A.
Furtherexaminationoftheresidualplots againstH and D suggestsa patternofincreasing
and then decreasingdispersion,thoughthis mightbe due simplyto the largernumberof
pointsin the centreof the plot. To check this,we fita quadraticresponsesurfacemodel in
H and D to the (log) dispersion,retainingthe linearmodel in H and D forthe mean. The
additional three parametersin the dispersionmodel reduce the deviance by 15.37, and
successivebackwardeliminationof termsfromthismodel reducesthe model to a quadratic
inD only,thedeviancechangeforthethreeomittedvariablesbeing4.69.The successivemodels
formean and dispersionare shown in Table 2, togetherwiththe mean model forconstant
dispersion(thestandarderrorsin thelast model are slightlysmallerthanthosefroman OLS
fitas a2 ratherthantheRMS estimators2is used).The standarderrorsin themeanmodelare
considerablyreduced,thoughtheparameterestimatesare close to thosefromthe OLS fit.A
notablefeatureof Table 2 is the verypoor agreementbetweenthe likelihoodratio testand
the Wald test(Xj/S.E.(Aj))forthe dispersionmodel parameters.
It may seem surprisingthatsuch strongdispersionheterogeneity can be identified.
Fig. 1
shows on a log scale the squared residualsfromthe mean model withconstantdispersion,
and thefittedvaluesofthevariancefromthedispersionmodel(smoothcurve)plottedagainstD.
Ifour interestis in predictionoftreevolumegiventreeheightand diameter,thedispersion
heterogeneity has a strongeffecton the precisionof prediction:treevolumesforsmall and
largediametertreesare morepreciselypredictedthan volumesformediumdiametertrees.
An alternativemodel fortreevolume regresseslog V on log H and log D. The analysis
above was repeatedforthismodel,usinga linearmodelin log H and log D forthemean,and
a quadraticresponsesurfacemodel in log H and log D forthe dispersion.The finalmodel
forthedispersionwas quadraticinlog D; thatforthemeanwas consistent withlog H + 2 log D,
the "solid of rotation"model discussedin Atkinson(1982).

TABLE 2
Mean and dispersonmodelsfor V113, MINITAB treedata

Mean model Dispersion model Deviance

const H D const H D H2 HD D2

-.099 ..0149 .150 -13.98 .109 .0322 0 0 0 -74.813

(.0021) (.0049) (.047) (.097)
-.104 .0149 .150 -13.94 .114 0 0 0 0 -74.746
(.0021) (.0048) (.041)
-.085 .0144 .152 -5.08 0 0 0 0 0 -69.636
(.0026) (.0054)
.0058 .0135 .149 -98.19 2.339 -0.366 -0.0213 .0813 -0.212 -90.188
(.0008) (.0015) (0.947) (1.314) (0.0070) (.0302) (0.054)
.0454 .0127 .151 -82.34 1.279 3.715 -0.00788 0 -0.129 -88.370
(.0011) (.0022) (0.944) (0.777) (0.00630) (0.027)
.0776 .0120 .152 -41.86 0.0879 4.317 0 0 -0.150 -87.640
(.0010) (.0020) (0.0475) (0.706) (0.025)
.0957 .0117 .153 -41.40 0 5.172 0 0 -0.177 -85.498
(.0010) (.0017) (0.699) (0.025)

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
 VARIANCE HETEROGENEITY 337
-2.8
-3 2
-36 6
-4 0
-2
44 0
-4.8 0 2
- 5.2
-5.6 00 0
-6.0 2o
-64 0
-68 oo\
-72

-80 o
-8.4
-8.8 0 ?
-9.6
-10.0 0
-10 4
-10 8 r__ _ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _ _ _ _
75 10.0 12 5 15.0 17 5 20.0 22 5

Fig. 1. Squared residualsfrommean model t 1/3 = /o + f3H + /2D withconstantdispersion,and fittedquadratic

regressionfordispersionmodel,plottedon log scale againstD.

4. Discussion
The macrosMEAN and VAR do thefitting ofthemeanand dispersionmodelsrespectively.
The model formulaehave to be specifiedby the user in macros MODM and MODV; these
macrosallow foroffsetswhichhave to be assignedvalues by the user.If no offsets
are to be
zero values mustbe specified:
fitted,
$ CALC OFSM = OFSV = 0 $
The responsevariableto be modelledis specifiedas an argumentto the VMOD macrowhich
does theoverallanalysis.VMOD calls an initialisingmacroSETUP whichsetsthemaximum
numberof iterations% N to 15 and definestheweightvariateforthefirstfit.VMOD passes
the y-variatename to a macro DRIVER which alternatesMEAN and VAR macros and
calculatesthe deviance at each iteration.In VAR the gamma fithas CYCLE set to 20 as
convergencemay take morethan 10 cycles.Iterationscease whensuccessivedeviancesdiffer
by less than0.001 or when 15 iterationshave been performed.Outputis switchedoffduring
iterationsexceptforthe printingof the deviance and iterationnumber.Afterconvergence
outputis switchedon again and one further iterationperformed,
withprintingof parameter
estimatesand standarderrorsfrommean and dispersionmodels.
A listingincludingtheGLIM macrosis givenforthetreedata withthelog V,log H, log D
model in an Appendixto the paper. The output fromach iterationhas been suppressed;
successive deviances are - 79.8937, - 87.4507, - 88.3032, - 88.3067.

5. Acknowledgement
I am grateful
to thereferees
forsuggestionsforimprovingthepaper,and to Gordon Smyth
forextensivediscussionsof computationalissues.This paper was revisedwhileI was visiting
the Departmentof Statistics,Universityof Tel Aviv,to whomI am gratefulforsupport.
References
Aitkinson,A. C. (1982) Regressiondiagnostics,transformations and constructedvariables(withDiscussion).J. R.
Statist.Soc. B, 44, 1-36.
Baker,R. J.and Nelder,J.A. (1978)GeneralizedLinearInteractive
Modelling:Release3. Oxford:NumericalAlgorithms
Group.
Box,G. E. P. and Cox, D. R. (1964) An analysisoftransformations
(withDiscussion).J. R. Statist.Soc. B, 26, 211-252.
Chen,C. F. (1983) Score testsforregressionmodels.J. Amer.Statist.Ass.,78, 158-161.
Cook, R. D. and Weisberg,S. (1983) Diagnosticsforheteroscedasticityin regression.Biometrika, 70, 1-10.
Pregibon,D. (1984) Book reviewof P. McCullagh and J. A. Nelder,"Generalized Linear Models". Ann. Statist.,
12, 1589-1596.
Ryan,T. A., Joiner,B. L. and Ryan,B. F. (1976) MinitabStudentHandbook.NorthScituate,Mass: DuxburyPress.
Smyth,G. K. (1985) Coupled and separable iterationsin nonlinearestimation.Ph.D. Thesis,AustralianNational
University.

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
338 AITKIN

Appendix
$UNIT 31
$DATA D H V
$READ
8.3 70 10.3 8.6 65 10.3 8.8 63 10.2 10.5 72 16.4
10.7 81 18.8 10.8 83 19.7 11.0 66 15.6 11.0 75 18.2
11.1 80 22.6 11.2 75 19.9 11.3 79 24.2 11.4 76 21.0
11.4 76 21.4 11.7 69 21.3 12.0 75 19.1 12.9 '74 22.2
12.9 85 33.8 13.3 86 27.4 13.7 71 25.7 13.8 64 24.9
14.0 78 34.5 14.2 80 31.7 14.5 74 36.3 16.0 '72 38.3
16.3 77 42.6 17.3 81 55.4 17.5 82 55.7 17.9 80 58.3
18.0 80 51.5 18.0 80 51.0 20.6 87 77.0

VARIABLES:
D- DIAMEITER (IN INCHES) OF 31 CHERRY TREES
AT A HEIGHT OF 4.5 FEET FROM THE GROUND

H- HEIGHT IN FEET OF THE TREES

V- VOLUME OF USEABLE WOOD IN CUBIC FEET

!This macro fits the mean regression model withi weights

!1/variance.
$macro mean!
$yvar %1 $error n $1ink i $weight ww $scale 1 $offset ofsm!
$fit *modm $!
$calc e2z=(%yv--%fv)**2!
$endmac!
!
'This macro fi-ts the log-linear variance model
$macro var!
$yvar e2z $error g $scale 2 $weight $link 1 $offset ofsv!
$ca %lp=1 !
$cycle 20$!
$fit 4modv$!
$ca %d=%cu(e2z/%fv+%log(%fv)+%log(2*%pi))$!
$ca ww=l/%fv$!
$endmac!
I
!This is an initialising macro
$macro setup!
$ca %n=15 :%e=1 :ww=1 $endmac!
I
;This macro drives the model fitting,alternating mean and
!variance modelling.
$macro driver!
$arg mean %1$!
$use mean$!
$use var$!
$ca %e=(1-%le(%c-%d,.001)*%ge(%c-%d,-.001)):%n=%n-1:%c=%d $!
$ca %e=%e*%gt(%n,0) $!
$ca %u=15-%n $output %poc $!
$pr 'iteration number' *-1 %u , deviance is ' *6 %d $!
'

$output$!
$endmac!

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions
 VARIANCE HETEROGENEITY 339

!Th-is is the macro you actually use, with argument the

!response variable.
$macro vmod!
$arg driver %1$!
$use setup$!
$0otpu1 t!
$while %e driver $!
$ou1tput %poc $!
$arg mean %1$!
$use mean$!
$pr 'Mean model' $d e$!
$use var$!
$pr 'Variance model,log scale' $d e$ !
$print 'Iteration number' * -1 u ' ,, deviance is'*6 %d*!
$output %poc-!
$endmac!
$calc l.h=%log(h) 1 d=%log(d) lv=%log(v) ld2=ld**2 $
$calc: of sm--ofsv=O $
$macro modm lh+id $endmac
$macro modv ld+ld2 $endmac
$use vmod lv$

it,eration number 5. , deviance is -88.3067

-- model changed
scaled deviance = 30.996
d-f. = 28

Mean model
esltimate s.e. parameter
1 -6.390 0.2526 1
2 1.080 0.06806 LJB
3 1.955 0.02281 LD
scale parameter taken as 1.000

nmodel1changed
scaled deviance = 45.1.6 at cycle 1.1
(I.f. 28

Variance model,log scale

estimate s.e. parameter
1 -179.8 27.88 1
2 133.9 21.79 LD
3 -25.52 4.234 LD2
scale parameter taken as 2.000

It eration number 5. , deviance is -88.3067

? $s t;op

This content downloaded from 64.250.240.78 on Tue, 4 Nov 2014 19:30:42 PM

All use subject to JSTOR Terms and Conditions