Glm talk Tomas
0 likesā¢3,347 views
The document discusses distributed implementations of generalized linear models (GLMs) on the H2O platform. GLMs generalize linear regression by adding a link function to transform the response variable and allow the noise variance to vary. The H2O implementation solves GLMs using an inner-outer loop approach, with the inner loop using an alternating direction method of multipliers solver and the outer loop averaging results across nodes. Regularization is added through elastic net penalties to avoid overfitting and obtain sparse solutions.
![H2O Implementation
Outer Loop:
(Map Reduce Task)
public class SimpleGLM extends MRTask {
@Override public void map(Chunk c) {
res = new double [p][p];
for(double [] x:c.rows()){
double eta,mu,var;
eta = computeEta(x);
mu = _link.linkInv(eta);
var = Math.max(1e-5,_family.variance(mu));
double dp = _link.linkInvDeriv(eta);
double w = dp*dp/var;
for(int i = 0; i < x.length; ++i)
for(int j = 0; j < x.length; ++j)
res[i][j] += x[i]*x[j]*w;
}
}
@Override public void reduce(SimpleGLM g) {
for(int i = 0; i < res.length; ++i)
for(int j = 0; i < res.length; ++i)
res[i][j] += g.res[i][j];
}
}
Inner Loop:
(ADMM solver)
public double [] solve(Matrix xx, Matrix xy) {
// ADMM LSM Solve
CholeskyDecomposition lu; // cache decomp!
lu = new CholeskyDecomposition(xx);
for( int i = 0; i < 1000; ++i ) {
// Solve using cached Cholesky decomposition!
xm = lu.solve(xyPrime);
// compute u and z update
for( int j = 0; j < N-1; ++j ) {
double x_hat = xm.get(j, 0);
x_norm += x_hat * x_hat;
double zold = z[j];
z[j] = shrinkage(x_hat + u[j], kappa);
u[j] += x_hat - z[j];
u_norm += u[j] * u[j];
}
}
}
double shrinkage(double x, double k) {
return Math.max(0,x-k)-Math.max(0,-x-k);
}](https://image.slidesharecdn.com/glmtalk2-130615114839-phpapp01/85/Glm-talk-Tomas-9-320.jpg)
Glm talk Tomas
- 1. Distributed GLM Implementation on H2O platform Tomas Nykodym, 0xDATA
- 2. Linear Regression Data: x, y + noise Goal: predict y using x i.e. find a,b s.t. y = a*x + b
- 3. Linear Regression Least Squares Fit Real Relation: y=3x+10+N(0,20) Best Fit: y = 3.08*x + 6
- 4. Prostate Cancer Example Data: x = PSA (prostate-specific antigen) y = CAPSULE 0 = no tumour 1 = tumour Goal: predict y using x
- 5. Prostate Cancer Example Linear Regression Fit Data: x = PSA (prostate-specific antigen) y = CAPSULE 0 = no tumour 1 = tumour Fit: Least squares fit
- 6. Generalized Linear Model Generalizes linear regression by: ā adding a link function g to transform the output z = g(y) ā new response variable ā noise (i.e.variance) does not have to be constant ā fit is maximal likelihood instead of least squares
- 7. Prostate Cancer Logistic Regression Fit Data: x = PSA (prostate-specific antigen) y = CAPSULE 0 = no tumour 1 = tumour GLM Fit: ā Binomial family ā Logit link ā Predict probability of CAPSULE=1.
- 8. Implementation - Solve GLM by IRLSM Input: ā X: data matrix N*P ā Y: response vector (N rows) ā family, link function, Ī±,Ī² INNER LOOP: Solve elastic net: ADMM(Boyd 2010, page 43): OUTER LOOP: While Ī² changes, compute: zk +1=Ī² k +( yāĪ¼ k ) d Ī· d Ī¼ W k +1 ā1 =( d Ī· d Ī¼ ) 2 Var(Ī¼ k ) Ī³ l+1 =( X T WX +Ļ I ) ā1 X T Wz+Ļ (Ī² l āu l ) Ī² l+1 =SĪ» /Ļ (Ī³ l+1 +ul ) ul+1 =uk +Ī³ l+1 āĪ²l+1 Output: ā Ī² vector of coefficients, solution to max-likellihood XX = X T W k+1 X Xz= X T Wzk +1
- 9. H2O Implementation Outer Loop: (Map Reduce Task) public class SimpleGLM extends MRTask { @Override public void map(Chunk c) { res = new double [p][p]; for(double [] x:c.rows()){ double eta,mu,var; eta = computeEta(x); mu = _link.linkInv(eta); var = Math.max(1e-5,_family.variance(mu)); double dp = _link.linkInvDeriv(eta); double w = dp*dp/var; for(int i = 0; i < x.length; ++i) for(int j = 0; j < x.length; ++j) res[i][j] += x[i]*x[j]*w; } } @Override public void reduce(SimpleGLM g) { for(int i = 0; i < res.length; ++i) for(int j = 0; i < res.length; ++i) res[i][j] += g.res[i][j]; } } Inner Loop: (ADMM solver) public double [] solve(Matrix xx, Matrix xy) { // ADMM LSM Solve CholeskyDecomposition lu; // cache decomp! lu = new CholeskyDecomposition(xx); for( int i = 0; i < 1000; ++i ) { // Solve using cached Cholesky decomposition! xm = lu.solve(xyPrime); // compute u and z update for( int j = 0; j < N-1; ++j ) { double x_hat = xm.get(j, 0); x_norm += x_hat * x_hat; double zold = z[j]; z[j] = shrinkage(x_hat + u[j], kappa); u[j] += x_hat - z[j]; u_norm += u[j] * u[j]; } } } double shrinkage(double x, double k) { return Math.max(0,x-k)-Math.max(0,-x-k); }
- 10. Regularization Elastic Net (Zhou, Hastie, 2005): ā Added L1 and L2 penalty to Ī² to: ā avoid overfitting, reduce variance ā obtain sparse solution (L1 penalty) ā avoid problems with correlated covariates No longer analytical solution. Options: LARS, ADMM, Generalized Gradient, ... Ī² =argmin( X Ī² ā y) T ( X Ī² āy)+Ī± ā„Ī²ā„1+(1āĪ± )ā„Ī²ā„2 2
- 11. Linear Regression Least Squares Method Find Ī² by minimizing the sum of squared errors: Analytical solution: Easily parallelized if XT X is reasonably small. Ī² =(X T X )ā1 X T y=( 1 n ā xi xi T ) ā1 1 n ā xi y Ī² =argmin( X Ī² ā y) T ( X Ī² āy)
- 12. Generalized Linear Model ā Generalizes linear regression by: ā adding a link function g to transform the response z = g(y) ā new response variable Ī· = XĪ² ā linear predictor Ī¼ = g-1 (Ī·) ā y has a distribution in the exponential family ā variance depends on Ī¼ e.g var(Ī¼) = Ī¼*(1-Ī¼) for Binomial family. ā fit by maximizing the likelihood of the model