1
Logistic Regression
Testing Fit of a Model Using the Hosmer-Lemeshow Test
If our logistic regression model has a single, continuous, explanatory variable, then a test of fit can be constructed by grouping values of the explanatory variable into several intervals and using the Pearson chi-square statistic, which under the null hypothesis will have an approximate chi-square distribution with degrees of freedom equal to the number of groups minus two (the number of parameters in the model).
However, if we have several explanatory variables, some of which are continuous, a test of fit becomes somewhat more difficult. If there are, for instance, two continuous explanatory variables, and we break up the range of values of each of them into several groups, we may find that there are sparse cells in the table. As the number of explanatory variables increases, the problem of sparse cells becomes more apparent.
Hosmer and Lemeshow1 proposed a different approach to grouping, one that does not depend on the number of explanatory variables. They proposed breaking up, not the ranges of values of continuous explanatory variables, but the probabilities estimated from the original, ungrouped data. The data set, of size n, is sorted according to the probabilities estimated from the final logistic regression model. Then the data set is partitioned into several (Hosmer and Lemeshow recommend 10) equal-sized groups. The first cell corresponds to the n/10 observations having the highest estimated probabilities. The next cell corresponds to the n/10 observations having the next highest estimated probabilities, etc. A Pearson-like statistic is constructed based on the observed and expected cell frequencies.
Let Y denote the (binary) response variable. Assume that the final model used q explanatory variables, and thus has q + 1 parameters. Let be the observed value of the response variable for the jth observation in the ith group of the partition, where i = 1, 2, …, g and j = 1, 2, …, ni. Then is the observed frequency in the ith cell of the partition. Let denote the estimated success probability for the jth observation in the ith cell of the partition. Then is the expected cell frequency for the ith cell of the partition. The statistic proposed by Hosmer and Lemeshow is
.
Hosmer and Lemeshow showed that, when the number of distinct patterns of covariate values equals the sample size, the null distribution of HL is approximately chi-square with d.f. = g – 2.
The SAS program below (using the flu shots example) calculates the value of the HL statistic, using g = 10. This program will work to test the fit of any final logistic regression model using the HL statistic. All program output has been suppressed except for the output of the final PROC MEANS, which gives the value of the HL statistic.
The test of fit proceeds as follows, for the final model estimated for the Flu Shot data:
Step 1: H0:
HA:
Step 2: We have n = 159, g = 10, and we choose a = 0.05.
Step 3: The test statistic is HL, as given above, and under the null hypothesis, this statistic has an approximate chi-square distribution with d.f. = g – 2 = 8.
Step 4: We will reject the null hypothesis if HL > .
Step 5: Using SAS output, we find that HL = 9.6376.
Step 6: We fail to reject the null hypothesis at the 0.05 level of significance. We do not have sufficient evidence to conclude that the data do not fit the hypothesized fitted logistic regression model.
SAS Program to Perform Hosmer-Lemeshow Test of Fit for an Estimated Logistic Regression Model, Using Flu Shots Data:
proc format;
value difmt 0 = "No "
1 = "Yes";
value sexfmt 0 = "Female"
1 = "Male ";
;
data one;
input y x1 x2 x3;
label y = "Flu Shot?"
x1 = "Age in Years"
x2 = "Health Awareness Index"
x3 = "Gender"
format y difmt. x3 sexfmt.;
obs = _n_;
cards;
The data set is listed in the appendix.
;
proc logistic data=one noprint;
model y (order=formatted event=LAST) = x1 x2;
output out=parest predicted=probs;
title "Final Multiple Logistic Regression Model";
;
proc sort data=parest;by probs;
;
data two;set parest;
dummy = 1;
;
proc means noprint;
var obs;
output out=sampsize max=size;
;
data temp;set sampsize;
dummy = 1;
;data three;merge two temp;by dummy;
drop dummy;
group = 10;
if _n_ < 0.9*size then group = 9;
if _n_ < 0.8*size then group = 8;
if _n_ < 0.7*size then group = 7;
if _n_ < 0.6*size then group = 6;
if _n_ < 0.5*size then group = 5;
if _n_ < 0.4*size then group = 4;
if _n_ < 0.3*size then group = 3;
if _n_ < 0.2*size then group = 2;
if _n_ < 0.1*size then group = 1;
;
data grp1;set three;if group = 1;
;
proc means noprint;
var y probs;
output out=grp1mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 1";
;
data grp2;set three;if group = 2;
;
proc means noprint;
var y probs;
output out=grp2mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 2";
;
data grp3;set three;if group = 3;
;
proc means noprint;
var y probs;
output out=grp3mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 3";
;
data grp4;set three;if group = 4;
;
proc means noprint;
var y probs;
output out=grp4mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 4";
;
data grp5;set three;if group = 5;
;
proc means noprint;
var y probs;
output out=grp5mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 5";
;
data grp6;set three;if group = 6;
;
proc means noprint;
var y probs;
output out=grp6mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 6";
;
data grp7;set three;if group = 7;
;
proc means noprint;
var y probs;
output out=grp7mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 7";
;
data grp8;set three;if group = 8;
;
proc means noprint;
var y probs;
output out=grp8mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 8";
;
data grp9;set three;if group = 9;
;
proc means noprint;
var y probs;
output out=grp9mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 9";
;
data grp10;set three;if group = 10;
;
proc means noprint;
var y probs;
output out=grp10mns sum=celln cellprob n=cellsize;
title "Create data set with cell frequencies";
title2 "And expected frequencies, for group 10";
;
data four;set grp1mns grp2mns grp3mns grp4mns grp5mns grp6mns grp7mns grp8mns grp9mns grp10mns;
numer = (celln - cellprob)**2;
denom = cellprob*(1-(cellprob/cellsize));
ratio = numer/denom;
;
proc means sum;
var ratio;
title "Calculation of Goodness-of-Fit Test Statistic";
title2 "Hosmer and Lemeshow (1980)";
;
run;
Output of HL Program for Flu Shot Model:
Calculation of Goodness-of-Fit Test Statistic
Hosmer and Lemeshow (1980)
The MEANS Procedure
Analysis Variable : ratio
Sum
ƒƒƒƒƒƒƒƒƒƒƒƒ
9.6376501
ƒƒƒƒƒƒƒƒƒƒƒƒ
1 Hosmer, D. W. and Lemeshow, S. (1980). “A goodness-of-fit test for multiple logistic regression model,” Communications in Statistics, Series A, 9, 1043-1069.
Appendix: Flu Shot Data
0 59 52 0
0 61 55 1
1 82 51 0
0 51 70 0
0 53 70 0
0 62 49 1
0 51 69 1
0 70 54 1
0 71 65 1
0 55 58 1
0 58 48 0
0 53 58 1
0 72 65 0
0 56 68 0
0 56 83 0
0 81 68 0
0 62 44 0
0 49 70 0
0 56 69 1
0 50 74 0
0 53 57 0
0 56 64 1
0 56 67 1
0 50 83 1
0 52 48 1
0 52 81 0
0 67 53 1
0 51 61 0
0 70 51 0
0 64 51 0
0 61 65 1
0 53 51 0
0 77 54 1
0 73 64 1
0 67 69 0
0 50 71 0
1 80 38 0
1 75 51 0
0 65 54 1
0 60 59 1
1 68 57 1
0 61 63 0
1 62 48 0
0 53 58 0
0 72 56 0
0 54 59 0
1 59 75 0
0 61 48 0
0 50 79 1
0 48 66 0
0 52 57 1
0 54 68 0
0 62 48 0
0 71 60 0
1 65 63 0
0 49 61 0
0 58 57 0
0 62 69 0
0 69 38 1
1 56 50 1
1 76 45 1
0 51 72 0
0 64 51 0
0 57 62 1
0 51 81 0
0 81 55 1
0 50 77 0
0 64 65 1
0 64 53 1
1 59 49 1
0 53 65 0
0 63 58 0
0 59 60 1
1 70 57 1
0 72 37 0
0 68 49 0
0 75 55 1
0 57 60 0
0 67 57 1
0 59 56 1
0 55 58 0
1 75 64 1
0 66 51 0
0 67 59 0
0 59 61 0
0 78 49 1
0 59 49 0
0 68 55 0
1 59 61 1
0 68 50 1
1 78 47 1
0 55 73 1
1 71 45 1
0 51 45 0
0 65 59 0
0 54 61 1
0 79 52 0
0 64 50 0
1 82 46 1
0 64 67 0
0 70 56 1
1 59 50 0
0 59 56 1
0 63 61 1
0 48 74 0
0 61 78 0
0 51 68 0
0 48 71 0
1 71 58 1
0 51 57 0
0 57 51 1
0 49 74 0
0 67 56 1
0 73 57 0
0 73 65 0
0 56 47 0
0 48 69 1
0 50 71 0
0 50 76 1
0 66 60 1
0 53 75 1
0 50 65 1
1 51 42 0
0 68 66 1
1 72 49 1
0 51 58 1
0 62 61 1
0 60 55 0
0 67 60 1
0 70 54 1
0 55 63 1
0 66 56 0
0 65 59 1
0 84 52 1
0 58 63 0
1 68 57 1
0 51 59 1
0 67 53 1
0 52 67 0
0 68 62 0
0 76 63 1
0 54 62 1
0 50 52 1
0 63 58 0
0 77 49 1
0 60 65 1
0 51 55 0
0 51 60 1
0 66 51 1
0 52 67 0
0 66 64 1
0 56 55 1
0 49 58 0
0 67 66 0
0 57 64 1
0 56 66 0
1 76 22 1
1 68 32 0
1 73 56 1