UCLA Academic Technology Services HomeServicesClassesContactJobs
Stat Computing > SPSS > Web Books > Reg
 Search

SPSS Web Books
Regression with SPSS
Chapter 2 - Regression Diagnostics

Chapter Outline
    2.0 Regression Diagnostics
    2.1 Unusual and Influential data
    2.2 Tests on Normality of Residuals
    2.3 Tests on Nonconstant Error of Variance
    2.4 Tests on Multicollinearity
    2.5 Tests on Nonlinearity
    2.6 Model Specification
    2.7 Issues of Independence
    2.8 Summary
    2.9 For more information

2.0 Regression Diagnostics

In our last chapter, we learned how to do ordinary linear regression with SPSS, concluding with methods for examining the distribution of variables to check for non-normally distributed variables as a first look at checking assumptions in regression.  Without verifying that your data have met the regression assumptions, your results may be misleading.  This chapter will explore how you can use SPSS to test whether your data meet the assumptions of linear regression.  In particular, we will consider the following assumptions.

Additionally, there are issues that can arise during the analysis that, while strictly speaking are not assumptions of regression, are none the less, of great concern to regression analysts.

Many graphical methods and numerical tests have been developed over the years for regression diagnostics and SPSS makes many of these methods easy to access and use. In this chapter, we will explore these methods and show how to verify regression assumptions and detect potential problems using SPSS.

2.1 Unusual and Influential data

A single observation that is substantially different from all other observations can make a large difference in the results of your regression analysis.  If a single observation (or small group of observations) substantially changes your results, you would want to know about this and investigate further.  There are three ways that an observation can be unusual.

Outliers: In linear regression, an outlier is an observation with large residual. In other words, it is an observation whose dependent-variable value is unusual given its values on the predictor variables. An outlier may indicate a sample peculiarity or may indicate a data entry error or other problem.

Leverage: An observation with an extreme value on a predictor variable is called a point with high leverage. Leverage is a measure of how far an independent variable deviates from its mean. These leverage points can have an unusually large effect on the estimate of regression coefficients.

Influence: An observation is said to be influential if removing the observation substantially changes the estimate of coefficients. Influence can be thought of as the product of leverage and outlierness.

How can we identify these three types of observations? Let's look at an example dataset called crime. This dataset  appears in Statistical Methods for Social Sciences, Third Edition by Alan Agresti and Barbara Finlay (Prentice Hall, 1997). The variables are state id (sid), state name (state), violent crimes per 100,000 people (crime), murders per 1,000,000 (murder),  the percent of the population living in metropolitan areas (pctmetro), the percent of the population that is white (pctwhite), percent of population with a high school education or above (pcths), percent of population living under poverty line (poverty), and percent of population that are single parents (single).   Below we read in the file and do some descriptive statistics on these variables.  You can click crime.sav to access this file, or see the Regression with SPSS page to download all of the data files used in this book.

Let's say that we want to predict crime by pctmetro, poverty, and single . That is to say, we want to build a linear regression model between the response variable crime and the independent variables pctmetro, poverty and single. We will first look at the scatter plots of crime against each of the predictor variables before the regression analysis so we will have some ideas about potential problems. We can create a scatterplot matrix of these variables as shown below. 

graph
  /scatterplot(matrix)=crime murder pctmetro pctwhite pcths poverty single .

Matrix of crime murder pctmetro pctwhite pcths poverty single
The graphs of crime with other variables show some potential problems.  In every plot, we see a data point that is far away from the rest of the data points. Let's make individual graphs of crime with pctmetro and poverty and single so we can get a better view of these scatterplots.  We will use BY state(name) to plot the state name instead of a point.

GRAPH /SCATTERPLOT(BIVAR)=pctmetro WITH crime BY state(name) .


Scatter of crime pctmetro 

GRAPH /SCATTERPLOT(BIVAR)=poverty WITH crime BY state(name) .

  Scatter of crime poverty
GRAPH /SCATTERPLOT(BIVAR)=single WITH crime BY state(name) .

Scatter of crime single
All the scatter plots suggest that the observation for state = "dc" is a point that requires extra attention since it stands out away from all of the other points. We will keep it in mind when we do our regression analysis.

Now let's try the regression command predicting crime from pctmetro poverty and single.  We will go step-by-step to identify all the potentially unusual or influential points afterwards. 

Let's examine the standardized residuals as a first means for identifying outliers. Below we use the /residuals=histogram subcommand to request a histogram for the standardized residuals.  As you see, we get the standard output that we got above, as well as a table with information about the smallest and largest residuals, and a histogram of the standardized residuals.  The histogram indicates a couple of extreme residuals worthy of investigation. 

regression
  /dependent crime
  /method=enter pctmetro poverty single
  /residuals=histogram.
Variables Entered/Removed(b)
Model Variables Entered Variables Removed Method
1 SINGLE, PCTMETRO, POVERTY(a) . Enter
a All requested variables entered.
b Dependent Variable: CRIME

Model Summary(b)
Model R R Square Adjusted R Square Std. Error of the Estimate
1 .916(a) .840 .830 182.068
a Predictors: (Constant), SINGLE, PCTMETRO, POVERTY
b Dependent Variable: CRIME

ANOVA(b)
Model Sum of Squares df Mean Square F Sig.
1 Regression 8170480.211 3 2723493.404 82.160 .000(a)
Residual 1557994.534 47 33148.820

Total 9728474.745 50


a Predictors: (Constant), SINGLE, PCTMETRO, POVERTY
b Dependent Variable: CRIME



Coefficients(a)

Unstandardized Coefficients Standardized Coefficients t Sig.
Model B Std. Error Beta
1 (Constant) -1666.436 147.852
-11.271 .000
PCTMETRO 7.829 1.255 .390 6.240 .000
POVERTY 17.680 6.941 .184 2.547 .014
SINGLE 132.408 15.503 .637 8.541 .000
a Dependent Variable: CRIME

Residuals Statistics(a)

Minimum Maximum Mean Std. Deviation N
Predicted Value -30.51 2509.43 612.84 404.240 51
Residual -523.01 426.11 .00 176.522 51
Std. Predicted Value -1.592 4.692 .000 1.000 51
Std. Residual -2.873 2.340 .000 .970 51
a Dependent Variable: CRIME

*zresid histogram
Let's now request the same kind of information, except for the studentized deleted residual.  The studentized deleted residual is the residual that would be obtained if the regression was re-run omitting that observation from the analysis.  This is useful because some points are so influential that when they are included in the analysis they can pull the regression line close to that observation making it appear as though it is not an outlier -- however when the observation is deleted it then becomes more obvious how outlying it is. To save space, below we show just the output related to the residual analysis.
*sdresid histogram
The histogram shows some possible outliers.  We can use the outliers(sdresid) and id(state) options to request the 10 most extreme values for the studentized deleted residual to be displayed labeled by the state from which the observation originated.  Below we show the output generated by this option, omitting all of the rest of the output to save space.  You can see that "dc" has the largest value (3.766) followed by "ms" (-3.571) and "fl" (2.620). 
regression
  /dependent crime
  /method=enter pctmetro poverty single
  /residuals=histogram(sdresid) id(state) outliers(sdresid).
Outlier Statistics(a)

Case Number STATE Statistic
Stud. Deleted Residual 1 51 dc 3.766
2 25 ms -3.571
3 9 fl 2.620
4 18 la -1.839
5 39 ri -1.686
6 12 ia 1.590
7 47 wa -1.304
8 13 id 1.293
9 14 il 1.152
10 35 oh -1.148
a Dependent Variable: CRIME

We can use the /casewise subcommand below to request a display of all observations where the sdresid exceeds 2.  To save space, we show just the new output generated by the /casewise subcommand. This shows us that Florida, Mississippi and Washington DC have sdresid values exceeding 2.

Now let's look at the leverage values to identify observations that will have potential great influence on regression coefficient estimates.  We can include lever with the histogram( )  and the outliers( ) options to get more information about observations with high leverage.  We show just the new output generated by these additional subcommands below.  Generally, a point with leverage greater than (2k+2)/n should be carefully examined. Here k is the number of predictors and n is the number of observations, so a value exceeding (2*3+2)/51 = .1568 would be worthy of further investigation.  As you see, there are 4 observations that have leverage values higher than .1568.


   
*lever histogram  

As we have seen, DC is an observation that both has a large residual and large leverage.  Such points are potentially the most influential.  We can make a plot that shows the leverage by the residual and look for observations that are high in leverage and have a high residual.  We can do this using the /scatterplot subcommand as shown below.  This is a quick way of checking potential influential observations and outliers at the same time. Both types of points are of great concern for us.  As we see, "dc" is both a high residual and high leverage point, and "ms" has an extremely negative residual but does not have such a high leverage. 

regression
  /dependent crime
  /method=enter pctmetro poverty single
  /residuals=histogram(sdresid lever) id(state) outliers(sdresid, lever)
  /casewise=plot(sdresid)  outliers(2)
  /scatterplot(*lever, *sdresid).
  *lever by *sdresid scatterplot

Now let's move on to overall measures of influence, specifically let's look at Cook's D, which combines information on the residual and leverage. The lowest value that Cook's D can assume is zero, and the higher the Cook's D is, the more influential the point is. The conventional cut-off point is 4/n, or in this case 4/51 or .078. Below we add the cook keyword to the outliers( ) option and also on the /casewise subcommand and below we see that for the 3 outliers flagged in the "Casewise Diagnostics" table, the value of Cook's D exceeds this cutoff.  And, in the "Outlier Statistics" table, we see that "dc", "ms", "fl" and "la" are the 4 states that exceed this cutoff, all others falling below this threshold.

Cook's D can be thought of as a general measure of influence.  You can also consider more specific measures of influence that assess how each coefficient is changed by including the observation. Imagine that you compute the regression coefficients for the regression model with a particular case excluded, then recompute the model with the case included, and you observe the change in the regression coefficients due to including that case in the model.  This measure is called DFBETA and a DFBETA value can be computed for each observation for each predictor. As shown below, we use the /save sdbeta(sdbf) subcommand to save the DFBETA values for each of the predictors. This saves 4 variables into the current data file, sdfb1, sdfb2, sdfb3 and sdfb4, corresponding to the DFBETA for the Intercept and for pctmetro, poverty and for single, respectively.  We could replace sdfb with anything we like, and the variables created would start with the prefix that we provide.

The /save sdbeta(sdfb) subcommand does not produce any new output, but we can see the variables it created for the first 10 cases using the list command below.  For example, by including the case for "ak" in the regression analysis (as compared to excluding this case), the coefficient for pctmetro would decrease by -.106 standard errors.  Likewise, by including the case for "ak" the coefficient for poverty decreases by -.131 standard errors, and the coefficient for single increases by .145 standard errors (as compared to a model excluding "ak"). Since the inclusion of an observation could either contribute to an increase or decrease in a regression coefficient, DFBETAs can be either positive or negative.  A DFBETA value in excess of  2/sqrt(n) merits further investigation. In this example, we would be concerned about absolute values in excess of 2/sqrt(51) or .28. 

list
  /variables state sdfb1 sdfb2 sdfb3
  /cases from 1 to 10.
STATE       SDFB1       SDFB2       SDFB3

ak        -.10618     -.13134      .14518
al         .01243      .05529     -.02751
ar        -.06875      .17535     -.10526
az        -.09476     -.03088      .00124
ca         .01264      .00880     -.00364
co        -.03705      .19393     -.13846
ct        -.12016      .07446      .03017
de         .00558     -.01143      .00519
fl         .64175      .59593     -.56060
ga         .03171      .06426     -.09120

Number of cases read:  10    Number of cases listed:  10

We can plot all three DFBETA values for the 3 coefficients against the state id in one graph shown below to help us see potentially troublesome observations.  We see changed the value labels for sdfb1 sdfb2 and sdfb3 so they would be shorter and more clearly labeled in the graph.  We can see that the DFBETA for single for "dc" is about 3, indicating that by including "dc" in the regression model, the coefficient for single is 3 standard errors larger than it would have been if "dc" had been omitted.  This is yet another bit of evidence that the observation for "dc" is very problematic. 

VARIABLE LABLES sdfb1 "Sdfbeta pctmetro"
                              /sdfb2 "Sdfbeta poverty"
                              /sdfb3 "Sdfbeta single" .

GRAPH
  /SCATTERPLOT(OVERLAY)=sid sid sid  WITH sdfb1 sdfb2 sdfb3 (PAIR) BY state(name)
  /MISSING=LISTWISE .


Scatter of sdfb1 sdfb2 sdfb3 sid sid sid
The following table summarizes the general rules of thumb we use for the measures we have discussed for identifying observations worthy of further investigation (where k is the number of predictors and n is the number of observations).

Measure

 Value
leverage >(2k+2)/n
abs(rstu) > 2
Cook's D > 4/n
abs(DFBETA) > 2/sqrt(n)

We have shown a few examples of the variables that you can refer to in the /residuals  , /casewise,  /scatterplot and /save sdbeta(  ) subcommands. Here is a list of all of the variables that can be used on these subcommands; however, not all variables can be used on each subcommand.

In addition to the numerical measures we have shown above, there are also several graphs that can be used to search for unusual and influential observations. The  partial-regression plot is very useful in identifying influential points. For example below we add the /partialplot subcommand to produce partial-regression plots for all of the predictors.  For example, in the 3rd plot below you can see the partial-regression plot showing crime by single after both crime and single have been adjusted for all other predictors in the model.   The line plotted has the same slope as the coefficient for single.   This plot shows how the observation for DC influences the coefficient.  You can see how the regression line is tugged upwards trying to fit through the extreme value of DC.  Alaska and West Virginia may also exert substantial leverage on the coefficient of single as well.  These plots are useful for seeing how a single point may be influencing the regression line, while taking other variables in the model into account.  

Note that the regression line is not automatically produced in the graph.  We double clicked on the graph, and then chose "Chart" and the "Options" and then chose "Fit Line Total" to add a regression line to each of the graphs below.

 


 

DC has appeared as an outlier as well as an influential point in every analysis.   Since DC is really not a state, we can use this to justify omitting it from the analysis saying that we really wish to just analyze states. First, let's repeat our analysis including DC below.

Now, let's run the analysis omitting DC by using the filter command to omit "dc" from the analysis.  As we expect, deleting DC made a large change in the coefficient for single .The coefficient for single dropped from 132.4 to 89.4.  After having deleted DC, we would repeat the process we have illustrated in this section to search for any other outlying and influential observations.

compute filtvar = (state NE "dc").
filter by filtvar.
regression
  /dependent crime
  /method=enter pctmetro poverty single .

<some output omitted to save space>



Coefficients(a)

Unstandardized Coefficients Standardized Coefficients t Sig.
Model B Std. Error Beta
1 (Constant) -1197.538 180.487
-6.635 .000
PCTMETRO 7.712 1.109 .565 6.953 .000
POVERTY 18.283 6.136 .265 2.980 .005
SINGLE 89.401 17.836 .446 5.012 .000
a Dependent Variable: CRIME

Summary  

In this section, we explored a number of methods of identifying outliers and influential points. In a typical analysis, you would probably use only some of these methods. Generally speaking, there are two types of methods for assessing outliers: statistics such as residuals, leverage, and Cook's D, that assess the overall impact of an observation on the regression results, and statistics such as DFBETA that assess the specific impact of an observation on the regression coefficients. In our example, we found out that  DC was a point of major concern. We performed a regression with it and without it and the regression equations were very different. We can justify removing it from our analysis by reasoning that our model is to predict crime rate for states not for metropolitan areas.

2.2 Tests for Normality of Residuals

One of the assumptions of linear regression analysis is that the residuals are normally distributed. It is important to meet this assumption for the p-values for the t-tests to be valid. Let's use the elemapi2 data file we saw in Chapter 1 for these analyses.   Let's predict academic performance (api00) from percent receiving free meals (meals), percent of English language learners (ell), and percent of teachers with emergency credentials (emer).  We then use the /save command to generate residuals.


We now use the examine command to look at the normality of these residuals. All of the results from the examine command suggest that the residuals are normally distributed -- the skewness and kurtosis are near 0, the "tests of normality" are not significant, the histogram looks normal, and the Q-Q plot looks normal.  Based on these results, the residuals from this regression appear to conform to the assumption of being normally distributed.
Histogram
Normal q-q plot
Detrended normal q-q plot
Boxplot  

2.3 Heteroscedasticity

Another assumption of ordinary least squares regression is that the variance of the residuals is homogeneous across levels of the predicted values, also known as homoscedasticity. If the model is well-fitted, there should be no pattern to the residuals plotted against the fitted values. If the variance of the residuals is non-constant then the residual variance is said to be "heteroscedastic." Below we illustrate graphical methods for detecting heteroscedasticity. A commonly used graphical method is to use the residual versus fitted plot to show the residuals versus fitted (predicted) values.  Below we use the /scatterplot subcommand to plot *zresid (standardized residuals) by *pred (the predicted values).  We see that the pattern of the data points is getting a little narrower towards the right end, an indication of mild heteroscedasticity.

 

*zresid by *pred scatterplot

Let's run a model where we include just enroll as a predictor and show the residual vs. predicted plot.  As you can see, this plot shows serious heteroscedasticity.  The variability of the residuals when the predicted value is around 700 is much larger than when the predicted value is 600 or when the predicted value is 500.

*zresid by *pred scatterplot
As we saw in Chapter 1, the variable enroll was skewed considerably to the right, and we found that by taking a log transformation, the transformed variable was more normally distributed. Below we transform enroll, run the regression and show the residual versus fitted plot. The distribution of the residuals is much improved.  Certainly, this is not a perfect distribution of residuals, but it is much better than the distribution with the untransformed variable.


*zresid by *pred scatterplot  

Finally, let's revisit the model we used at the start of this section, predicting api00 from meals, ell and emer.  Using this model, the distribution of the residuals looked very nice and even across the fitted values.   What if we add enroll to this model.  Will this automatically ruin the distribution of the residuals?  Let's add it and see.