Mplus FAQ
How can I compute a chi-square test for nested models with the MLR or MLM estimators?

Chi-square difference tests are frequently used to test differences between nested models in confirmatory factor analysis, path analysis and structural equation modeling. Nested models are two models (or more if one is fitting a series of models) that are identical except that one of the models constrains some of the parameters (the null model) and one does not have those constraints (the alternative model). Examples of this include the introduction of a set of dichotomous predictors representing a single nominal (categorical) variable to the model, or a test for differences across groups in a multiple group model. Typically a chi-square difference test involves calculating the difference between the chi-square statistic for the null and alternative models, the resulting statistic is distributed chi-square with degrees of freedom equal to the difference in the degrees of freedom between the two models. However, when a model is run in Mplus using the MLM or MLR estimators, the following warning message is displayed, warning the user that the standard chi-square difference test is not valid:

*   The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used
    for chi-square difference tests.  MLM, MLR and WLSM chi-square difference
    testing is described in the Mplus Technical Appendices at www.statmodel.com.
    See chi-square difference testing in the index of the Mplus User's Guide.

This page shows how to calculate the correct chi-square difference test statistic for models estimated with the MLM or MLR estimator as described on the Mplus website (see http://www.statmodel.com/chidiff.shtml for additional information). For both MLM and MLR the Satorra-Bentler scaled chi-square difference test can be used. An additional test is available to test for differences in nested models that use the MLR estimator; this test is based on the log-likelihoods from nested models. Information on how to calculate the Satorra-Benter scaled chi-square test appears first, followed by information on using the log-likelihood to calculate a difference test. Each section includes both a discussion of the formulae and an example.

Cautions

While the above message appears for a number of estimators, the procedure described below is for use with the MLM and MLR estimators only. If the model uses the MLMV or WLSMV estimator, the difftest command can be used to test for differences across nested models. A Wald test can also be used to test nested models using the model test command. See the Mplus manual for more information on these commands. If you are unsure of what estimator is being used in your model, you can find the estimator listed towards the top of the output file.

Also keep in mind that this type of test is only valid if the models are nested; that is, the models must be the same except that one of the models includes additional constraints on the parameters.

The Satorra-Bentler scaled chi-square difference test

In order to calculate the Satorra-Bentler scaled chi-square difference test, we will need a number of pieces of information.  Below is a list of the information needed, along with the symbol (i.e., letter and number) used to represent each value.

c0 = scaling correction factor for the null model.
c1 = scaling correction factor for the alternative model.

d0 = degrees of freedom for the null model.
d1 = degrees of freedom for the alternative model.

SB0 = Satorra-Bentler scaled chi-square value for the null model.
SB1 = Satorra-Bentler scaled chi-square value for the alternative model.
In order to calculate the test statistic, T, we first need to calculate the value cd:
cd = (d0 * c0 - d1 * c1)/(d0 - d1)
Once we have calculated cd, we can compute:
T = (SB0 * c0 - SB1 * c1)/cd
T is distributed chi-square with degrees of freedom:
df = d0 - d1

Example

Below are two Mplus input files. On the left is the null model, which constrains the paths from x2 to f1 and from x3 to f1 to be zero (using x2@0 x3@0 respectively). On the right is the alternative model which estimates the regression path from x2 to f1 and from x3 to f1.

The null modelThe alternative model
Title: My null model;
  Data:
    File is D:\data\mydata.dat ;
  Variable:
    Names are
      v1 v2 v3 x1 x2 x3;
  Analysis:
      Type = general ;
      estimator = mlr;
  Model:
      f1 by v1 v2 v3;
      f1 on x1 x2@0 x3@0;
Title: My alternative model;
    Data:
    File is D:\data\mydata.dat ;
  Variable:
    Names are
      v1 v2 v3 x1 x2 x3;
  Analysis:
      Type = general ;
      estimator = mlr;
  Model:
      f1 by v1 v2 v3;
      f1 on x1 x2 x3;

Once we have run the model, we can find all the information we need to compute the test statistic in the section of the output labeled "Chi-Square Test of Model Fit." This section of the output appears below (all other output is omitted). Output for the null model appears on the left, and output for the alternative model appears on the right. The information we need appears in bold. Below the output, each of the values highlighted appears on the appropriate line. If your output does not include the scaling correction factors (c0 and c1), instructions for calculating them from other output appears below.

The null modelThe alternative model
Chi-Square Test of Model Fit

   Value                            178.097*
   Degrees of Freedom                     8
   P-Value                           0.0000
   Scaling Correction Factor          1.018
   for MLR
Chi-Square Test of Model Fit

   Value                             35.122*
   Degrees of Freedom                     6
   P-Value                           0.0000
   Scaling Correction Factor          0.958
     for MLR
c0 = 1.018   (scaling correction factor for the null model)
c1 = 0.958   (scaling correction factor for the alternative model)

d0 = 8       (degrees of freedom for the null model)
d1 = 6       (degrees of freedom for the alternative model)

SB0 = 178.097 (the Satorra-Bentler adjusted chi-square value for the null model)
SB1 = 35.122  (the Satorra-Bentler adjusted chi-square value for the alternative model)
Once we have identified and noted all the necessary values in our output, the test statistic (T) is computed in two steps.
cd = (d0 * c0 - d1 * c1)/(d0 - d1)
   = (8 * 1.018 - 6 * 0.958)/(8 - 6) = 1.198

T = (SB0 * c0 - SB1 * c1)/cd
  = (178.097 * 1.018 - 35.122 * 0.958) / 1.198 = 123.25198
The test statistic T is distributed chi-square with df = d0-d1. We can look up the p-value for a chi-square statistic of 123.25, with two degrees of freedom using a table or some other method (chi2(2) = 123.25, p < 0.01).

Calculating the scaling correction factor

If your output does not include the scaling correction factor, you can calculate this value from the unadjusted chi-square and the Satorra-Bentler scaled chi-square statistic. So, we are going to pretend that we have output that contains only the unadjusted chi-square, the Satorra-Bentler scaled chi-square statistic and the degrees of freedom.
T0 = Unadjusted chi-square value for the null model.
T1 = Unadjusted chi-square value for the alternative model.

SB0 = The Satorra-Bentler scaled chi-square for the null model.
SB1 = The Satorra-Bentler scaled chi-square for the alternative model.
Given those values, the scaling correction factors (c0 and c1) can be calculated as shown below. c0 is the scaling correction factor for the null model, and c1 is the scaling correction factor for the alternative model.
c0 = T0/SB0
c1 = T1/SB1

For example, taking the appropriate values from the output:

T0 = 181.303
T1 = 33.647 

SB0 = 178.097
SB1 = 35.122
The scaling correction factors can be calculated:
c0 = 181.303/178.097  = 1.018 
c1 = 33.647/35.122 = 0.958

A test using the log-likelihood

For the MLR estimator there is an additional test for nested models. This test compares the log-likelihoods for the null and alternative models rather than the chi-square values. Below is a list of the information needed, along with the symbol (i.e., letter and number) used to represent each value.
L0 = log-likelihood for the null model.
L1 = log-likelihood for the alternative model.

c0 = scaling correction factor for the null model.
c1 = scaling correction factor for the alternative model.

p0 = number of parameters estimated in the null model.
p1 = number of parameters estimated in the alternative model.
From this information the tests statistic TRd can be calculated (note that cd is calculated first then used to calculate TRd), along with the degrees of freedom:
cd = (p0*c0-p1*c1)/(p0-p1)
TRd = -2*(L0-L1)/cd
df = p1-p0

Example

Below is an example of output from two nested models. The information necessary to calculate this test statistic appears in the sections of output labeled "Loglikelihood" and the section immediately after it labeled "Information Criteria" (all other output has been omitted). The information needed for the calculations appears in bold. Note that the log likelihoods and scaling correction factors identified as H0 should be used, not the values labeled H1. Below the output, each of the values highlighted appears on the appropriate line.

The null modelThe alternative model
Loglikelihood

      H0 Value                       -3063.145    
      H0 Scaling Correction Factor       0.942
        for MLR
      H1 Value                       -2972.450
      H1 Scaling Correction Factor       0.976
        for MLR

Information Criteria

      Number of Free Parameters             10   
      Akaike (AIC)                    6146.290
      Bayesian (BIC)                  6179.273
      Sample-Size Adjusted BIC        6147.592
        (n* = (n + 2) / 24)
Loglikelihood

      H0 Value                       -2989.266    
      H0 Scaling Correction Factor       0.985  
        for MLR
      H1 Value                       -2972.450  
      H1 Scaling Correction Factor       0.976  
        for MLR

Information Criteria

      Number of Free Parameters             12
      Akaike (AIC)                    6002.533
      Bayesian (BIC)                  6042.112
      Sample-Size Adjusted BIC        6004.095
        (n* = (n + 2) / 24)
L0 = -3063.145     (log-likelihood for the null model)
L1 = -2989.266     (log-likelihood for the alternative model)

c0 = 0.942         (scaling correction factor for the null model)
c1 = 0.985         (scaling correction factor for the alternative model)

p0 = 10            (number of parameters estimated in the null model)
p1 = 12            (number of parameters estimated in the alternative model)

Once we have identified all the necessary values from the output, the test statistic (TRd) can be computed in two steps.

cd = (p0 * c0 - p1 * c1)/(p0-p1)
   = (10 * 0.942 - 12 * .985)/(10-12) = 1.2

TRd = -2*(L0 - L1)/cd
TRd = -2*(-3063.145 - [-2989.266])/1.2 = 123.132

df = p1-p0
   = 12-10 = 2
The test statistic TRd is distributed chi-square with df = p1-p0. We can look up the p-value for a chi-square statistic of 123.13167, with two degrees of freedom using a table or some other method (chi2(2) = 123.13167, p < 0.001).

See also

The Mplus website, specifically Chi-Square Difference Testing Using the Satorra-Bentler Scaled Chi-Square.

How to cite this page

Report an error on this page or leave a comment

The content of this web site should not be construed as an endorsement of any particular web site, book, or software product by the University of California.