Stata FAQ
How can I estimate effect size for xtmixed?

NOTE:  Code for this page was tested in Stata 12.

This page is will show one method for estimating effects size for xtmixed models. Specifically, we will estimate Cohen's f2 effect size measure. The definitional formula for Cohen's f2 is the following.


         R2
f2 = ----------
       1 - R2
Here is a generic version of the formula for multilevel mixed models to estimte f2b.

       R2ab - R2a
f2b = ------------
       1 - R2ab
In this formula a and b each stands for one or more variables. When you subtract R2a from R2ab you are left with the proportion of variability due to b. For our example, we will use the hsbdemo dataset with read being the a and female being the b. We begin by running the xtmixed model with write as the response variable.
use http://www.ats.ucla.edu/stat/data/hsbdemo, clear

xtmixed write read female || cid:, var

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -627.40316  
Iteration 1:   log likelihood = -627.40316  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =       200
Group variable: cid                             Number of groups   =        20

                                                Obs per group: min =         7
                                                               avg =      10.0
                                                               max =        12


                                                Wald chi2(2)       =     43.68
Log likelihood = -627.40316                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |  -.1843358   .0640048    -2.88   0.004     -.309783   -.0588886
      female |   3.903391   .6984429     5.59   0.000     2.534469    5.272314
       _cons |   60.41435   4.006324    15.08   0.000      52.5621     68.2666
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
cid: Identity                |
                  var(_cons) |   86.06628   29.98087      43.48306    170.3515
-----------------------------+------------------------------------------------
               var(Residual) |   21.43537   2.278678      17.40381    26.40083
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    95.62 Prob >= chibar2 = 0.0000
As you can see both female and read are statistically significant. But what is the estimate of the effect size for each of these two variables? We begin by capturing the residual error variance component into a global macro called Vab. The reason that the expression looks a little strange is that Stata actually estimates the natural log of the standard deviation. To get the variance we need to exponentiate it estimate and then square it.
global Vab = exp(_b[lnsig_e:_cons])^2
Next we need to get the residual error variance for the null model, e.g., a model with no predictors. We will store this variance in the global macro Vnull.
xtmixed write || cid:, var

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood =  -645.9126  
Iteration 1:   log likelihood =  -645.9126  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =       200
Group variable: cid                             Number of groups   =        20

                                                Obs per group: min =         7
                                                               avg =      10.0
                                                               max =        12


                                                Wald chi2(0)       =         .
Log likelihood =  -645.9126                     Prob > chi2        =         .

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _cons |   52.92105   1.785709    29.64   0.000     49.42112    56.42097
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
cid: Identity                |
                  var(_cons) |   61.01018   20.15208      31.93336    116.5628
-----------------------------+------------------------------------------------
               var(Residual) |     27.297   2.877081      22.20235    33.56069
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =   174.36 Prob >= chibar2 = 0.0000

global Vnull = exp(_b[lnsig_e:_cons])^2
To obtain the f2 effect size for female we need one more residual error variance. We will run the model with read as the predictor, storing the residual error variance in Va.
xtmixed write read || cid:, var

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -641.89541  
Iteration 1:   log likelihood = -641.89541  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =       200
Group variable: cid                             Number of groups   =        20

                                                Obs per group: min =         7
                                                               avg =      10.0
                                                               max =        12


                                                Wald chi2(1)       =     10.23
Log likelihood = -641.89541                     Prob > chi2        =    0.0014

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |  -.2189082   .0684415    -3.20   0.001    -.3530511   -.0847653
       _cons |   64.37154   4.194607    15.35   0.000     56.15026    72.59282
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
cid: Identity                |
                  var(_cons) |   93.18282   32.61012         46.93     185.021
-----------------------------+------------------------------------------------
               var(Residual) |   24.95401   2.653639      20.25921    30.73676
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    94.34 Prob >= chibar2 = 0.0000

global Va = exp(_b[lnsig_e:_cons])^2
Next, we compute two R-squared values, one for the model with both read and female, and another R-squared for the model with just read.
global R2ab = ($Vnull - $Vab)/$Vnull
global R2a  = ($Vnull - $Va)/$Vnull
Finally, we can compute the f2 estimate for female using the formula from the introduction.
global f2b  = ($R2ab - $R2a)/(1 - $R2ab)

display "effect size for female: $f2b" 

effect size for female: .1641510709933181
We see that the f2 effects size for female is .1642. Now to compute the effect size for read we will need to compute Vb and R2b. After which, we can compute f2b.
xtmixed write female || cid:, var

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -630.71368  
Iteration 1:   log likelihood = -630.71368  

Computing standard errors:

Mixed-effects ML regression                     Number of obs      =       200
Group variable: cid                             Number of groups   =        20

                                                Obs per group: min =         7
                                                               avg =      10.0
                                                               max =        12


                                                Wald chi2(1)       =     33.01
Log likelihood = -630.71368                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      female |   4.139693   .7205186     5.75   0.000     2.727502    5.551883
       _cons |   50.64262   1.801699    28.11   0.000     47.11136    54.17389
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
cid: Identity                |
                  var(_cons) |   59.43055   19.52269      31.21718    113.1425
-----------------------------+------------------------------------------------
               var(Residual) |   23.13721   2.438688      18.81886    28.44648
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =   191.15 Prob >= chibar2 = 0.0000

global Vb = exp(_b[lnsig_e:_cons])^2

global R2b  = ($Vnull - $Vb)/$Vnull

global f2a  = ($R2ab - $R2b)/(1 - $R2ab)

display "effect size for read: $f2a" 

effect size for read: .0793938292686101
The f2 effect size for read is .0794.

This page is based on the following paper: Selya AS, Rose JS, Dierker LC, Hedeker D, Mermelstein RJ. A Practical Guide to Calculating Cohenís f2, a Measure of Local Effect Size, from PROC MIXED. Frontiers in Psychology 2012.

How to cite this page

Report an error on this page or leave a comment

The content of this web site should not be construed as an endorsement of any particular web site, book, or software product by the University of California.