Stat Computing > Seminars >
Intro Survey
|
|
|
||||
|
Stat Computing > Seminars >
Intro Survey
|
|
||||
The purpose of this seminar is to introduce you to the use of Stata, SUDAAN, WesVar and SAS for the analysis of survey data. It will draw much of its materials and examples from Choosing the Correct Analysis for Various Survey Designs.
Regular statistical software (that is not designed for survey data) analyzes data as if the data were collected using simple random sampling. For experimental and quasi-experimental designs, this is exactly what we want. However, when surveys are conducted, a simple random sample is rarely collected. Not only is it nearly impossible to do so, but it is not as efficient (both financially and statistically) as other sampling methods. When any sampling method other than simple random sampling is used, we need to use survey data analysis software to take into account the differences between the design that was used and simple random sampling. The sampling design affects the calculation of the standard errors of the estimates. If you ignore the sampling design, e.g., if you assume simple random sampling when another type of sampling design was used, the standard errors will likely be underestimated, possibly leading to results that seem to be statistically significant, when in fact, they are not. The difference in point estimates and standard errors obtained using non-survey software and survey software with the design properly specified will vary from data set to data set, and even between variables within the same data set. While it may be possible to get reasonably accurate results using non-survey software, there is no practical way to know beforehand how far off the results from non-survey software will be.
Most people do not conduct their own surveys. Rather, they use survey data that some agency or company collected and made available to the public. The documentation must be read carefully to find out what kind of sampling design was used to collect the data. This is very important because many of the estimates and standard errors are calculated differently for the different sampling designs. Hence, if you mis-specify the sampling design, the point estimates and standard errors will likely be wrong.
Below are some common features of many sampling designs.
Weights: There are many types of weights that can be associated with a survey. Perhaps the most common is the sampling weight, sometimes called a pweight, which is used to denote the inverse of the probability of being included in the sample due to the sampling design (except for a certainty PSU, see below). The pweight is calculated as N/n, where N = the number of elements in the population and n = the number of elements in the sample. For example, if a population has 10 elements and 3 are sampled at random with replacement, then the pweight would be 10/3 = 3.33. The sum of the pweights should equal the population total.
PSU: This is the primary sampling unit. This is the first unit that is sampled in the design. For example, school districts from California may be sampled and then schools within districts may be sampled. The school district would be the PSU. If states from the US were sampled, and then school districts from within each state, and then schools from within each district, then states would be the PSU. One does not need to use the same sampling method at all levels of sampling. For example, probability-proportional-to-size sampling may be used at level 1 (to select states), while cluster sampling is used at level 2 (to select school districts). In the case of a simple random sample, the PSUs and the elementary units are the same.
Strata: Stratification is a method of breaking up the population into different groups, often by demographic variables such as gender, race or SES. Once these groups have been defined, one samples from each group as if it were independent of all of the other groups. For example, if a sample is to be stratified on gender, men and women would be sampled independent of one another. This means that the pweights for men will likely be different from the pweights for the women. In most cases, you need to have two or more PSUs in each stratum. The purpose of stratification is to improve the precision of the estimates.
FPC: This is the finite population correction. This is used when the sampling fraction (the number of elements or respondents sampled relative to the population) becomes large. The FPC is used in the calculation of the standard error of the estimate. If the value of the FPC is close to 1, it will have little impact and can be safely ignored. In some survey data analysis programs, such as SUDAAN, this information will be needed if you specify that the data were collected without replacement (see below for a definition of "without replacement"). To see the impact of the FPC for samples of various proportions, suppose that you had a population of 10,000 elements.
Sample size (n) FPC
1
1.0
10
.9995
100
.9950
500
.9747
1000
.9487
5000
.7071
9000
.3162
Imputation flag: This is a 0/1 variable that is associated with a variable in the data set and indicates whether the corresponding value in the associated variable was imputed or given by the respondent. For example, in the data set below
Subject Response ImputeFlag
1
60 0
2 60
1
3 63
0
the data for subject number 2 was imputed. The flag does not tell you how the imputation was done (i.e., mean substitution, multiple imputation, etc.). These variables are useful for determining how much missing data each variable has.
Non-response weight: There are both unit and item non-response weights. The former down-weights an entire case because the respondent did not respond to any of the items on the survey (perhaps he wasn't home that day). The later down-weights "responses" from respondents who did not answer that item.
Certainty PSU: This is a PSU that was guaranteed to be in the sample. This is independent of the sampling design: any sampling design can have one or more certainty PSUs. Certainty PSUs are also called self-representing units.
Poststratification: This is stratification that happens after the sample has been collected, either because the information needed to do stratification was not available when the sample was collected, or because it was not known at the time of data collection that stratification on this variable would be necessary/desirable. The purpose of poststratification is to improve the precision of the estimates or to reduce bias caused by non-response.
Clearly, not all surveys will have all of the features listed above. We will concentrate only on the first four features because they are the most common.
Most samples collected in the real world are collected "without replacement". This means that once a respondent has been selected to be in the sample and has participated in the survey, that particular respondent cannot be selected again to be in the sample. Many of the calculations change depending on if a sample is collected with or without replacement. Hence, programs like SUDAAN request that you specify if a survey sampling design was implemented with our without replacement, and an FPC is used if sampling without replacement is used, even if the value of the FPC is very close to one.
Replicate weights are a feature of an increasing number of public use survey data sets. Replicate weights are a series of weight variables that are used instead of PSUs and strata in an effort to protect the respondents' identity. Either replicate weights or a Taylor series linearization, which is bases on PSUs and/or strata, are necessary for variance estimation.
We are now going to summarize some of the features of four survey data analysis packages: Stata, SUDAAN, WesVar and SAS. On feature that all four programs share is that once you specify the sampling design, it is either 1) applied to all analyses until you change it or exit the program (Stata and WesVar) or 2) very easy to apply to all analyses (SUDAAN and SAS). In other words, you only need to go through the work of specifying the design once, and then it applies to all analyses of that data.
Stata:
SUDAAN:
WesVar:
SAS:
Now we are going to try some analyses using the different packages. All of the examples shown below were presented in Levy and Lemeshow's Sampling of Populations. These and other examples from that text and other texts can be found on our website. We will focus on Stata and SUDAAN. If there is time, we will show an example in WesVar. The code to do these examples in SAS is given at the end of this handout, along with some explanation.
The entire Stata version 7 program can be downloaded here. The entire Stata version 8 program can be downloaded here. We will use the Stata version 7 code for this seminar.
Although simple random sampling (SRS) is almost never used, we start with this example because it is the least complex and it will serve as a comparison for later examples. Note that in SRS sampling, each observation is a PSU. The Stata code that would be used with a SRS design is given below.
use http://www.ats.ucla.edu/stat/books/sop/momsag.dta, clear list birth weight1 momsag in 1/10
birth weight1 momsag 1. 773 30.92 0 2. 773 30.92 1 3. 773 30.92 1 4. 773 30.92 1 5. 773 30.92 1 6. 773 30.92 1 7. 773 30.92 1 8. 773 30.92 1 9. 773 30.92 1 10. 773 30.92 1
svyset pweight weight1
svyset fpc birth
svymean momsag
Survey mean estimation
pweight: weight1 Number of obs = 25
Strata: <one> Number of strata = 1
PSU: <observations> Number of PSUs = 25
FPC: birth Population size = 773
------------------------------------------------------------------------------
Mean | Estimate Std. Err. [95% Conf. Interval] Deff
---------+--------------------------------------------------------------------
momsag | .92 .0544746 .8075699 1.03243 1
------------------------------------------------------------------------------
Finite population correction (FPC) assumes simple random sampling without
replacement of PSUs within each stratum with no subsampling within PSUs.
Weights must represent population totals for deff to be correct when
using an FPC. Note: deft is invariant to the scale of weights.
Now let's see what happens when you ignore the sampling design. We will clear the survey sets from Stata and use the ci command to get the mean, standard error and confidence interval. Hence, in this analysis, the pweight and the fpc are ignored. We will use the svyset command with no options to check that no variables have be set.
* PLEASE REMEMBER THAT THE ANALYSIS BELOW IS INCORRECT!! svyset, clear svyset no variables have been set ci momsag
Variable | Obs Mean Std. Err. [95% Conf. Interval]
-------------+-------------------------------------------------------------
momsag | 25 .92 .0553775 .8057065 1.034294
As you can see, the mean is the same as that obtained when including the sampling design information. However, the standard error is larger. If we multiply the standard error by the square root of the fpc, we will obtain the correct standard error.
display sqrt((773-25)/773)*.0553775
.05447464
svyset pweight weight1 svyset fpc birth svytotal momsag Survey total estimation pweight: weight1 Number of obs = 25 Strata: <one> Number of strata = 1 PSU: <observations> Number of PSUs = 25 FPC: birth Population size = 773 ------------------------------------------------------------------------------ Total | Estimate Std. Err. [95% Conf. Interval] Deff ---------+-------------------------------------------------------------------- momsag | 711.16 42.10889 624.2515 798.0685 1 ------------------------------------------------------------------------------ Finite population correction (FPC) assumes simple random sampling without replacement of PSUs within each stratum with no subsampling within PSUs. Weights must represent population totals for deff to be correct when using an FPC. Note: deft is invariant to the scale of weights.
The difference between the example above and the example below is that stratification has been added.
use http://www.ats.ucla.edu/stat/books/sop/hospsamp.dta, clear list in 1/10
hospno oblevel weighta tothosp births 1. 15 1 10.5 42 480 2. 80 1 10.5 42 426 3. 86 1 10.5 42 342 4. 136 1 10.5 42 174 5. 7 2 19.799988 99 2022 6. 26 2 19.799988 99 576 7. 62 2 19.799988 99 1999 8. 90 2 19.799988 99 482 9. 101 2 19.799988 99 836 10. 28 3 2.8333321 17 3108
svyset pweight weighta
svyset strata oblevel
svyset fpc tothosp
svytotal births
Survey total estimation
pweight: weighta Number of obs = 15
Strata: oblevel Number of strata = 3
PSU: <observations> Number of PSUs = 15
FPC: tothosp Population size = 157.99993
------------------------------------------------------------------------------
Total | Estimate Std. Err. [95% Conf. Interval] Deff
---------+--------------------------------------------------------------------
births | 183982.9 34014.33 109872 258093.8 .7035474
------------------------------------------------------------------------------
Finite population correction (FPC) assumes simple random sampling without
replacement of PSUs within each stratum with no subsampling within PSUs.
Weights must represent population totals for deff to be correct when
using an FPC. Note: deft is invariant to the scale of weights.
svytotal births, by (oblevel)
Survey total estimation
pweight: weighta Number of obs = 15
Strata: oblevel Number of strata = 3
PSU: <observations> Number of PSUs = 15
FPC: tothosp Population size = 157.99993
------------------------------------------------------------------------------
Total Subpop. | Estimate Std. Err. [95% Conf. Interval] Deff
---------------+--------------------------------------------------------------
births |
oblevel==1 | 14931 2669.857 9113.882 20748.12 .15648
oblevel==2 | 117116.9 33067.66 45068.68 189165.2 1.089405
oblevel==3 | 51934.98 7508.399 35575.58 68294.37 .0330073
------------------------------------------------------------------------------
Finite population correction (FPC) assumes simple random sampling without
replacement of PSUs within each stratum with no subsampling within PSUs.
Weights must represent population totals for deff to be correct when
using an FPC. Note: deft is invariant to the scale of weights.
use http://www.ats.ucla.edu/stat/books/sop/tab9_1a.dta, clear list devlpmnt HH wt1 M NVSTNRS NGE65 hhneedvn in 1/10
devlpmnt HH wt1 M NVSTNRS NGE65 hhneedvn 1. 2 1 2.5 5 1 2 1 2. 2 2 2.5 5 0 1 0 3. 2 3 2.5 5 0 2 0 4. 2 4 2.5 5 1 1 1 5. 2 5 2.5 5 0 1 0 6. 2 6 2.5 5 0 1 0 7. 2 7 2.5 5 1 2 1 8. 2 8 2.5 5 1 1 1 9. 2 9 2.5 5 1 3 1 10. 2 10 2.5 5 1 1 1
svyset pweight wt1 svyset fpc M svyset psu devlpmnt svytotal NVSTNRS NGE65 Survey total estimation pweight: wt1 Number of obs = 40 Strata: <one> Number of strata = 1 PSU: devlpmnt Number of PSUs = 2 FPC: M Population size = 100 ------------------------------------------------------------------------------ Total | Estimate Std. Err. [95% Conf. Interval] Deff ---------+-------------------------------------------------------------------- NVSTNRS | 57.5 1.936492 32.89454 82.10546 .0707804 NGE65 | 167.5 1.936492 142.8945 192.1055 .0393542 ------------------------------------------------------------------------------ Finite population correction (FPC) assumes simple random sampling without replacement of PSUs within each stratum with no subsampling within PSUs. Weights must represent population totals for deff to be correct when using an FPC. Note: deft is invariant to the scale of weights.
svymean NVSTNRS hhneedvn
Survey mean estimation
pweight: wt1 Number of obs = 40
Strata: <one> Number of strata = 1
PSU: devlpmnt Number of PSUs = 2
FPC: M Population size = 100
------------------------------------------------------------------------------
Mean | Estimate Std. Err. [95% Conf. Interval] Deff
---------+--------------------------------------------------------------------
NVSTNRS | .575 .0193649 .3289454 .8210546 .0707804
hhneedvn | .525 .0193649 .2789454 .7710546 .0977444
------------------------------------------------------------------------------
Finite population correction (FPC) assumes simple random sampling without
replacement of PSUs within each stratum with no subsampling within PSUs.
Weights must represent population totals for deff to be correct when
using an FPC. Note: deft is invariant to the scale of weights.
svyratio NVSTNRS NGE65
Survey ratio estimation
pweight: wt1 Number of obs = 40
Strata: <one> Number of strata = 1
PSU: devlpmnt Number of PSUs = 2
FPC: M Population size = 100
------------------------------------------------------------------------------
Ratio | Estimate Std. Err. [95% Conf. Interval] Deff
------------------+-----------------------------------------------------------
NVSTNRS/NGE65 | .3432836 .0075924 .2468131 .4397541 .0325067
------------------------------------------------------------------------------
Finite population correction (FPC) assumes simple random sampling without
replacement of PSUs within each stratum with no subsampling within PSUs.
Weights must represent population totals for deff to be correct when
using an FPC. Note: deft is invariant to the scale of weights.
The SAS data files that are used for the SUDAAN and SAS examples can be
downloaded here:
momsag.sas7bdat
hospsamp.sas7bdat
tab9_1c.sas7bdat
The entire SUDAAN program can be downloaded here.
proc descript data = momsag filetype = sas design = wor total ; weight weight1; nest _one_; totcnt birth; var momsag; run; Number of observations read : 25 Weighted count : 773 Denominator degrees of freedom : 24 Variance Estimation Method: Taylor Series (WOR) by: Variable, One. ----------------------------------------------------- | | | | Variable | | One | | | 1 | ----------------------------------------------------- | | | | | MOMSAG | Sample Size | 25 | | | Weighted Size | 773.00 | | | Total | 711.16 | | | SE Total | 42.11 | | | Mean | 0.92 | | | SE Mean | 0.05 | -----------------------------------------------------
proc descript data = hospsamp filetype = sas design = wor totals; nest oblevel; weight weighta; totcnt tothosp; var births; subgroup oblevel; levels 3; setenv decwidth = 3; run;
Number of observations read : 15 Weighted count : 158 Denominator degrees of freedom : 12 Variance Estimation Method: Taylor Series (WOR) by: Variable, OBLEVEL. ------------------------------------------------------------------------------------ | | | | Variable | | OBLEVEL | | | Total | 1 | ------------------------------------------------------------------------------------ | | | | | | BIRTHS | Sample Size | 15.000 | 4.000 | | | Weighted Size | 158.000 | 42.000 | | | Total | 183982.904 | 14931.000 | | | SE Total | 34014.329 | 2669.857 | | | Mean | 1164.449 | 355.500 | | | SE Mean | 215.281 | 63.568 | ------------------------------------------------------------------------------------ Variance Estimation Method: Taylor Series (WOR) by: Variable, OBLEVEL. ------------------------------------------------------------------------------------ | | | | Variable | | OBLEVEL | | | 2 | 3 | ------------------------------------------------------------------------------------ | | | | | | BIRTHS | Sample Size | 5.000 | 6.000 | | | Weighted Size | 99.000 | 17.000 | | | Total | 117116.928 | 51934.977 | | | SE Total | 33067.664 | 7508.399 | | | Mean | 1183.000 | 3055.000 | | | SE Mean | 334.017 | 441.671 | ------------------------------------------------------------------------------------
proc descript data = tab9_1c filetype =sas design = wor means totals; nest _one_ devlpmnt; totcnt m _zero_; weight wt1; var nge65 nvstnrs hhneedvn; run;
Number of observations read : 40 Weighted count : 100 Denominator degrees of freedom : 1 Variance Estimation Method: Taylor Series (WOR) by: Variable, One. ------------------------------------------------------ | | | | Variable | | One | | | 1 | ------------------------------------------------------ | | | | | NGE65 | Sample Size | 40.00000 | | | Weighted Size | 100.00000 | | | Total | 167.50000 | | | SE Total | 1.93649 | | | Mean | 1.67500 | | | SE Mean | 0.01936 | ------------------------------------------------------ | | | | | NVSTNRS | Sample Size | 40.00000 | | | Weighted Size | 100.00000 | | | Total | 57.50000 | | | SE Total | 1.93649 | | | Mean | 0.57500 | | | SE Mean | 0.01936 | ------------------------------------------------------ | | | | | HHNEEDVN | Sample Size | 40.00000 | | | Weighted Size | 100.00000 | | | Total | 52.50000 | | | SE Total | 1.93649 | | | Mean | 0.52500 | | | SE Mean | 0.01936 | ------------------------------------------------------ proc ratio data = tab9_1c filetype = sas design = wor; nest _one_ devlpmnt; totcnt M _zero_; weight wt1; numer nvstnrs; denom nge65; setenv decwidth = 5; run;
Number of observations read : 40 Weighted count : 100 Denominator degrees of freedom : 1 Variance Estimation Method: Taylor Series (WOR) by: Variable, One. ------------------------------------------------------ | | | | Variable | | One | | | 1 | ------------------------------------------------------ | | | | | NVSTNRS/NGE65 | Sample Size | 40.00000 | | | Weighted Size | 100.00000 | | | Weighted X-Sum | 167.50000 | | | Weighted Y-Sum | 57.50000 | | | Ratio Est. | 0.34328 | | | SE Ratio | 0.00759 | ------------------------------------------------------
The entire SAS program can be downloaded here.
proc surveymeans data = momsag n = 773 mean sum std; weight weight1; var momsag; run;
The SURVEYMEANS Procedure
Data Summary
Number of Observations 25
Sum of Weights 773.000002
Statistics
Std Error
Variable Mean of Mean Sum Std Dev
------------------------------------------------------------------------
MOMSAG 0.920000 0.054475 711.160002 42.108894
------------------------------------------------------------------------
data second138; input id _TOTAL_ oblevel; cards; 1 42 1 2 42 1 3 42 1 4 42 1 5 99 2 6 99 2 7 99 2 8 99 2 9 99 2 10 17 3 11 17 3 12 17 3 13 17 3 14 17 3 15 17 3 ; run;
NOTE: You cannot get the totals for both the whole group and the
sub-groups in the same proc surveymeans.
NOTE: The data set second138 is used to tell SAS what the totals
are in each stratum. These totals are used to compute the finite population correction (fpc).
SAS allows only one number to be supplied on the proc surveymeans
statement. Because the totals change from one stratum to the next, we need to supply them to SAS in a
data set. You can include these data in the primary data set or in a
secondary data set. In this example, we will use a secondary data set.
Also note that the secondary data set can be "collapsed"; in other words, just
one line (observations) for each strata. In the secondary data set,
the variable that contains the totals must be called _TOTAL_. The
variable oblevel is copied from the original data set because SAS
requires all of the variables listed on the strata statement to appear in this
data set. In our example, there is only one variable listed on the
strata statement, but in other cases, there may be two or more variables
listed.
proc surveymeans data = hospsamp n = second138 sum ; weight weighta; strata oblevel; var births; run;
The SURVEYMEANS Procedure
Data Summary
Number of Strata 3
Number of Observations 15
Sum of Weights 157.999931
Statistics
Variable Sum Std Dev
----------------------------------------
births 183983 34014
----------------------------------------
proc surveymeans data = hospsamp n = second138 sum; weight weighta; strata oblevel; by oblevel; var births; run;
oblevel=1
The SURVEYMEANS Procedure
Data Summary
Number of Strata 1
Number of Observations 4
Sum of Weights 42
Statistics
Variable Sum Std Dev
----------------------------------------
births 14931 2669.856738
----------------------------------------
oblevel=2
The SURVEYMEANS Procedure
Data Summary
Number of Strata 1
Number of Observations 5
Sum of Weights 98.999939
Statistics
Variable Sum Std Dev
----------------------------------------
births 117117 33068
----------------------------------------
oblevel=3
The SURVEYMEANS Procedure
Data Summary
Number of Strata 1
Number of Observations 6
Sum of Weights 16.9999924
Statistics
Variable Sum Std Dev
----------------------------------------
births 51935 7508.399372
----------------------------------------
proc surveymeans data = tab9_1c n = 5 sum mean; weight wt1; cluster devlpmnt; var nge65 nvstnrs hhneedvn; run;
The SURVEYMEANS Procedure
Data Summary
Number of Clusters 2
Number of Observations 40
Sum of Weights 100
Statistics
Std Error
Variable Mean of Mean Sum Std Dev
------------------------------------------------------------------------
NGE65 1.675000 0.019365 167.500000 1.936492
NVSTNRS 0.575000 0.019365 57.500000 1.936492
HHNEEDVN 0.525000 0.019365 52.500000 1.936492
------------------------------------------------------------------------
UCLA Researchers are invited to our Statistical Consulting Services
We recommend others to our list of Other Resources for Statistical Computing Help
These pages are Copyrighted (c) by UCLA Academic Technology Services