In this document we present a non-technical discussion of our approach to estimating the reliability of survey measurement for specific survey questions. The estimation of reliability of survey data one needs, first, a model that specifies the linkage between true and observed variables; second, a research design that permits the estimation of parameters of such a model; and third, an interpretation of these parameters that is consistent with the concept of reliability (see Alwin and Krosnick, 1991). As we discuss in the following, this study meets these requirements, and here we discuss the rationale and background for thinking about measurement error using longitudinal data and the design requirements employed.
The estimates of reliability in this data base are based on the longitudinal analysis of panel survey data. The approach is rooted in classical ideas about measurement error, and it capitalizes on the availability of statistical models that separate measurement error from the measurement of change or stability in the underlying variables of interest. As an example, an individual’s true-score on some variable, say income level, or years of schooling, or any variable that can be thought of as basically continuous, may change over time, and measurement error can contribute to the observed change. Our approach uses three waves of panel survey data to separate the amount of true change from the amount of random measurement error.
Our study design for estimating the reliability of survey questions requires
In order to estimate the reliability of a particular survey question, it was necessary that questions were selected for use only if they were exactly replicated, that is, using the exact wording, response categories, mode of interviewing, etc.) across the three waves. It was also assumed that the question assessed an underlying variable that could be expressed as continuous (rather than categoric) in nature.
Continue reading