Loading...
Please wait, while we are loading the content...
Similar Documents
Estimating Reliability 1 Running Head : ESTIMATING RELIABILITY Estimating Reliability in Primary Research
| Content Provider | Semantic Scholar |
|---|---|
| Author | Brannick, Michael T. |
| Copyright Year | 2005 |
| Abstract | The current paper describes and illustrates three things: (a) sampling variance and confidence intervals for Cronbach’s Alpha, (b) the relative precision of reliability estimates from local studies and meta-analyses, and (c) how to blend the local and metaanalytic information to create an optimal local reliability estimate according to Bayesian principles. The paper is not about artifact corrections used to compute a meta-analysis. Rather it is about using information contained in a meta-analysis to improve local estimates of reliability. The improved estimates can result in better estimates and corrections for artifacts at the local level. Estimating Reliability 3 Estimating Reliability in Primary Research Measurement experts routinely call for the estimation of the reliability of all measures (scores) used in a study based on that study’s data (e.g., Thompson, 2003; Whittington, 1998). That is, primary researchers are asked to report estimates of the reliability of their measures based on their data. Despite such calls for reporting local estimates, many researchers fail to report any reliability estimates at all or else simply report estimates taken from test manuals or other literature reporting the development of the measure (e.g., Vacha-Haase, Ness, Nilsson, & Reetz, 1999, Yin & Fan, 2000). Measurement experts note that reliability estimates reported in test manuals or in articles reporting the development of a measure may not adequately represent the reliability of the data in any particular study because of the influence of the research context, including the variability of the trait in the population of interest and the context of measurement, including such factors as the purpose of measurement (e.g., selection vs. developmental feedback) and the testing conditions (e.g., noise, light, time of day, etc.). On the other hand, such calls for local reliability estimates typically fail to mention of the importance of sampling error on the precision of the local study estimate (Hunter & Schmidt, 2004; for recent exceptions, see Cronbach & Shavelson, 2004; Vacha-Haase, Henson, & Caruso, 2002). With small samples, the local estimate of reliability will usually be much less precise than a comparable estimate taken from the test manual or from a meta-analysis. There may be a tradeoff between precision and applicability of primary study estimates and meta-analytic reliability estimates. That is, the local estimate may be more applicable than is the meta-analytic estimate (because of the influence of research context), but the local estimate may be less precise than is the metaEstimating Reliability 4 analytic estimate (because of sampling error). Clearly we would like to know the precision of the local estimate and to be able to articulate what the tradeoff may be. It is also possible to blend the local and meta-analytic estimates. The current paper therefore describes and illustrates three things: 1. sampling variance and confidence intervals for Cronbach’s Alpha, 2. the relative precision of estimates from local studies and meta-analyses, 3. how to blend the local and meta-analytic information to create an optimal local estimate according to Bayesian principles (Lee, 1989; Brannick & Hall, 2003). Confidence Intervals for Alpha Cronbach’s alpha appears to be the most commonly reported estimate of reliability in the psychological research literature (Hogan, Benjamin, & Brezinski, 2000). Because it is an intraclass correlation, its sampling distribution is awkward and confidence intervals have only recently become available for it. However, it is quite important to report the precision of the estimate of alpha (that is, its standard error or confidence interval, Iacobucci & Duhachek, 2003) so that researchers can understand the likely magnitude of error associated with the estimate. An asymptotic (large sample) formula for the sampling variance of the function of Cronbach’s Alpha ) ˆ ( α α − n is shown by (Iacobucci & Dubachek, 2003, p. 480, Equation 2; van Zyl, Neudecker, & Nel, 2000, p. 276,Equations 20, 21): [ ] ) )( ( 2 ) )( ( ) ( ) 1 ( 2 2 2 2 3 2 2 j V j trV V tr trV Vj j Vj j k k Q ′ − + ′ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ′ − = (1) where k is the number of items that are added for form the composite whose reliability is indexed by alpha, j is a k x 1 column vector of ones, V is the covariance matrix of the Estimating Reliability 5 items (that is, the sample estimate of the population covariance matrix), and tr is the trace function (the sum of the diagonal elements of a matrix). The asymptotic 95 percent confidence interval is given by (Iacobucci & Dubachek, 2003): ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ ± n Q 96 . 1 α̂ (2) where n = (N-1). A small-scale simulation by van Zyl, Neudecker and Nel (2000) shows that the asymptotic estimate appears to yield reasonable results provided that N is a least 100. However, Yuan, Guarnaccia, and Hayslip (2003) recommended that bootstrap estimates be used to compute confidence intervals rather than asymptotic estimates. They based their recommendations on a comparison of methods using data from the Hopkins Symptom Checklist. Bootstrap estimates are computed by taking repeated samples of data with replacement from the study data to compute an empirical sampling distribution. The empirical sampling distribution is then inspected to see where the extremes of the distribution fall, that is, the bootstrap estimates allow for the calculation of an empirical confidence interval. SAS programs that can be used to compute both asymptotic and bootstrap confidence intervals for alpha can be found at http://luna.cas.usf.edu/~mbrannic/software/softdir.htm. The programs contain sample data from students who completed a questionnaire composed of some IPIP extroversion items. Based on responses of 100 people to the ten items, the estimated alpha is .89, with asymptotic confidence interval (95 percent) of .85 to .92. The bootstrap confidence interval (95 percent) ranges from .85 to .91, so there appears to be good agreement between the two different confidence intervals estimated by the asymptotic and bootstrap Estimating Reliability 6 methods in this case. Either method can be used to estimate the sampling variance of alpha for a local study. Unfortunately, both the asymptotic sampling variance and bootstrap methods require information that is not typically presented in journal articles. The asymptotic method requires the covariance matrix for the items, and the bootstrap method requires the raw data. Both methods are of interest to primary researchers, but meta-analysts typically do not have access to the required data. It is possible, however, to assume compound symmetry for the covariance matrix (that is, to assume that the covariance or correlation among all items is the same). Under the assumption of compound symmetry, it is possible to solve for the common covariance and then to use the asymptotic formula to estimate the sampling variance of alpha. Such a procedure is analogous to using the reported estimates of the mean and standard deviation to estimate the reliability (KR-21) of tests composed of dichotomous items. Under the assumption of compound symmetry, the expression for alpha becomes (van Zyl, Neudecker, & Nel, 2000, p. 272, Equation 3) ) 1 ( 1 − + = k k ρ ρ α (3) where k is the number of items and ρ is the common element in the covariance matrix (i.e., the correlation of each item with all other items). A little algebra allows us to solve for the common element, thus: ) 1 ( − − = k k α α ρ . (4) Estimating Reliability 7 For example, if alpha is .8 and there are 3 items, then the implied correlation matrix is 1 0.5714 0.5714 0.5714 1 0.5714 0.5714 0.5714 1 If the primary researcher reports alpha, the number of items, and the sample size, then the meta-analyst can find an approximate sampling variance and therefore confidence intervals for the alpha estimate. Precision of Estimates Although measurement experts routinely call for local estimates of reliability (and there is no real reason NOT to report them), measurement experts typically fail to note the relative precision of local and meta-analytic estimates of reliability. That is, they fail to note that the local estimates tend to contain much more sampling error than do metaanalytic estimates (see also Sawilowsky, 2000, for further aspects of the controversy about reporting local reliability estimates and their meaning). The second contribution of the current paper is to quantify the precision of the two estimates to allow an explicit comparison of the precision of the two different estimates. There can be no stock answer to the question of the relative precision of the estimated reliability in a given sample versus the mean reliability of a meta-analysis. For the local study, the uncertainty of the reliability depends chiefly on the number of items, the magnitude of the covariances, and the number of people[check this]. As the number of items, the size of the covariance, and the size of the sample all increase, the local estimate will become precise. The meta-analytic result will become precise as the local samples become precise and also as the number of studies in the meta-analysis increases. In both cases, the sampling variance can be used to index the precision of the estimate. Estimating Reliability 8 Let’s look at a single example. The data in the following table were copied from Thompson and Cook (2002). The study was a meta-analysis of reliability estimates for a survey assessing user satisfaction with library services across 43 different universities. Table 1 shows the alpha estimates for the total score (based on k=25 items), as well as the alpha estimates for one of the subscales (5 items), Information Access. The sample size for each sample is also reported. From the information given in Table 1, I computed an estimate of the common element (rho) based on the assumption of compound symmetry. I then calculated the estimated sampling variance for each |
| File Format | PDF HTM / HTML |
| Alternate Webpage(s) | http://mypages.iit.edu/~morriss/siop2005/Brannick05.pdf |
| Language | English |
| Access Restriction | Open |
| Content Type | Text |
| Resource Type | Article |
