Loading...
Please wait, while we are loading the content...
Application of G Theory to Concept-Map 1 Running Head : Application of G theory to Concept-Map Application of Generalizability Theory to Concept-Map Assessment Research ( Draft )
| Content Provider | Semantic Scholar |
|---|---|
| Author | Shavelson, Richard J. |
| Copyright Year | 2004 |
| Abstract | In the first part of this paper we discuss the feasibility of using Generalizability (G) Theory (see Footnote 1) to examine the dependability of concept map assessments and to design a concept map assessment for a particular practical application. In the second part, we apply G theory to compare the technical qualities of two frequently used mapping techniques: construct-a-map with created linking phrases (C) and construct-a-map with selected linking phrases (S). We explore some measurement facets that influence concept-map scores and estimate how to optimize different concept mapping techniques by varying the conditions for different facets. We found that C and S were not technically equivalent. The G coefficients for S were larger than those for C. Furthermore, a D study showed that fewer items (propositions) would be needed for S than C to reach desired level of G coefficients if only one occasion could be afforded. Therefore, S might be a better candidate than C in the large-scale summative assessment, while C would be preferred as a formative assessment in classroom. Application of G Theory to Concept-Map 3 Application of Generalizability Theory to Concept-Map Assessments Assessment of learning is typically narrowly defined as multiple-choice and shortanswer tests; achievement is typically what multiple-choice tests measure. There is more to achievement than this, and a definition of achievement might well consider the structure of a student’s knowledge, not just the quantity. To this end, concept maps provide one possible approach. Once this approach is taken, often the technical quality of the concept-map assessment is assumed; it is not clear just how to evaluate reliability, for example. In this paper, we present a Generalizability Theory framework (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) for examining the dependability of concept-map assessments and demonstrate it application. Concept Maps A concept map is a network that includes nodes (terms or concepts), linking lines (usually with a uni-directional arrow from one concept to another), and linking phrases which describe the relationship between nodes. Linking lines with linking phrases are called labeled lines. Two nodes connected with a labeled line are called a proposition. Moreover, concept arrangement and linking line orientation determine the structure of the map (e.g., hierarchical or non-hierarchical). Concept maps were originally proposed to be used as an instructional tool (e.g., Novak & Gowin, 1984) and later as an assessment as well (Ruiz-Primo & Shavelson, 1996). Concept maps hold promise in tapping students’ declarative knowledge structures which traditional assessments are not good at. This feature of concept maps attracted assessment researchers’ attention. Ruiz-Primo and Shavelson (1996) characterized the Application of G Theory to Concept-Map 4 variation among concept-map assessments in a framework with three dimensions: a task that invites students to provide evidence for their knowledge structure in a content domain, a response form that students use to do the task, and a scoring system that the raters can use to evaluate students’ responses (Appendix 1). To get a comprehensive review of the variations, readers can refer to the paper written by Ruiz-Primo and Shavelson (1996). Even though thousands of concept-map assessment permutations are possible, not all alternatives are suited for assessment (Ruiz-Primo & Shavelson, 1996). Ruiz-Primo and Shavelson pointed out that reliability and validity information about different mapping techniques should be supplied before concept maps are used for assessment. Our study is one such effort. In particular, in the first part of this paper, we discuss the feasibility of using G theory to evaluate the dependability of concept map scores. In the second part of this paper, we illustrate how G theory can be applied in this kind of research by comparing two frequently used concept-mapping tasks: construct-a-map by creating linking phrases (C) and construct-a-map by selecting linking phrases (S). Part 1. Application of G theory to Concept-Map Assessment Issues and Problems Related to the Technical Properties of Concept-map Assessments Concept maps vary greatly from one another both for instruction and assessment. When the concept maps are used as an assessment, it becomes critical to narrow down options by finding reliable, valid, and efficient mapping techniques. Ruiz-Primo et al.(Ruiz-Primo, Shavelson, & Schultz, 1997, March, p. 7) suggested four criteria for eliminating alternatives: “(a) appropriateness of the cognitive demands required by the Application of G Theory to Concept-Map 5 task; (b) appropriateness of a structural representation in a content domain; (c) appropriateness of the scoring system used to evaluate the accuracy of the representation; and (d) practicality of the technique”. Even though criterion (c) only talked about the scoring system, we (Yin, Vanides, Ruiz-Primo, Ayala, & Shavelson, In Press) found that the accuracy of the scores is not only related to the scoring systems, but also related to the task format. For example, using the same scoring form, some task formats might be scored more reliably and accurately than others (Yin et al., In Press). This paper, then, mainly focuses on criteria (b) and (c), which have typically been gauged by traditional statistical analyses and classical test theory. For example, mainly using those methods, researchers examined scores for inter-rater reliability/agreement (Herl, O'Neil, Chung, & Schacter, 1999; Lay-Dopyera & Beyerbach, 1983; Lomask, Baron, Greig, & Harrison, 1992, March; McClure, Sonak, & Suen, 1999; Nakhleh & Krajcik, 1991); stability(Lay-Dopyera & Beyerbach, 1983); convergent validity—the correlation between concept map score and other assessment score in the same content domain (Anderson & Huang, 1989; Baker, Niemi, Novak, & Herl, 1991, July; Markham, Mintzes, & Jones, 1994; Novak, Gowin, & Johansen, 1983; Rice, Ryan, & Samson, 1998; Schreiber & Abegg, 1991); predictive validity (Acton, Johnson, & Golldsmith, 1994); equivalence of different scoring methods (McClure et al., 1999; Rice et al., 1998); and equivalence of different concept-map tasks(Ruiz-Primo, Shavelson, Li, & Schultz, 2001; Yin et al., In Press). Those studies have supplied important information about technical properties of different concept map tasks, response formats, and scoring systems, which can undoubtedly help to eliminate improper alternatives. However, because the variations Application of G Theory to Concept-Map 6 among concept map assessments are so great that classical test theory cannot handle those variations simultaneously and efficiently. Examining Concept-Map Assessments’ Technical Properties with G theory If we view a concept map assessment score as a sample from a universe of conditions with all kinds of variations, for example, tasks, response formats, and scoring systems, we can examine concept map assessments in the framework of G theory. Strength of G theory. Compared with classical test theory, G theory can (1) integrate conceptually and simultaneously evaluate test-retest reliability, internalconsistency, convergent validity, and inter-rater reliability; (2) estimate not only the influence of individual measurement facets, but also interaction effects; (3) permit us to optimize an assessment’s dependability (“reliability”) within given dollar and time cost constraints. For example, the concept-map assessment designers can obtain information about how many occasions, how many concepts, and how many raters are needed to reach a dependable result; (4) as a general advantage in assessing students’ performance, G study can supply dependability information on students’ absolute level of knowledge structure quality as well as relative level of it. Object of Measurement. Typically education research using concept maps focuses on the variation in the quality and complexity of students’ declarative knowledge structures in a certain subject. This is the variability that the concept-map assessments intend to measure—the object of measurement. Variation among students, then, is desirable variation and should not be confused with variation caused by other sources. There are many other sources of variation in concept map scores that contribute error to the measurement. They include individual Application of G Theory to Concept-Map 7 factors and the interaction between/among more than one factor. The individual factors leading to measurement error are called facets in G theory. Facets. The following are some possible facet examples of individual factors that are characteristic of concept maps. (1) Concept/term Sampling—Concept maps sample concepts from some domain, for example, key concepts. Concept sampling may give rise to variability in a student’s performance; performance might differ with another sample of concepts. Concept sampling, then, introduces error to the measurement when we want to generalize a student’s performance from one map with certain concept sample to a universe of maps with concept-terms sampled. (2) Proposition Sampling—Each proposition in a concept map can be regarded as an independent item in a test, sampled from some domain. Different propositions vary in difficulty level. Proposition sampling, then, can cause variation in the measures of the proficiency of students’ declarative knowledge structures. Notice that facet (2) is similar to facet (1) in that they are both related to the variation due to concept sampling; however, facet (1) focuses on sampling at a macro level, analogous to alternate form reliability in the classic test theory, while facet (2), analogous to internal consistency, focuses on sampling at a micro level. The two facets’ similarity and difference again show the strength of G theory in that it allows researchers to flexibly focus on the interested error type in the analysis to meet specific need |
| File Format | PDF HTM / HTML |
| Alternate Webpage(s) | http://web.stanford.edu/dept/SUSE/projects/ireport/articles/concept_maps/YUE%20YIN%202004%20AERA%20Concept%20Map%20Generalizability.pdf |
| Language | English |
| Access Restriction | Open |
| Content Type | Text |
| Resource Type | Article |
