Reliability, validity, and fairness are the three major indicators of quality of an assessment; and they are also broad and controversial topics in educational assessments (Davies, 2010; Kane, 2010). There are three main approaches to the examination of these topics in educational assessments. They are: (a) the classical test theory (CTT) approach; (b) the generalizability (G-) theory approach; and (c) the item response theory (IRT) approach. Due to the limited time, the IRT approach will not be covered in this workshop.

G-theory, developed by Cronbach and his colleagues (Cronbach, Gleser, Nanda, & Rajaratnam, 1972), is grounded in the framework of a factorial design and the analysis of variance. G-theory can be viewed as “an extension and liberalization of classical theory that is achieved primarily through the application of analysis of variance procedures to measurement data” (Feldt & Brennan, 1989, pp. 127-128). G-theory has an “important role in all forms of educational assessment, including direct writing assessments and performance assessments in other content areas” (Ferrara, 1993, p. 2). That is, in contrast to CTT, G-theory considers multiple sources of error variance simultaneously and allows different sources of variability (e.g., raters, test forms, items, and persons) to be estimated independently within one analysis (Brennan, 2001; Webb, Rowley, & Shavelson, 1988). The G-theory approach is a more powerful approach than CTT for examining assessment quality issues (Shavelson & Webb, 1991; Kraiger & Teachout, 1990).

The framework of G-theory incorporates two stages corresponding to the two stages of test design: (a) a generalizability study (G-study) which yields information that can be used to design and (b) a decision study (D-study) which can be used to make decisions about individuals or groups of individuals (e.g., schools). G-studies are used to evaluate the relative importance of various sources of measurement error and investigate the impact of various changes in the measurement design (e.g., different number of tasks or raters/ratings). D-studies incorporate the best design to allow the interpretation of score reliability in a relative or norm-referenced frame of reference or in an absolute or criterion-referenced frame of reference (Brennan, 2001; Gao & Brennan, 2001).

This 3-hour workshop is designed to introduce researchers in the social sciences to the G-theory approach in research designs and data analyses. Using GENOVA (the program is free for downloading) and authentic research data (both small scale and large-scale), the participants will learn how to conduct G-studies and D-studies. This workshop is free for all participants attending the 1st Untested Ideas International Research Conference.

Book
Huang, J. (2012). Fairness in large-scale ESL writing assessments – Investigating evidence through generalizability theory. Saarbrücken: LAMBERT Academic Publishing GmbH & Co. [ISBN: 978-3-659-19364-4]

Referred Journal Articles
Huang, J. & Han, T. (in press). Holistic or analytic? An EFL institutional writing assessment dilemma for policy makers. Leadership and Policy Quarterly. 2(1)

Elorbany, R., & Huang, J. (2012). Examining the impact of rater educational background on ESL writing assessment: A generalizability theory approach. Language and Communication Quarterly, 1(1), 2-24.

Huang, J. (2012). Using generalizability theory to examine the accuracy and validity of large-scale ESL writing. Assessing Writing, 17(3), 123-139.

Huang, J. (2011). Generalizability theory as evidence of concerns about fairness in large-scale ESL writing assessments. TESOL Journal, 2(4), 423-443.

Huang, J., & Foote, C. (2011). Using generalizability theory to examine scoring reliability and variability of judging panels in skating competitions. Journal of Quantitative Analysis in Sports, 7(3), 1-21.

Huang, J., Foote, C. (2010). Grading between the lines: What really impacts professors’ holistic evaluation of ESL graduate student writing. Language Assessment Quarterly, 7(3), 219-233.

Huang, J. (2008). How accurate are ESL students’ holistic writing scores on large-scale assessments? – A generalizability theory approach. Assessing Writing, 13(3), 201-218.

Lewis, C. & Huang, J. (2012). Motivators in the elementary classroom: A quantitative investigation of gender differences. International Journal of Education and Culture, I(I), 2-29.