Title
Not Just Generalizability: A Case for Multifaceted Latent Trait Models in Teacher Observation Systems
Document Type
Article
Publication Date
11-1-2019
Publication Title
Educational Researcher
Volume
48
First Page
521
Last Page
533
Keywords
measurements, psychometrics, RASCH models, school/teacher effectiveness, teacher assessment, validity/reliability
Abstract
© 2019 AERA. Teacher evaluation systems often include classroom observations in which raters use rating scales to evaluate teachers’ effectiveness. Recently, researchers have promoted the use of multifaceted approaches to investigating reliability using Generalizability theory, instead of rater reliability statistics. Generalizability theory allows analysts to quantify the contribution of multiple sources of variance (e.g., raters and tasks) to measurement error. We used data from a teacher evaluation system to illustrate another multifaceted approach that provides additional indicators of the quality of observational systems. We show how analysts can use Many-Facet Rasch models to identify and control for differences in rater severity, identify idiosyncratic ratings associated with various facets, and evaluate rating scale functioning. We discuss implications for research and practice in teacher evaluation.
Recommended Citation
Wind, Stefanie A. and Jones, Eli, "Not Just Generalizability: A Case for Multifaceted Latent Trait Models in Teacher Observation Systems" (2019). Faculty Bibliography. 2742.
https://csuepress.columbusstate.edu/bibliography_faculty/2742