Title

Not Just Generalizability: A Case for Multifaceted Latent Trait Models in Teacher Observation Systems

Document Type

Article

Publication Date

11-1-2019

Publication Title

Educational Researcher

Volume

48

First Page

521

Last Page

533

Keywords

measurements, psychometrics, RASCH models, school/teacher effectiveness, teacher assessment, validity/reliability

Abstract

© 2019 AERA. Teacher evaluation systems often include classroom observations in which raters use rating scales to evaluate teachers’ effectiveness. Recently, researchers have promoted the use of multifaceted approaches to investigating reliability using Generalizability theory, instead of rater reliability statistics. Generalizability theory allows analysts to quantify the contribution of multiple sources of variance (e.g., raters and tasks) to measurement error. We used data from a teacher evaluation system to illustrate another multifaceted approach that provides additional indicators of the quality of observational systems. We show how analysts can use Many-Facet Rasch models to identify and control for differences in rater severity, identify idiosyncratic ratings associated with various facets, and evaluate rating scale functioning. We discuss implications for research and practice in teacher evaluation.

This document is currently not available here.

Share

COinS