Date of Award

Spring 2014




Doctor of Education


Counseling, Foundations & Leadership


This mixed-methods study explores the reliability of portfolios as evidence of mastery of the Interstate School Leaders Licensure Consortium (ISLLC) leader standards. The quantitative portion of the study was conducted to determine to what extent are portfolios reliable as evidence of mastery of ISLLC standards and functions of students enrolled in educational leadership specialist programs when measured by intra-class correlation. The qualitative portion of the study was to explore why consistency or inconsistency existed among the portfolio raters. Many colleges and universities utilize portfolios as performance-based assessments to measure growth of educational leadership students. The results of this study are provided as evidence of what is working and what is not working in regard to the utilization of portfolios as evidence of leader skill mastery. iii The two-phase sequential mixed-methods study was used. Portfolios from two cohorts of educational specialist students were rated (N=30) using a six-point Likert-type scale at the function and standard levels. Three raters from a pool of nine were randomly assigned to each of the thirty portfolios, so that each of the nine raters evaluated between six and nine portfolios, and that each portfolio was evaluated by three of the nine raters. After establishing normal distribution of ratings, the data were evaluated through intra-class correlation to determine agreement of the raters regarding standards mastery for each of the ISLLC functions. The intra-class correlation coefficients ranged from -.257 to .626 at the function level and from 105 to .521 at the standard level. Although no agreement was found at the function or standard level, five themes which were supported by prior research emerged from the qualitative data provided by raters. Weak artifacts, confusing or unclear artifacts, misplaced or incorrectly aligned artifacts, incomplete artifacts, and implied artifacts contributed to discrepancies in scores earned by participants in the Educational Leadership Specialist program. Poor reflections lowered scores portfolios earned. Some raters gave credit for characteristics while others did not. A number of raters gave full credit for all functions in a standard if some of the functions were met, while other raters gave credit only for functions which were explicitly represented. Raters also lacked agreement regarding quality of particular artifacts. All of these issues led to poor consensus among the raters of these portfolios. In conclusion, students who participate in performance-based programs which required a portfolio to document the action component need to be well-trained on the purpose of the portfolio and how to create its components. Additionally, students enrolled in Ed. S. Leadership programs need to internalize the importance of reflection, as it is a practice which will iv assist them in creating a high-quality portfolio. Finally, extensive rater training and practice is essential to reliably scoring educational leadership portfolios. v

Bowen, Cindy.pdf (50 kB)
Signature Page