The Productive Use of Disagreement in Observational Ratings

For at least two decades researchers have studied the consistency of ratings of classroom instruction by trained observers.    Despite significant efforts focused on protocol design, observer training and calibration, and refined analytic methods, rater inconsistency continues to be a source of frustration and concern for those who are researching teaching and teacher quality.

In this talk, I make the case that rater inconsistency can be construed as more than a source of measurement error.  Instead, inconsistency can be used as a stimulus for the development of the understanding of teaching in multiple ways.  First, disagreements can reveal ambiguities and lack of clarity of the observation protocol itself. Second, rater disagreements can elicit differences in perspectives on the observation that are both legitimate and sensitive to the complexity of teaching.  Third, engagement with different perspectives can enhance professional knowledge of participants in the observation process.

I first introduce a conceptual framework for understanding why inconsistency in observer ratings is inevitable and why the pursuit of high levels of reliability was always an unrealistic endeavor.  I then describe and illustrate strategies and processes that can be used to leverage inconsistencies to promote deeper understanding of teaching practice.  Finally, I consider productive research directions that can enhance our understanding of teacher observation and teacher quality.

Empowerment through collaboration? Analyzing collaborative school improvement processes at different levels of the education system based on a social network approach

The current educational landscape is faced with a multitude of challenges. Rapid responses are required to adapt to changes such as technological innovations or migration, and good strategies must be developed for inclusion and equity within an increasingly diverse student body. Additionally, the professional development of various actors must be advanced continuously – all of these while at the same time improving educational quality. This situation makes effective collaboration and innovative school improvement indispensable.

This keynote addresses the concept of effective collaboration between various actors within a multilevel structured education system with the goal of improving schools. It explores how such collaboration can serve as a central element in preparing schools and teachers to meet current and future demands. The keynote will develop both a theoretical and empirical understanding of collaboration between teachers, principals and administrators with the goal to improve schools.

Increasingly, collaboration in the education sector is being researched using a social network approach. This approach focuses on examining social relationships between actors by capturing and graphically representing interactions between actors, and by analyzing  factors which have an influence on interactions, also longitudinally. I will provide insights into our current research efforts based on this network approach in the keynote.

The results of our research endeavors underscore the relevance and necessity of collaborative school improvement processes at different levels of the education system. At the same time, the keynote will critically discuss the fact that collaboration is not a panacea per se for improving school and teaching quality. From my reflections, I will derive implications for the practical implementation of collaboration on school improvement and provide an outlook on further research.

Educational Effectiveness as a scientific research discipline

One of my first plenary presentations for what was then the school effectiveness research community (ICSEI, 1987) had roughly the same title, although I may have omitted the term “scientific”. I chose this title to stress the need for disciplined inquiry, despite, or rather especially, because of our strong commitment to school improvement. Now, almost 40 years after, I feel the same urge to “defend” the scientific approach, even though the impatience of “improvers” has not diminished and vocal critics have depicted “evidence-based practice” as “oppressive” (Brunker, 2024) and “like the training of dogs” (Biesta, 2021). The core of my presentation is an impression of the state of the art of the knowledge base, and methodological reflections thereon, which leads to a discussion on effectiveness & ineffectiveness and what I have indicated as “limited malleability”. But despite this empiricist emphasis the discussion on research findings is embedded in “theory”.  By which I mean three different kinds of theory: (1) notions about the educational effectiveness research paradigm, (2) key dimensions of a substantive theory on educational effectiveness and examples of (3) theory based explanatory mechanisms in partial “instrumental” theories.

And finally, I would like to connect all of this to the decades-old polemics about traditional and progressive education and make a plea for a modern-traditional perspective as opposed to progressive orthodoxy.