Optimizing the selection and training of raters: How do rater characteristics and training features affect the quality of judgments of teaching quality?
Classroom observations are frequently used in teacher evaluation, training, and professional development as well as in research. Ensuring that the observations are reliable and valid is therefore of great importance. Studies have shown however, that they are often neither, even when the raters are trained (Bell et al., 2014; Gitomer et al., 2014; Praetorius & Charalambous, 2018). There is limited systematic and causal evidence about ways of enhancing the reliability and validity of such ratings (Cohen & Goldhaber, 2016; Hill, Charalambous, & Kraft, 2012; Praetorius, 2014). One promising avenue of research is looking at the characteristics of the raters and how well they are trained (Bell et al., 2014; Cash et al., 2012; Styck et al., 2020).The aim of this project is to systematically investigate fundamental rater characteristics and aspects of rater training and their effect on the quality (i.e., reliability and validity) of judgments of teaching quality. We aim to generate evidence for how the selection and training of raters can be optimized, taking into account feasibility issues. The project focuses on the Three Basic Dimensions of teaching quality (see Praetorius, Klieme, Herbert, & Pinger, 2018): classroom management, student support, and cognitive activation. We will investigate the rele-vance of the following aspects of observations on teaching quality: (RQ 1) the raters’ content-related character-istics (content knowledge and pedagogical content knowledge with respect to the content of the observed les-sons, their study major) and their generic characteristics (personality, pedagogical knowledge, beliefs, and teach-ing experience) and (RQ 2) specific rater constellations based on typically used numbers of raters (n = 2-4) in a heterogeneous pool of raters, (RQ 3) experimental manipulations of raters’ content knowledge and pedagogical content knowledge of the topic of the observed lessons, (RQ 4) experimental manipulations of the practice phase (guided discussion vs. self-directed reflection) of rater training, and (RQ 5) experimental manipulations of training duration (12h versus 6h). We will maximize the variance of the characteristics of the project’s rater pool by selecting participants with a range of educational attainment (BA and MA students and qualified teachers), who are on different tracks (training to teach in primary schools, secondary schools, or upper secondary baccalaureate schools, or studying educational science) and specializing in a variety of subjects (science education, mathematics education, gen-eral education, or other). We will also take advantage of the uniquely heterogeneous nature of education set-tings in Germany and Switzerland. The lessons that will be rated comprise two grade levels and two core sub-jects, science (focus on floating and sinking, grades 3-4) and mathematics (focus on the Pythagorean theorem, grades 8-9). RQ 1 and RQ 2 will be investigated using the control groups in the four experiments (using multi-level regression analyses and real-data simulations). RQ 3-5 are investigated using experimental manipulations of each of the areas of interest. The central dependent variables are inter-rater reliability (based on generalizabil-ity theory), rater agreement with reference ratings (based on distance scores), and predictive validity with respect to students’ conceptual understanding as well as content-specific interest (using two-level regression models). For RQ 3 and RQ 4, think-alouds and cognitive interviews with subsamples will be conducted to gain further insight into the differences between the experimental conditions (using qualitative content analysis).
- Start/End: 08.2022-31.7.2025
- Project-Leader: Praetorius Anna-Katharina, Institute of Education, University of Zurich
- Co-applicants: Heinze Aiso (Didaktik der Mathematik IPN an der CAU), Kleickmann Thilo (Universität zu Kiel IPL), Steffensky Mirjam (Leibniz Institute for Science and Mathematic Department of Educational Research)
- Partner: Hartig Johannes (Educational Measurement Bildungsqualität und Evaluation DIPF Leibniz-Inst. für Bildungsforschung)
- Staff: Gitzi Valerie
The joint project COLD (Competencies of school teachers and adult educators in teaching German as a second language in linguistically diverse classrooms) is headed by the German Institute for Adult Education (DIE). The project is funded within the framework of the Cooperative Excellence Programme with funds from the Leibniz Competition 2019.
The content of the project is the assessment of the professional competences of teachers in schools and adult education in teaching German as a second language in linguistically heterogeneous learning groups. It also addresses the challenges in teaching and didactics that have arisen across educational sectors as a result of the immigration of children, young people and adults. The addressed research interests are examined in a real teaching context with teachers in integration courses and preparation classes.
- Duration: April 2019 – March 2022
- Project Management: Prof. Dr. Josef Schrader (DIE)
- Coordination: PD Dr. Hannes Schröter (DIE), Dr. Stefanie Bredthauer (University of Cologne)
- Cooperation partners: Prof. Dr. Michael Becker-Mrotzek (University of Cologne), Prof. Dr. Hans-Joachim Roth (University of Cologne), Prof. Dr. Eckhard Klieme (DIPF), Prof. Dr. Uta Klusmann (IPN, Kiel University), Prof. Dr. Fani Lauermann (TU Dortmund University), Prof. Dr. Irit Bar-Kochva (University of Cologne), Prof. Dr. Johannes König (University of Cologne), Prof. Dr. Walt Detmar Meurers (University of Tübingen), Prof. Dr. Anna-Katharina Praetorius (University of Zurich)
- Research assistants/doctoral students: Moritz Sahlender (DIE), Anastasia Knaus (University of Cologne), Zarah Leonie Weiss (University of Tübingen)
TALIS Video Study
Various countries and regions participate in the OECD study TALIS-Video. In addition to Germany, Japan, Shanghai, England, Chile, Colombia, Mexico and the Madrid region are also involved. The aim of the study is to investigate the connection between teaching processes and learning success as well as the motivational characteristics of pupils. The study is led by an international consortium. In addition to the DIPF, this consortium includes the two US organisations ETS and RAND.
Further information can be found under TALIS Video Study.
- Duration: 2016 - 2020
- Interational consortium: Prof. Dr. Darleen Opfer (RAND), Prof. Dr. Brian Stecher (RAND), Dr. Courtney Bell (ETS), Dr. Dan McCaffrey (ETS), Dr. Tom van Essen (ETS), Prof. Dr. Eckhard Klieme (DIPF), Prof. Dr. Anna-Katharina Praetorius (University of Zurich; associate scientist of the DIPF)
TALIS Video Study for Germany
The TALIS Video Study for Germany is a project of the Leibniz Association, which follows the international video study Teaching and Learning International Survey (TALIS-Video) and is being processed by the DIPF within the framework of a research network. The study is expected to accompany 85 mathematics classes at secondary level 1 on the topic of "quadratic equations". The teaching units conducted on this topic will be recorded on video and evaluated with regard to different quality dimensions of teaching. In addition, performance tests as well as student and teacher surveys will be used. The aim of the study is to take a closer look at teaching processes and to relate them to the learning success of pupils.
Further information can be found under TALIS-Video Study for Germany.
- Duration: 2017 - 2019
- Project management: Prof. Dr. Eckhard Klieme (DIPF), Dr. Juliane Grünkorn (DIPF), Prof. Dr. Anna-Katharina Praetorius (University of Zurich)
- Project team: Patrick Schreyer (DIPF), Julia Käfer (DIPF), Benjamin Herbert (DIPF)
Leibniz network on instructional research
The network supported by the Leibniz Association is an interdisciplinary circle of teaching researchers. They work together on current challenges in teaching research. The network members are researchers with a doctorate from various institutions and disciplines such as educational science, educational psychology, mathematical didactics and psychometry.
Further information can be found under Leibniz network on instructional research.
- Scientific Coordination: Prof. Dr. Eckhard Klieme (DIPF), Prof. Dr. Anna-Katharina Praetorius (University of Zurich), Dr. Juliane Grünkorn (DIPF)
Triangulation Instruction (Tri-U)
The project "Triangulation Instruction (Tri-U)" approaches teaching processes in German mathematics teaching on the topic of "quadratic equations" with different methodological approaches. In three classes in Berlin, quantitative and qualitative methods of video analysis are brought together. In the quantitative approach, the teaching is assessed with regard to different quality dimensions by means of highly inferential ratings. In the qualitative approach, the pedagogical-phenomenological videography is evaluated. The results from the different methodological approaches are related to each other and compared with each other with regard to cognitive activation in the classroom.
Further information can be found under Triangulation Instruction (Tri-U).
- Duration: 2018 - 2019
- Project management: Prof. Dr. Anna-Katharina Praetorius (University of Zurich), Prof. Dr. Malte Brinkmann (HU Berlin)
- Project team: Wida Rogh (University of Zurich), Johannes Türstig (HU Berlin), Martin Weber-Spanknebel (HU Berlin)
Development of a subject-independent instrument for a video-based classification of metacognitive-discursive instructional quality
The aim of the project is to develop a coding and rating instrument for recording metacognitive-discursive teaching quality. In particular, the instrument not only records the teacher's metacognitive stimuli, but also the discursive quality of teaching and the learners' metacognitive activities.
Further information can be found under MeDUQua.
- Duration: 2015 - 2017
- Project management: Prof. Dr. Frank Lipowsky (University of Kassel), Prof. Dr. Elmar Cohors-Fresenborg (Osnabrück University), Prof. Dr. Anna-Katharina Praetorius (University of Zurich), Dr. Edyta Nowinska (Osnabrück University), Prof. Dr. Martina Blasberg-Kuhnke (Osnabrück University), Prof. Dr. Meik Zülsdorf-Kersting (University of Oldenburg)
Observation-based evaluation of teaching quality
The project was launched on behalf of the "Arbeitsgemeinschaft Externe Evaluation von Schulen" (argev) and comprises the development and piloting of a set of instruments for the observation-based evaluation of teaching quality as well as the subsequent development of a qualification offer for evaluation experts. The instruments to be developed are geared towards external school evaluation and competence-oriented teaching according to curriculum 21.
Further information can be found at argev.
- Duration: 2018 - 2023
- Project management: Prof. Dr. Anna-Katharina Praetorius (University of Zurich), Wida Rogh (University of Zurich)
- Project team: Leonie Gossner (University of Zurich), Roya Hüppi (University of Zurich), Fabian Hug (University of Zurich)