This project will create a suite of digital tools to observe and measure college teaching that documents frequency and duration of classroom activities but does not require subject matter expertise.
Although classroomobservations are becoming more prevalent in peer evaluations of collegeteaching, there are very few standardized observation protocols of teaching inexistence (Wieman & Inkelas, 2014). In addition, those that are availabletend to require extensive training to use, specialized subject matter expertise(e.g., a detailed knowledge of physics to evaluate a physics class), and areprimarily evaluative in nature (Smith, Jones, Gilbert, & Wieman, 2013).
Karen Kurotsuchi Inkelas,Associate Professor in the Curry School of Education developed the firstiteration of the Real-time Observation of Classroom Activities (ROCA). The ROCAcharts classroom activities through a series of menu-based options in a webapplication. Unlike other observation protocols, however, it documents bothfrequency and duration of classroomactivities. Interval-based protocols, which is the prevalent form of mostavailable observation protocols, can only measure frequency, or how often(within a certain time period) something happens in a class. The ROCA alsomeasures the duration of the activity, or for how long the activity took place.In addition, as mentioned previously, the ROCA is not evaluative but is insteaddescriptive. The user-friendly application does not require extensive trainingto use or a sophisticated knowledge of the course’s subject matter. And, theROCA is designed to be used in any type of class (e.g., lecture, seminar,flipped) and with any discipline (e.g., STEM, humanities, professional).
Observers using the ROCAclick on various options to document, moment-by-moment, what both instructorsand students are doing in the class session. The options on the ROCA wereselected based on the identification of best practices in teaching outlined inthe scholarship of teaching and learning, were prioritized in consultation witheducational developers, and then were streamlined for use in anapplication-based protocol. The options include:
- Teacher-focusedinstructional activities (lecturing, lecturing from pre-made media, interactivelecture, clicker questions, etc.)
- Student-focusedinstructional activities (small group work, presentations, deskwork, etc.)
- Teacher-led dialogues(display questions, comprehension questions, feedback, etc.)
- Student-led dialogues(display questions, responses to instructor, etc.)
- Instructor-focused effort(problem solving, writing, monitoring student work, etc.)
Two versions of the ROCA will be developed, one will be designed to be used in live classroomobservations, and the other utilizing digitally recorded observations. The ROCAversion using digital recordings is meant to be used for research purposes,since it is critical to synchronize time in the class sessions being observedfor purposes of inter-rater reliability. The ROCA version for live observationswill primarily be used by educational developers for formative assessment purposes.
Please note that Lindsay Wheeler will self-fund at $6,667.00, and Karen Inkelas will self-fund at $6,667.00
In order to build thefull suite of digital tools associated with the ROCA, the following are thenext steps of the process:
- To pilot-test the ROCA inlive and recorded classrooms across the University in differentSchools/Colleges and compare the results to other widely available and utilizedobservation protocols.
- To create a visualizationtool utilizing the ROCA output data to illustrate how both instructors andstudents engage in a classroom session.
- To develop a tool thatcan translate ROCA codes to codes used in other popular observation protocols
- To work with educationaldevelopers to refine the use of the ROCA to improve instruction
Students are an integralpart of this project.
First, Computer Sciencestudents from both SEAS and Arts & Sciences will work with an instructionaldesigner to build the suite of digital tools, including the ROCA.
Second, undergraduate andgraduate students will serve as the observers of classrooms at UVA topilot-test the ROCA web-application.
Third, the students whodesigned the ROCA and those who pilot-tested it in UVA classrooms will be vitalmembers of our design thinking process, particularly the ideation of the test portion of the process.