Using Teacher Feedback Program to Improve Learning: A SIEF-supported Impact Evaluation in Brazil

Problem
Policy makers around the world struggle with how to ensure that teachers are effective in the classroom. Pay-for-performance schemes, which provide teacher bonuses, have become increasingly popular, but they can be costly and their effectiveness is mixed. One problem is that even if a teacher wants to do better, he or she may not have the tools or skills to do so. Across Latin America and the Caribbean, classroom observations have shown that teachers lose one day a week in teaching time and even when they are teaching, students are not engaged -- texting, chatting or otherwise tuned out during almost three-quarters of every class.

Intervention 
SIEF researchers, working with Brazil's Lehmann Foundation and the Ceara state government, evaluated a one-year program to test whether providing secondary school teachers with classroom observation feedback and expert coaching could improve classroom practice and raise student learning. The program ran for one year between 2014 and 2015.

Evaluation design 
Cluster randomized control trial. The 150-school treatment group received detailed feedback from Stallings method classroom observations, four 1-day face-to-face coaching sessions, and bi-weekly support from the expert coaches via Skype. Teachers were observed at the end of the 2014 school year and again at the end of the 2015 school year. Student achievement was measured on two different tests -- the state achievement test and the national high school leaving exam.

Results
Teachers increased time on instruction by 10% by reducing time on classroom management and out of the classroom, translating into almost 3 additional weeks of instruction per year. Teachers also used more question and answer based pedagogy, which increased student engagement. Student test scores improved across the board, in both Portuguese and math, on both tests. Average gains were modest, but statistically significant. But in the 30% of schools that implemented the program most intensively, test scores rose 0.13-0.23 SD -- very big improvements in a single school year. Delivering the coaching via Skype kept costs at $2.40 per student, making the program highly cost-effective compared with other rigorously-evaluated teacher training programs in developing countries, and a promising strategy for whole-school efforts to raise teacher effectiveness.

Policy impact
Based on the findings from the impact evaluation and the Stallings classroom observations, the Ceara government plans to launch a voluntary training program for teachers in all municipal primary schools in 2017.

For more information about the Footprints of other SIEF-funded impact evaluations, visit www.worldbank.org/sief or email SIEFimpact@worldbank.org.

Researchers randomly selected 400 schools from Ceara’s estimated 600 schools. Two hundred of the schools selected were randomly assigned to be the treatment group; the other 200 schools were the control group and they did not receive the program. The program that the treatment group schools received included an information campaign, which consisted of detailed feedback on the school's results from the classroom observations, as well as information on teacher performance to enable schools to determine which teachers exhibited the best practices. The treatment group also received a set of self-help materials including a book, videos and exercises about effective teaching strategies, as well as a log book and classroom observation templates to record teachers’ observations of one other. Researchers assessed the results of student test scores to determine whether the campaign helped improve classroom learning.