The numbers were supposed to shed light on what was happening in public schools. That was the idea behind the No Child Left Behind Act of 2001. It mandated that every third through eighth grade student had to take an annual test to see who was performing at grade level.
In the years after the law went into effect, the testing and data industries flourished, selling school districts interim assessments to track student progress throughout the year along with flashy data dashboards that translated student achievement into colored circles and red warning flags. Policymakers and advocates said that teachers should study this data to understand how to help students who weren’t doing well.
Teachers are spending a lot of time talking about student data. In a 2016 survey by Harvard’s Center for Education Policy Research, 94 percent of middle school math teachers said they analyzed student performance on tests in the prior year, and 15 percent said they spent over 40 hours on this kind of data analysis. In high poverty schools, where student test scores are often low and there is pressure from state and local governments to raise them, data analysis can dominate weekly or monthly meetings among teachers.
Apart from controversies over the use of tests and cheating scandals, researchers are asking another basic question: has all that time teachers spent studying data helped students learn? The emerging answer from education researchers is no. That conclusion is like dropping a bomb on a big part of what happens at schools today.
“Studying student data seems to not at all improve student outcomes in most of the evaluations I’ve seen,” said Heather Hill, a professor at the Harvard Graduate School of Education, at a February 2022 presentation of the Research Partnership for Professional Learning, a new nonprofit organization that seeks to improve teaching.
“It’s a huge industry and there are major sales to schools,” said Hill in an interview afterward. “The market forces continue to push this on schools even with very, very limited efficacy evidence unfortunately.”
Hill reviewed 23 student outcomes from 10 different data programs used in schools and found that the majority showed no benefits for students. Only two were positive for students and in one study, students were worse off.
Another pair of researchers also reviewed studies on the use of data analysis in schools, much of which is produced by assessments throughout the school year, and reached the same conclusion. “Research does not show that using interim assessments improves student learning,” said Susan Brookhart, professor emerita at Duquesne University and associate editor of the journal Applied Measurement in Education. “The few studies of interim testing programs that do exist show no effects or occasional small effects.”
Two randomized controlled experiments found weak gains in math but not in reading. Two additional studies with control or comparison groups found no significant results. Brookhart co-wrote a chapter summarizing the research in an upcoming new edition of the book Educational Measurement, an influential text in the field. She and her co-author, Charles DePascale, a retired assessment consultant, provided me with a pre-publication draft of their chapter.
Why doesn’t data analysis work? All three researchers explained that while data is helpful in pinpointing students’ weaknesses, mistakes and gaps, it doesn’t tell teachers what to do about them. Most commonly, teachers review or re-teach the topic the way they did the first time or they give a student a worksheet for more practice drills.
Teachers need to change their approach to address student misunderstandings, Hill said.
The upside is that the data analysis bandwagon has prompted many schools to allocate more time for teachers to meet. And researchers believe this collaboration, apart from the solitary work of classroom teaching, is valuable for teachers in improving their craft.
“As long as it’s not studying student data,” said Hill.
The most effective use of teachers’ time together is not clear and Hill said she and her colleagues at Research Partnership for Professional Learning are studying that now.
This story about teacher use of data was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter.