Photo By Michael D. Spencer

Research

A World Of Data

Marcus Winters analyzes decades of data to reveal which education reforms are working—and which are not

In February 2019, Marcus Winters published a scathing op-ed about New York City Mayor Bill de Blasio in the New York Post.

“Four years and $773 million later, Hizzoner announced this week that Renewal Schools has come to an end,” Winters wrote about the decision to end one of de Blasio’s marquee education programs. “More objective observers had seen this coming for quite some time.”

In case it wasn’t clear: Winters, a BU Wheelock associate professor and chair of educational leadership and policy studies, counts himself as one of those observers. As a quantitative researcher, Winters tries to cut through the political noise to reveal underlying data that tells us what’s working in our school systems—and what isn’t. In a 2017 study published by the Manhattan Institute for Policy Research, a free-market think tank, he found that although Renewal Schools showed a slight boost in student test scores, a third of the schools were still underperforming.

The Renewal Schools program, which directed additional funds and services to underperforming schools, also cost more than the previous administration’s policy of grading schools’ performances and closing those that continued to fail. Long before that contrast had developed, Winters had even met with the de Blasio team to explain his findings that when Mayor Michael Bloomberg’s education department gave schools an “F” grade, those schools began performing better the next year. “Every time we’ve looked at this, we’ve seen this positive effect,” he told them. Yet that policy was abandoned.

The intersection of education and politics can be messy, fraught with partisan agendas, red tape, and budget constraints. Decisions aren’t always made strictly by what the numbers say. And Winters knows that de Blasio’s team might have seen other consequences to the Bloomberg policy—including clashes with the teachers’ unions—that they wished to avoid. “No one’s doing exactly what researchers say they should do, nor should they,” he says. But, he adds, it’s frustrating when data consistently points in one direction and policy moves in the other. “They moved away from Bloomberg’s policy precisely when we were getting evidence that it was working really well,” he says.

Though the acceptance and interpretation of quantitative education research can vary, Winters has seen its use expand greatly during his career. And with ever-increasing amounts of data available, there’s growing demand for policy researchers across the education spectrum, from local school districts to philanthropic and nonprofit organizations.

“How we use evidence to inform existing policies and future policies can, hopefully, keep us going in the right direction and also can stop us from making mistakes,” Winters says. In recognition of this trend, he and a group of colleagues have designed a master’s program in education policy studies that welcomed its first cohort this fall.

An Unorthodox Path

Winters began studying education in 2002, as a research associate at the Manhattan Institute. “The more I got into education, the more I enjoyed it,” he says. “This is an area that’s extremely important, so it has a lot of resources attached to it—and a lot of emotions.” He saw a field where rigorous quantitative research could have a significant impact.

His timing was fortuitous. Michael Lewis’ book Moneyball, published in 2003, revealed how the Oakland Athletics were using analytics to make baseball decisions, upending an industry that had long relied on observations and gut instincts. “Moneyball came about in part because the whole world was moving toward a quantitative analysis of things,” Winters says. “Education research has definitely had that trajectory.”

Still, the idea that analysts who hadn’t played professional baseball could offer new insights on the game wasn’t popular with many baseball lifers and the infusion of data experts in education was met with similar skepticism. When Winters, who has never taught in a K–12 classroom, submitted one of his first studies—on a Florida policy of retaining students in third grade based on test scores—to Educational Evaluation and Policy Analysis (EEPA), a leading journal, he got a terse response. “The letter from the editor was pretty nasty,” Winters recalls. “They pushed back on the idea of using quantitative methods to study that issue—of doing it broadly instead of within a specific context.”

That was the mind-set in the education policy research world at the time, Winters says. And though he still sees that sentiment in some areas, much has changed. “Today, EEPA is almost an economics journal—it’s primarily quantitative based,” he says. “I’m on the board of that journal now.

“I don’t think it’s especially important to have worked in K–12 in order to study it. The more context you have helps you develop research questions, but doesn’t necessarily help you to answer those questions very well.”

So when Winters decided to go back to college to further his expertise in the quantitative analysis of education, he didn’t consider education programs. “I looked around and asked, ‘Who’s doing what I think is the most interesting, compelling work?’ And it was labor economists.”

In 2008, he graduated from the University of Arkansas with a PhD in economics. “I didn’t get the PhD for a love of microeconomics,” he says. “It was really in order to learn how to study schools better.”

After a brief return to the Manhattan Institute, Winters moved into academia where he could focus on policy research and publishing in academic journals. “I want my work to be impactful both within the academic literature and also within policy conversations,” he says. In 2010, he joined the faculty at the University of Colorado–Colorado Springs College of Education, before moving to BU Wheelock in 2016.

A New Frontier of Data

“Early in my career, data was really difficult to come by,” Winters says. But in today’s data-rich era, quantitative researchers have access to more information than ever before, allowing for a breadth of research that would have been unimaginable two decades ago. Standardized test results now span several decades and provide a core set of data for researchers. Increasingly, that information can be matched with attendance and discipline records, courses taken, and even survey results to further our understanding of education policies.

As these pools of data grow, researchers can analyze students and teachers in new ways and over longer periods, strengthening findings or revealing new trends along the way. For example, with some states considering making personnel decisions based on teachers’ valued-added measures—essentially their ability to elevate students’ test scores—Winters looked at data from Florida to project the impact of such a policy. What he and his coauthor discovered was that students of teachers who would have been dismissed based on their value-added score performed worse than other students. Another benefit of more data: those same students can be followed through high school graduation. Now, some states are tracking students beyond school, meaning that researchers could, for example, connect teachers to the earnings of their students years later.

Winters also focuses on school choice and accountability. “There have been a lot of policies developed over a long period of time that don’t seem to be aligned with high productivity for the public school system,” he says. “Some of those policies have had reasons underlying them, but I think it’s fair to say if we were going to start a school system over, you probably wouldn’t replicate what we have now.”

“Ideally we would have a public school system that is both flexible and accountable,” Winters says. That idea guides much of his research as he analyzes the effectiveness of new policies or debunks arguments against reforms.

For a 2018 paper for EEPA, Winters turned to teacher pension systems and found that young, risk-averse teachers would prefer a retirement plan that provided consistent growth across their career rather than a more traditional pension system tied to their final salary. “Teachers are the most important components within the school system,” Winters says. “I think that schools should have flexibility on how they work with teachers, how they retain the best ones, and remove the least effective ones.”

One data set that Winters has spent more than a decade studying comes from Florida’s public school system. The state was the first of 19 to implement a policy of retaining students in third grade if they failed a reading proficiency exam. Winters wondered whether that strategy helped the students. With access to test results and students’ school records in the years after third grade, Winters tracked their progress and saw that retained students made immediate improvements in their reading and math skills. He’s revisited this data multiple times since first publishing his results in 2007, tracking those same students into high school. Those third-grade gains helped the students reach high school but, as Winters and his coauthors found, did not make them any more likely to graduate from high school. Those results—early gains that taper off—might provide future avenues for research, and could help inform retention policies.

Rising Demand

Just as other baseball teams quickly copied the Oakland A’s stat-based approach, quantitative research has spread through education.

“States and districts are adopting more of the Moneyball-style of using research to understand their own school systems,” Winters says. “There’s an ever-increasing need for people doing this sort of work.”

So Winters and his BU colleagues Stephanie Curenton, director of the Ecology of School Readiness Lab, and Nate Jones, an associate professor of special education, set out to design a program that will prepare BU Wheelock students for research-intensive careers, whether they’re performing their own studies or synthesizing the work of others. Students in the program will take qualitative and quantitative research methods courses, while exploring their interests, such as pre-K, STEM, higher education, or other fields.

“Research can be a really strong tool—if you understand how it works, what its limitations are, and what its benefits are,” Winters says. “Having more of those people working at high levels in government and in education-related areas can only make the system stronger.”

Taking the politics out of education reform

Marcus A. Winters is an associate professor and chair of educational leadership and policy studies. He’s also program director for policy, planning, and administration. In 2019, he helped design and launch one of BU Wheelock’s newest programs, a master’s in education policy studies.

“Moneyball came about in part because the whole world was moving towards a quantitative analysis of things. Education research has definitely had that trajectory.”

ONE LESSON

“Quantitative research has spread through education with more states and districts using a Moneyball- style approach to study whether they’re making the right policy calls—or not.”