How can you identify a truly ineffective school? To paraphrase retired Supreme Court Justice Potter Stewart, you know it when you see it.
But state policy requires greater specificity than that. And in Massachusetts, we identify the lowest performing schools primarily by using scores from the state standardized test — the Massachusetts Comprehensive Assessment System — administered annually to students in most grades.
Test scores, as scholars have demonstrated, constitute a highly imperfect measure of school quality. But there are better and worse ways to interpret these scores. And currently, Massachusetts uses the worst possible method — one that tells us more about a school’s socioeconomic status than it does about its quality. Yet in a few weeks, when the state board of education next meets, there will be a chance to change this.
The matter of how a state identifies the “lowest performing” schools is a high-stakes enterprise. After all, labeling a district as such can lead to the flight of quality-conscious parents—weakening the district’s capacity and increasing segregation by class. And additionally, schools identified among the lowest ten percent of performers face the charter “cap lift” provision, allowing the state to send up to 18 percent of the district’s funds to charter schools — twice as much as in other districts. When the cap is raised and more charters are granted, students leave, budgets are cut, and schools can close. While some families now have a new choice, others find their chosen school closed.
To be clear: some districts really are ineffective. And no one wants children to be trapped in failing schools. Yet the simple truth is that current approaches for measuring effectiveness are methodologically weak and ethically dubious.
Standardized test scores, which constitute the lion’s share of how we evaluate school effectiveness, are highly problematic. Standardized tests capture a narrow slice of life in schools and reflect only a fraction of what the public values. They are designed to be time efficient and cost-effective rather than to align with what we know about cognitive development. And they are subject to gaming.
Perhaps most troublingly of all, test scores correlate very highly with socioeconomic status. What that means is that schools serving low-income and minority students are disproportionately identified as ineffective. And the impact is that resources can be pulled out of good schools that just happen to be working with high-needs students.
So if we are going to use standardized test scores — and that’s a big “if’’ — then we need to use a fairer measure.
That fairer measure is growth, which even the state Department of Elementary and Secondary Education recognizes. According to a 2011 DESE Interpretive Guide, “Research shows that there are correlations between a student’s demographic group and their performance on the MCAS. Is the same true with growth? Not necessarily.”
MCAS growth scores — Student Growth Percentile — show how much progress students are making in raising their scores. Based on standardized test scores, they do remain a limited measure of school quality. But growth calculations are far preferable to absolute scores because they better reflect the contribution of the school, and not just a child’s socioeconomic status.
In the past few months, the state board of education moved to include growth scores as one fifth of the calculation that determines the charter cap — bringing it in line with the calculation used for other state accountability purposes. And there was even discussion of moving further in the direction of growth scores. Yet at last month’s board meeting, the board postponed a decision after resistance from charter school advocates, who fear losing the access they’ve long enjoyed. If growth scores are more heavily weighted in the calculation, charter expansion will be directed toward smaller, higher-income districts where parents often assume that traditional public schools are of adequate quality, and where they may not be attracted to alternatives. Their resistance, then, is more about self-preservation than it is about serving students in the weakest schools. But given their political clout, we are now regressing toward a measure that inaccurately and unfairly identifies ineffective schools.
Neither of us has much confidence in the exclusive or near-exclusive reliance on test scores in any permutation to measure something as complex as school quality. In fact, what we favor is a multi-dimensional model that goes far beyond such narrow measures.
Still, if we are going to rely on scores for fundamental decisions that impact so many lives, we have to do so in the fairest way possible. And that means adopting a measure that won’t so blatantly disadvantage schools and districts working with high-needs students.
In changing the way it assesses school effectiveness, the state should adopt growth scores today. And recognizing the limits of what standardized tests measure, they should begin working tomorrow on a method of evaluation that truly captures school quality.
Jack Schneider is an assistant professor of education at the College of the Holy Cross. Senator Pat Jehlen of Somerville is vice-chair of the Senate Committee on Education.