Wonk time: An insider’s critique of new Minnesota school rating system


St. Paul accountability chief Matt Mohs isn’t sold yet on the new Minnesota system designed to identify schools that are succeeding and failing.

As the head of academics and Title I programs for St. Paul Public Schools, his concerns revolve around peculiarities in equations, wonky details that only a real insider would catch, but ones that impact whether or not schools are recognized for true successes and true failures.

This article is part of a series, Testing: Who wins, who loses with high stakes, standardized testing in Minnesota schools

His biggest and wonkiest concern is that the correlation between school demographics and Minnesota’s new Multiple Measurement Ranking (MMR) system is a little too predictable. He worries that the new school accountability system just might be better at identifying challenging demographics than it is at recognizing high-functioning schools.

But he’s not sure yet. This is the first year that Minnesota schools are functioning under the state’s No Child Left Behind waiver. So far, the new system is proving to be a strange animal and one that education leaders say still needs time to be house-broken.

“You don’t want to trade one bad system for another bad system, and I’m not saying that the MMR system is bad, but it’s just too early to tell,” Mohs said. ““We have to be careful not to over-interpret it.”

A system too simple traded for a system too complex?

In rejecting the obviously flawed AYP system [see sidebar], Minnesota welcomed a new scoring system so complex that a non-expert would have a hard time determining what a score says about a school.

Its complexity is part of what many pundits are celebrating.

No Child Left Behind’s Adequate Yearly Progress system was so simple it was stupid, because it defined school success based solely on high test scores.

St. Paul’s Maxfield Elementary school principal Nancy Stachel said the old system failed to recognize students at her school who started the year more than two years behind and finished only half a year behind. Low test scores masked high growth.

The new scores equally count student growth, test scores, the school’s contribution to the state achievement gap and, at high schools, graduation rates. The system ranks schools based on their composite score. The top-scoring 25 percent of Title I schools are labeled “reward” or “celebration” schools and the bottom 15 percent are labeled “priority” or “focus.”

Scores fluctuate wildly

In May, the Minnesota Department of Education released the first batch of MMR results, from the 2010-2011 school year.

The results so far? MMR labels 213 Minnesota schools as underperforming, compared to more than 1,000 under the AYP system. Urban, high poverty and non-white schools still scored lowest.

Strangely, the second batch of numbers, released in August for 2011-2012, looked much different from the first batch. Many schools saw jumps or drops in their MMR score of 10, 20 even 40 percent. Many of the score fluctuations in the Twin Cities erred downwards.

It’s unlikely that student performance at all those schools changed dramatically between the two years. According to Mohs, the fluctuations may have had to do more with the mechanisms of the calculations.

Where MMR came from

The new system came thanks to Congress’s failure to reauthorize or reform No Child Left Behind, the federal legislation that required states to punish schools not making Adequate Yearly Progress or AYP.

Under No Child Left Behind, schools where kids scored low on tests had to set aside funding for unproven tutoring programs and for buses that would transport kids who chose to switch to a different school. If a school continued to score low for several years, the school would have to fire and rehire staff, change into a charter school, close or undergo some other harsh restructuring.

This was all in an effort to move towards the mathematically impossible goal of every single student scoring proficient in math and reading by 2014.

The system was nearly universally reviled.

As a compromise to states impatient with Congress’s inaction, President Barack Obama introduced No Child Left Behind waivers, which would free schools from some of the legislation’s most onerous requirements.

Minnesota applied for and won a waiver last year.

The waiver did away with the impossible 2014 goal, and replaced it with a goal of cutting the state’s achievement gap in half in six years. It eliminated the restructure requirements, giving struggling schools more flexibility to create turnaround plans. And it introduced a new school scoring system called MMR.

It also required states to evaluate teachers at least partly based on student test scores. A state working group is still developing that evaluation.

The Wonky Stuff

Mohs worries that the MMR depends too heavily on peer-to-peer comparisons. It grades on a predictable curve.

A third of an elementary school’s score still depends on test proficiency.

The system divides each school’s kids into subgroups determined by ethnicity, language learner status, qualification for free or reduced lunch, and special education status. A subgroup must have 20 students to be counted.

For a school to get a high proficiency score, a targeted proportion of each subgroup has to pass the test.

The thing is, about half of Minnesota schools get all their proficiency points in part because they don’t have many subgroups — they’re not very diverse.

That skews the curve against the other half of Minnesota schools, which get their scores chopped up into subgroups that are examined individually. If one 20-person subgroup in a school does poorly, the school can end up with a low MMR score.

Mohs worries that there’s potential for schools to be disproportionately penalized for the test scores of one group of kids. The system might not be sensitive enough to recognize schools like Maxfield that he said are doing impressive work.

A skewed curve

The system might be telling us what we already know.

It’s undeniably more challenging to effectively educate students who are experiencing stress related to poverty, who have a disability, or who are learning English. Schools with those groups start off at a testing disadvantage when compared to schools with a less challenging student body.

It’s predictable that the most affluent, least diverse schools would sit at the front end of the testing curve and the poorest, most diverse would sit at the back end, and it doesn’t necessarily indicate high performing and low performing schools.

“I definitely think the makeup of a student body will have an impact on a score,” said Minnesota Department of Education spokesman Keith Hovis. He said that’s why it’s so important that the new system provides more flexibility for schools to cater turnaround plans to their particular circumstances.

To get off that predictable curve, Twin Cities schools would have to produce more than a year’s growth in students who start out behind.

That’s why so many people are celebrating the fact that a third or a fourth of the MMR score is based on student growth. But that measure is imperfect, too.

Growth, graduation and the gap

The growth measure is calculated by looking at a student’s score the previous year. If they earned a 50 last year, their score this year would be compared to all students in the state who also earned a 50 last year. They either have above average or below average growth.

The growth measure puts students learning English, alongside students who had a bad day, alongside students who have developmental and cognitive disabilities. Slow growth for one student might be impressive growth for another, but the scores are unable to reveal that.

The growth measure only counts kids who attend a school for a full year and kids who have Minnesota test scores from the previous year. At Maxfield, that means a large proportion of the student body doesn’t get counted.

The achievement gap measure is tied to the growth measure. Schools are scored based on how much students in their lower-achieving subgroups grow in a year. Their growth is compared to their opposite.

Language learners are compared to English speakers. Low-income students are compared to higher-income students. To get a high achievement gap score, schools with a large population of students in special education would have to show more growth in their students who have disabilites than students without disabilites show statewide.

The MMR score assumes that on-time graduation is always the right thing for a student, but many students with disabilities benefit from district transition programs, where they work on life skills as well as academics. In order to qualify for the programs, students forego graduation, even though they may have enough credits to earn a diploma.

A change in the test, a snowy day, a fight in the hallway, test anxiety — all can influence how well a student does on the test.

More from Mohs

“Unfortunately the law still is in place, so we weren’t able to make a clean break from the entire system and come up with a Minnesota system,” Mohs said. “States were left with trying to come up with a new system that would get them closer to what they want, while still meeting the requirements that are essentially in No Child Left Behind.”

“There were some instances where schools actually improved from the previous calculation of MMR in 2010-2011 to the 2012 results, but wound up having their MMR go down. It’s one of those things where you have to look at the context and the details,” Mohs said. “That’s where the policymakers are going to have to determine with their system, is whether or not they want to live with an accountability system that has a certain amount of ambiguity.”

“Quantitative measures in some ways are a lot easier to grab ahold of and talk about than the qualitative. We tend to see quantitative as objective and qualitative as not objective,” Mohs said. “There’s a lot of human work that goes into the development of standardized tests, and any human action is not objective.”

This article is part of a series on testing in Minnesota schools. The articles in the series, published during the week of December 10, 2012, are:

When tests tell teachers nothing: Special needs not met by standardized tests

At St. Paul’s Maxfield Elementary, low ranking masks real transformation

Winner or loser? Test scores for Minneapolis’s Kenny elementary school say both at the same time

Wonk time: An insider’s critique of new Minnesota school rating system

St. Paul teacher: Too much testing hurts kids who need help most

One teacher’s story: Testing makes teaching a nightmare

Leave a Reply

Your email address will not be published. Required fields are marked *