Invasion of the Student Body Snatchers: A Statistical Lie Exposed

December 22, 2005

Picture a sleepy little public school at the end of your street. The students at that school, like all such schools, take standardized tests each year, and last year was no different. The student scores were all averaged, and the school was recognized for its level of academic performance.

But prior to this year's round of standardized testing, something shocking happened. Students were replaced. Not just a few, or even half, but well over 90% of the entire student body was replaced by a different group of students, a group which had a worse history of academic achievement than the original group.

This new, alien group of students (or pod-students) took the test in place of the former group, and no one in authority said anything, it was as if it were just business as usual.

When the scores came in--surprise--the average score was lower than it had been the year before. Instead of recognizing the statistical mismatch, the school was promptly chastised for declining in academic performance!

Believe it or not, this is a true story. What's even more bizarre is that the story is replicated like those creepy pod-people every year in states and cities across America, notably in every public and charter school in Pennsylvania.

What is going on here?

It's easy, really. Schools are measured by how their students do in certain targeted grades. In Pennsylvania, for example, schools are judged mainly by the PSSA scores of their 3rd, 5th, 8th, and 11th graders. The other grades are not assessed at the state level.

Every year a different crop of students is measured, as each passes under the magnifying glass. These students are inherently different, so of course there will be some variablility of the class's scores as a whole. Each year the scores will either tick up or tick down, but rarely will stay exactly the same.

For example, take 300 students in 11th grade, test them in reading, writing, and math. The following year, take 300 different 11th grade students and test them again. What are the chances that the measured average will be exactly the same? Somewhere between Slim and None, and Slim just left town.

Yet based on this simple (and simplistic) comparison, schools are declared to be "improving" or "declining." The measurement is a very important indicator for whether a school is meeting Adequate Yearly Progress (AYP) for compliance with No Child Left Behind.

It's easy to blame the Feds, although the ire would be misplaced. According to the Federal Department of Education, under No Child Left Behind, states are supposed to be measuring every child, every year.

Measuring a group of students, and finding the mean of the data, is not a perfect measurement, as demonstrated by repeating the exact same measurement (what are the chances that every student will get the exact same score?) or by dividing the class into two random groups, and taking the mean of each. Chances are very good the means won't be identical.

Statisticians have known this for years! But what is shocking is that the folks doing the comparisons of these schools by comparing means of different populations of students either don't know the proper way to deal with such data, or don't bother. After all, it is very easy to just compare means.

Easy, but not terribly valid.