No Child Left Behind and Growth Models

December 11, 2005

Several weeks ago Education Secretary Margaret Spellings announced a new pilot program for No Child Left Behind, permitting states to use growth models in their task of measuring student improvement.

Many states measure student improvement year-over-year, comparing this year's 11th graders with last year's 11th graders, for example. This is sometimes called a cohort comparison, for it measures different groups, or cohorts, of students. It doesn't appear any statisticians had anything to do with this method of analysis, since it goes no further than middle-school-level statistics: computing the mean of a group of scores. If this year's students have a mean different than last year's students (almost a sure bet, since they're different students) then school districts proclaim "the school is improving" or "the school is declining."

Growth models in comparison are a huge improvement. These measure last year's 10th graders and compare the scores with this year's 11th graders, essentially measuring the same group twice, over time.

What is confusing to us is why the Feds have to announce they're now permitting states to use growth models. Why haven't states been using these models all along? Anyone who spends even a little bit of time analyzing the problem should come up with the same conclusion: the only valid way of measuring school growth is to compare the same students over time, rather than different groups of non-random samples (cohorts).

One possibility is that states have fallen under the spell of using tests that are longer, more comprehensive, and have more open-ended questions.

These tests take a lot of time to administer, and are very expensive to grade. Thus states like Pennsylvania only test students in the 3rd, 5th, 8th, and 11th grades, using the PSSA. They then have no choice but to compare cohorts, since there's no way of measuring year-over-year growth.

What do we recommend?

First of all, dump these lengthy, expensive standardized tests! In our view, these long tests, with their open-ended (thus subjective) questions, were designed more to mollify critics of standardized tests rather than to improve the data gathered.

In Pennsylvania's case, this would mean dumping the PSSA, which is the state's "let's reinvent the wheel" standardized test. This is probably a sacred cow, but it needs to go. It's not standardized, it does not show "grade equivalence" in the scores (since it's not standardized), and it's basically a poorly-written test, with all the hallmarks of a test "written by committee."

With the money saved by not administering these more expensive tests, states could afford to use a tried-and-true test like the Iowa Test of Basic Skills, and test each student every year. The Iowa test is relentlessly standardized, and has been used for years and is thus time-proven to be a reliable measure. Plus since there are no open-ended questions the results come back very quickly (and quick turnaround helps schools help kids).

Which brings us back to No Child Left Behind.

In the Federal documentation to help States comply with the law, there is a "road map" for implementation, with the following quote:

Assessments in Grades 3-8: What gets measured is what gets done. States must test all students annually in reading and math in grades 3 through 8 and once in high school by the 2005-06 school year – not every other year or every other class, but all students every year.

Emphasis ours.

We'll write more on "test each student every year" and some of the analyses that can be performed with the data, in a future article.