Scoring the FCAT: Calling all rocket scientists and psychometricians!

March 03, 2004

One aspect of modern schooling which we find frustrating is the irresistable urge to make things more complicated. Which is a crying shame, because education reform is not rocket science.

Today we turn to a favorite subject of Number 2 Pencil (in a shameless attempt to steal thunder), Florida's FCAT. (We typed "FCAT" into Kimberly Swygert's search engine and smoke came out of our CPU.)

Lakeland, Florida's Ledger has a story critical of the FCAT's scoring methodology. Initially, the criticism centers on the fact that some questions are weighted, thus two students can get the same raw number of answers correct, but end up with different scores.

But FCAT officials with the Florida Department of Education say it's more complicated than that.

"The scoring is the most fair and widely accepted method among . . . professionals in the field because it incorporates so much more about the students than just a simple number (of correct answers)," [Cornelia] Orr said.

By "professionals in the field" we assume he means psychometricians. But the phrase "incorporates so much more about the students" sounds a bit Orwellian.

The program factors in three things about every question for each student: the question's difficulty; how many students usually try to guess its answer; and the distribution of students who know the skill involved and those who guess right or wrong.

This sounds horribly complicated. Exactly how do they determine all these things? How do they figure out how many students "usually" try to guess an answer? Wouldn't this mean the question is worded in a certain way so as to be possibly misleading? (In other words, a hard question which looks like an easy question.) How exactly did they separate students (in their pilot testing phase) who got a question right due to skill from those who got it right due to a lucky guess?

For example, if a student named John and his classmate Mary answered 20 questions correctly, their scale scores could be different. John correctly answered questions deemed easy, so his final score is lower than Mary's, who correctly answered more questions considered difficult.

Um, hello? Can someone explain to us why Smart Mary skipped the easy questions? We don't buy this argument for a second, but truth be told, our quarrel is not with weighted questions. However, we could make the test simpler still (remember, standardized testing is not rocket science) if each correct answer was worth a point. Period. End of discussion. (They'll never go for such a crazy idea.)

But here's where it turns creepy:

Even if John answered a few harder questions correctly, the computer would deem these answers as lucky guesses based on his pattern of answers. Lucky guesses carry a lesser value than those calculated correctly.

In other words, since John's belt doesn't go through all the loops, he doesn't even stand a chance answering hard questions! He's still going to lose points, even if he gets them right! The kindly computer will simply assume that he guessed. (Memo to John: Bend over.)

We fully admit we're not experts at testing, but we'd hope our BS detector still works out here in these waters.

Our questions are simple: Why all the drama associated with psychoanalyzing each question and each student? Is this a psychology test or a skills test? Why all the complicated (and unexplainable to parents and students, therefore they aren't told) blather about point shaving due to "possible" lucky guesses? Only the student and the Man Upstairs knows if any given answer was a lucky guess!

What especially irks us is that this "problem" (of trying to counter lucky guesses) was solved years ago, and in a manner that is immediately understandable by students and parents alike!

We're speaking of the College Board's SAT, and the way they deal with wrong answers.

In a multiple-choice test (which we infinitely prefer over any subjective human-scored assessments), technically a student could guess his way though an entire test and get around 25% (assuming 4 possible answers per question) just from brute luck. In any group of four questions, chances are that one will be right (the lucky guess) and three will be wrong (the unlucky guesses).

The SAT folks figured that if a third of a point was subtracted for each wrong answer, then an all-guessing strategy would be a wash: One point minus three thirds, for a sum of zero.

But what's brilliant is how easy this is to communicate to students. Just tell them that if they don't know the answer, to leave it blank, without guessing! If they have narrowed down the choices (like 50/50 on that insipid Who Wants to be a Millionaire), then maybe they can make an "informed guess." But if they don't have a clue, leave it blank.

Returning to our group of 4 questions, say a student knows a little bit about each question, and is thus able to reduce it to a "50/50" for each, and makes four informed guesses. On average, this student will get two right and two wrong, and the SAT will score this with one and a third points (Two minus two thirds).

We consider this 1.3 points to be like "partial credit" for knowing partial knowledge. Contrast this with the student who knows all four correct answers, who'll get four points, or with the student who knows nothing, who will get zero points, whether he guesses or not.

We think that's a pretty simple system. Easy to communicate, easy to score, and no psychoanalysis.

But returning to Flordia's FCAT, we feel their method of scoring, while it may be oh-so-sophisticated, using cutting-edge techniques and all that, is increasing the drama associated with the test! Our friend Kimberly's already got her hands full trying to convince folks that rational, reasonable standardized testing is not one of the Four Horsemen.

But since the FCAT will help determine if a student even receives a diploma at the end of high school, we can't help but think a mind-reading, point-shaving computer isn't the best way to go.