The Scientific Method and Reading

January 30, 2004

Joanne Jacobs' "Feds for Phonics" post today really floored us. She noticed that out of one hundred thousand studies on reading that the feds examined in forming new guidelines, only forty used the scientific method.

What the . . . ?!

And here we thought we were exaggerating when we said that most educational research isn't trustworthy. We had no idea, really.

For those of you not in the know, the Scientific Method is actually really cool. Basically, the principle is to apply rigor to any study, in four general steps:

1. Analyze the problem (either by direct observation, or by reading up on it)
2. Form a Hypothesis (basically, make a statement which may or may not be true, something that can be possibly proved--or disproved--by experimentation)
3. Design and perform an Experiment (remember the best experiments have a control group which receives "no treatment" and can serve as a comparison for your experimental group)
4. Form a Conclusion (after analyzing your experimental data, does your Hypothesis hold up?)

If all this sounds quite obvious--that you should perform an experiment to find out the outcome--then you clearly are not familiar with education research.

A good example is exploring the hypothesis "Heavier Things Fall Faster."

Many folks don't know this, but for millennia the brightest scientific minds thought that heavier objects fell faster due to gravity pulling on them with greater force. The ancient Greeks had a lot of good ideas, but they did not have the Scientific Method! They believed that if you studied a problem (and thought about really, really hard) then you could deduce a conclusion that would be sound.

So without ever picking up and dropping an object, scientists believed (up until about the 16th century) that heavier things in fact fell faster.

It is only when one actually drops two objects (of similar density, such as a coin and a textbook) that the hypothesis "Heavier Things Fall Faster" is soundly disproved.

But if you just make casual observations, such as Aristotle did, you may notice that most objects fall faster than an autumn leaf, which in turn falls faster than a dandelion seed--hence the incorrect conclusion . . .

. . . Which brings us back to Educational Research. In our view, since there are so many ideologically-driven folks (not bad people, really, just misguided) who really want their progressive theories of education to be proven correct, we end up with a lot of really bad research.

One joke in the sciences is "plot the curve, then fit the data"--essentially saying the conclusion comes first, then the search for data that fits. We noticed this when we analyzed the research from the University of Chicago's Everyday Math program. What they'd done is pick one pre-test which showed a control group and an experimental group as being on the same level, then they picked a different post-test to show that the Everyday Math group did better.

If their post-test was in the affective domain--that is, measured "math appreciation"--then we have no doubt the Everyday Math group would score higher.

Another way to conduct bad research is by pure obfuscation. We noted an example of this when we examining a research study touted by proponents of small class sizes. In fact, the study had done four things with the experimental group--including using a rigorous curriculum--thus permanently obscuring any effects attributable to the small class size.

In all fairness, the Scientific Method is not the only way to conduct research, because it is not always possible to perform an experiment with real people. (Think of child seat belts: could you do an experiment where you have one group not wear seatbelts?) So yes, it is possible to take data that has already been collected and perform analyses on that.

But while such research may be valid, it is not as rigorous as an experiment performed using the Scientific Method. As you can imagine, it is far easier to manipulate the results of a study performed benefit of experimentation.

This is probably why there are a hundred thousand studies on reading, but only a handful which use the Scientific Method, and it is those studies which show that the explicit teaching of phonics gives children the best start to reading.

There's probably a good reason why it's called The Scientific Method, not The SkyBeatific Method.

We know which flavor we prefer in our educational research.