Myth: Education Research can be trusted

Many people think all Education Research is valid.

You could commission any study to say almost whatever you wanted. For example, in the 1940s and '50s cigarette advertising used to quote doctors and medical research up and down showing how beneficial smoking was for you.

Were they lying?

No, because they simply focused on the calming effect of nicotine in the bloodstream. They studied people who were nervous, jumpy, and irritable and found that they were soothed when they lit up. Smoking had "taken the edge off."

Of course we all now know that smoking has some not-so-wonderful effects on a person's lungs and circulatory system. And while we are still aware that smoking relaxes people (health benefit) we know the corrosive effects on one's heart and lungs (health risk) are vastly more devastating.

The same goes for educational research.

Some is good, some is bad. It all depends on what research question you ask.

You could ask all sorts of "wonderful" questions about "wonderful" educational programs and find research that will confirm your diagnosis of "wonderful."

Confirming questions of this sort tend to be affective in nature. "How did the student feel?" "Does the student like it?" And our favorite, "Does it improve self-esteem?" (We certainly are obsessed with self-esteem!)

But we challenge you to instead give more weight to research that is effective in nature, especially with regards to academic achievement that can be measured on standardized tests. Yes, we know, standardized tests are said to be evil. But they often give a disinterested, objective view—the computer scoring the test does not care a whit what program the kids are on—which can be very helpful in determining a program's success.

Another thing to consider when looking at so-called educational research is to consider whether it was a pilot study (in an experimental setting) or a longitudinal study based on real classrooms. For example, the Hawthorne Effect is a well known phenomena, showing that any changes to a system (plus increased scrutiny) can lead to an increase in the variable you're studying.

One classic study examined workplace productivity. When they painted the walls a different color (let's say orange), there was a surge in productivity. But when the walls were painted again, restoring the original color, productivity surged again. It is interesting to note that if they stopped the study after the first repainting, they would have drawn an entirely false conclusion about orange paint.

Oftentimes trial-period educational studies involve a similar surge in productivity simply due to the newness of the program under scrutiny. Plus, people tend to be "on point" with researchers crawling all over one's classroom.

Rather than risk the Hawthorne effect in a pilot study, it is better to perform longitudinal research, measuring progress year after year.

Many research projects look at a variable, for example two varieties of Math textbook, and measure pre- and posttest scores for one class with Book A and another class with Book B. If the achievement in Class A with Book A were higher, it is easy to say that Book A caused the increase in achievement.

Hopefully you're sharp enough to realize that any differences in Class A and Class B could well be due to the teacher rather than the math book! As the story goes, a master teacher can teach out of a phone book.

This brings us to another point. Correlation does not prove causation. This trips a lot of people, including researchers who should know better. Correlation doesn't prove anything, for there could be another variable (like the teacher in this example) that is really causing the change.

To take an example from real life, let's say for every day in July you counted the number of kids in a swimming pool and the amount of ice cream sales at the corner mom-and-pop. You'd notice that as there were more kids swimming, there were also more ice cream sales. When one goes down, so does the other.

It doesn't take a rocket scientist to figure out that eating ice cream doesn't cause kids to go swimming, or vice versa! Both are affected by the heat on a summer's day.

Let's discuss a few other factors which can hurt the rigor of a study. One is the use of different pre- and posttests. It should go without saying that the pre- and post-test should be the same, but you'd be surprised how often researchers like to mix-and-match to "prove" the desired outcome.

One example is to use an effective pretest (such as a standardized test score) and an affective posttest (such as a measure of self esteem).

We confidently bet you a nickel that the "wonderful" program will come out ahead in that horse race!

One final note. Did the researchers reveal where the study took place? If the location is "undisclosed" then you can't be sure the researchers didn't pull the data out of their, um, Erlenmeyer flasks.