In the beginning
My first encounter with the scientific method was in grade 4. Purpose? Check. Hypothesis? Check. Procedure? Check. Results and possible sources of error? Check. Conclusion? Check. Congratulations! You’ve just done science! The approach seemed elegant and had rigour. Scientific rigour, if you will. The beauty of it all was that an 8 year old can grasp it.
Flash forward to the present. I hear phrases such as, “Oh, I don’t buy the results of that study” or its equally-evil twin, “Yeah, it’s true. I read it somewhere.” Do you see what I see? Alack! A lack of understanding of the tenets of the scientific method. Or perhaps some laziness in speech — that is more forgivable. However, it often seems as though it is the former.
“I don’t buy that”
Certainly, one can express disbelief in the results of a study1 , but since the result is the outcome of performing some measurement, expressing disbelief is the same as saying the data has been fabricated. Notwithstanding the existence of academic dishonesty, what the speaker probably means is one of four things:
- “I think the results are a statistical anomaly”,
- “I think the procedure, execution, or analysis is (more) flawed (than usual)”,
- “I think the conclusions drawn from the analysis are incorrect”, or
- “I haven’t read the paper and/or I either don’t understand the scientific method or like to haphazardly apply belief in the scientific method to whenever the conclusions suit my a priori worldview.”
Hint: If you ever need to hazard a guess about what most people mean, you can bet dollars to doughnuts on the last one and end up with enough fried snacks to make Homer Simpson jealous.
Yes, certainly #1 is possible and my most frequent objection to a paper is #2. That’s why theory-based replications are important. In good publications, #3 seems to be less prevalent (yes, papers with issues #2 and #3 definitely make it to print and, probably, the converse is true). When someone means #4, that means somewhere, somehow, someone probably made some kind of mistake between grade school and the present.
Bonus hint: If you’re still betting dollars to doughnuts, you might want to bet double or nothing that the utterer of said phrase also does not fully grasp the difference between correlation and causation nor how to design an experiment to distinguish the former from the latter.
Acceptance of the scientific method entails acceptance of conclusions in studies where #1, #2, and #3 do not apply. Science is not a buffet: one does not get to pick and choose what conclusions to believe on the basis of prior beliefs; I’ll certainly defend your right to express your belief that the degree of certainty expressed in a conclusion has been overstated. I’ll listen to you, even if your views contradict mine2. But only if you promise me that you didn’t really mean #4.
There is a fifth alternative: the speaker believes the scientific method is flawed. This is a completely valid position; some epistemological schools of thought adopt logical frameworks that deny the validity of empiricism. But I think most educational systems teach the scientific method and logical positivism, so #5 does not apply to the orthodox masses. Instead, the most notable group of people who seem to hold the scientific method in contempt are some politicians who instead subscribe to “science by democracy”.
“I read it somewhere”
I’m sure you did read it… somewhere. I don’t want to get “all preachy” about it, but without getting into the ability of writers for newspapers and magazines to “get science right” and journalistic integrity, let alone propaganda piffle, I will assert that it is better to read the original publication than second hand sources. That is, synoptic science is no substitute for Q. Secondary and tertiary sources certainly have their uses, but whenever it comes to important matters of science (whatever they may be), they should be treated like a Twitter tweet . Even a spectacular science writer cannot be expected to cram the contents of a journal article into the confines of a newspaper column; at best, all the writer can really do is let you know that some paper exists and give you enough information to determine if it is worth reading in full. It’s like playing broken telephone with a goldfish, if you will. Imagine if Fermat had tried to summarize his proof for Fermat’s Last Theorem into the margin of his book. The important points may be expressed, but the devil is in the details.
Without the fuller picture provided by the original paper3, one misses a chance to critique the study’s procedure and conclusions. As much as we in the ivory might loathe to admit, papers do get published with flaws; reviewers are only human. This side of academia notwithstanding, the original paper contains details that show the scope of the conclusions; usually, a summary will state things more broadly, making the conclusions sound broader than they are. This is why it is only after reading the paper that one can decide what conclusions to put stock in.
Now, I have to admit that I’m often guilty of exhibiting “I Read It Somewhere” (IRIS) syndrome. Unless the conclusions are interesting or important, I rarely do my homework by finding the original paper. However, even if I can’t remember the source of what I read, I make the distinction between having read a paper and having read about a paper for the astute listener. Some of my colleagues help maintain a healthy, IRIS syndrome-free environment by asking for the source of some information that pops into my head and hunting it down on-line. This fact- and source-checking also fixes numbers that have become jumbled in my head such as when Jorge went and checked my faulty recollection that Grand Central Station in New York had 200 000 lightbulbs. Upon some quick pondering, I realized this was absurd and we discovered that there are in fact about 4 000 lightbulbs in public areas of Grand Central Station and that 200 000 was the estimated annual savings (in dollars) of replacing them with compact fluorescent lightbulbs. I just thought it was neat that it takes six full-time employees to keep the lights on!
Some people might be getting their facts wrong; no big deal, right? Well, perhaps it isn’t when it comes to figuring out how to eat healthily and which toothpaste will result in the brightest teeth, but what about making decisions that affect everyone? For example, wasting tax dollars on passing/enforcing mundane laws prohibiting hand-held cellular phone usage while driving: even if enforced, such laws have little public benefit since hands-free cellular phones result in the same degree of danger (Strayer et al., 2003)4. And what of the adoption of policies that not only waste resources and give a false sense of security but are also counterproductive like increasing the lengths of prison sentences for criminal offences (Gendreau et al., 1999)? And that’s without getting into issues that have players with extremely deep pockets to sow discord5.
Do I think the end of the world is coming? Extrapolation from the evidence suggests that, despite warnings from scientists physical or social, we’ll continue to succumb to what psychologists call confirmation bias and believe whatever we already believed. The result is that one day, even if most perceived threats are the results of statistical anomalies, a threat will emerge that we will ignore at our own peril, be it climate change, societal collapse, or zombie attack. However, there are few certainties in science, so I’ll choose to embrace my optimistic inner 5-year old: I don’t buy that result.
- I’m making a distinction here between result and conclusion: the results are the results from measurements performed while performing the procedure. This is distinct from the set of conclusions, which involves interpretation, drawn from the results. [↩]
- Actually, I love reading/hearing new opposing points of view presented cogently and politely; I’ve had parts of my worldview shattered more than once this way. And even if your views fly in the face of the evidence, as long as it’s possible, I’ll support your holding of that view. [↩]
- I say “fuller” because even a journal article with no length restrictions has omissions, let alone a 10-page conference paper. [↩]
- In fact, holding a block of wood to your head while driving is safer than talking on a hands-free device (Strayer and Johnston, 2001) [↩]
- Exxon Mobil’s annual profits exceeded those of AT&T and Verizon Communications combined in 2008. With rising oil prices, this may not have been true in 2009. [↩]