In the beginning
My first encounter with the scientific method was in grade 4. Purpose? Check. Hypothesis? Check. Procedure? Check. Results and possible sources of error? Check. Conclusion? Check. Congratulations! You’ve just done science! The approach seemed elegant and had rigour. Scientific rigour, if you will. The beauty of it all was that an 8 year old can grasp it.
Flash forward to the present. I hear phrases such as, “Oh, I don’t buy the results of that study” or its equally-evil twin, “Yeah, it’s true. I read it somewhere.” Do you see what I see? Alack! A lack of understanding of the tenets of the scientific method. Or perhaps some laziness in speech — that is more forgivable. However, it often seems as though it is the former.
“I don’t buy that”
Certainly, one can express disbelief in the results of a study ((I’m making a distinction here between result and conclusion: the results are the results from measurements performed while performing the procedure. This is distinct from the set of conclusions, which involves interpretation, drawn from the results.)) , but since the result is the outcome of performing some measurement, expressing disbelief is the same as saying the data has been fabricated. Notwithstanding the existence of academic dishonesty, what the speaker probably means is one of four things:
- “I think the results are a statistical anomaly”,
- “I think the procedure, execution, or analysis is (more) flawed (than usual)”,
- “I think the conclusions drawn from the analysis are incorrect”, or
- “I haven’t read the paper and/or I either don’t understand the scientific method or like to haphazardly apply belief in the scientific method to whenever the conclusions suit my a priori worldview.”
Hint: If you ever need to hazard a guess about what most people mean, you can bet dollars to doughnuts on the last one and end up with enough fried snacks to make Homer Simpson jealous.
Yes, certainly #1 is possible and my most frequent objection to a paper is #2. That’s why theory-based replications are important. In good publications, #3 seems to be less prevalent (yes, papers with issues #2 and #3 definitely make it to print and, probably, the converse is true). When someone means #4, that means somewhere, somehow, someone probably made some kind of mistake between grade school and the present.
Bonus hint: If you’re still betting dollars to doughnuts, you might want to bet double or nothing that the utterer of said phrase also does not fully grasp the difference between correlation and causation nor how to design an experiment to distinguish the former from the latter.
Acceptance of the scientific method entails acceptance of conclusions in studies where #1, #2, and #3 do not apply. Science is not a buffet: one does not get to pick and choose what conclusions to believe on the basis of prior beliefs; I’ll certainly defend your right to express your belief that the degree of certainty expressed in a conclusion has been overstated. I’ll listen to you, even if your views contradict mine ((Actually, I love reading/hearing new opposing points of view presented cogently and politely; I’ve had parts of my worldview shattered more than once this way. And even if your views fly in the face of the evidence, as long as it’s possible, I’ll support your holding of that view.)). But only if you promise me that you didn’t really mean #4.
There is a fifth alternative: the speaker believes the scientific method is flawed. This is a completely valid position; some epistemological schools of thought adopt logical frameworks that deny the validity of empiricism. But I think most educational systems teach the scientific method and logical positivism, so #5 does not apply to the orthodox masses. Instead, the most notable group of people who seem to hold the scientific method in contempt are some politicians who instead subscribe to “science by democracy”.
“I read it somewhere”
I’m sure you did read it… somewhere. I don’t want to get “all preachy” about it, but without getting into the ability of writers for newspapers and magazines to “get science right” and journalistic integrity, let alone propaganda piffle, I will assert that it is better to read the original publication than second hand sources. That is, synoptic science is no substitute for Q. Secondary and tertiary sources certainly have their uses, but whenever it comes to important matters of science (whatever they may be), they should be treated like a Twitter tweet . Even a spectacular science writer cannot be expected to cram the contents of a journal article into the confines of a newspaper column; at best, all the writer can really do is let you know that some paper exists and give you enough information to determine if it is worth reading in full. It’s like playing broken telephone with a goldfish, if you will. Imagine if Fermat had tried to summarize his proof for Fermat’s Last Theorem into the margin of his book. The important points may be expressed, but the devil is in the details.
Without the fuller picture provided by the original paper ((I say “fuller” because even a journal article with no length restrictions has omissions, let alone a 10-page conference paper.)), one misses a chance to critique the study’s procedure and conclusions. As much as we in the ivory might loathe to admit, papers do get published with flaws; reviewers are only human. This side of academia notwithstanding, the original paper contains details that show the scope of the conclusions; usually, a summary will state things more broadly, making the conclusions sound broader than they are. This is why it is only after reading the paper that one can decide what conclusions to put stock in.
Now, I have to admit that I’m often guilty of exhibiting “I Read It Somewhere” (IRIS) syndrome. Unless the conclusions are interesting or important, I rarely do my homework by finding the original paper. However, even if I can’t remember the source of what I read, I make the distinction between having read a paper and having read about a paper for the astute listener. Some of my colleagues help maintain a healthy, IRIS syndrome-free environment by asking for the source of some information that pops into my head and hunting it down on-line. This fact- and source-checking also fixes numbers that have become jumbled in my head such as when Jorge went and checked my faulty recollection that Grand Central Station in New York had 200 000 lightbulbs. Upon some quick pondering, I realized this was absurd and we discovered that there are in fact about 4 000 lightbulbs in public areas of Grand Central Station and that 200 000 was the estimated annual savings (in dollars) of replacing them with compact fluorescent lightbulbs. I just thought it was neat that it takes six full-time employees to keep the lights on!
Revelation
Some people might be getting their facts wrong; no big deal, right? Well, perhaps it isn’t when it comes to figuring out how to eat healthily and which toothpaste will result in the brightest teeth, but what about making decisions that affect everyone? For example, wasting tax dollars on passing/enforcing mundane laws prohibiting hand-held cellular phone usage while driving: even if enforced, such laws have little public benefit since hands-free cellular phones result in the same degree of danger (Strayer et al., 2003) ((In fact, holding a block of wood to your head while driving is safer than talking on a hands-free device (Strayer and Johnston, 2001) )). And what of the adoption of policies that not only waste resources and give a false sense of security but are also counterproductive like increasing the lengths of prison sentences for criminal offences (Gendreau et al., 1999)? And that’s without getting into issues that have players with extremely deep pockets to sow discord ((Exxon Mobil’s annual profits exceeded those of AT&T and Verizon Communications combined in 2008. With rising oil prices, this may not have been true in 2009.)).
Do I think the end of the world is coming? Extrapolation from the evidence suggests that, despite warnings from scientists physical or social, we’ll continue to succumb to what psychologists call confirmation bias and believe whatever we already believed. The result is that one day, even if most perceived threats are the results of statistical anomalies, a threat will emerge that we will ignore at our own peril, be it climate change, societal collapse, or zombie attack. However, there are few certainties in science, so I’ll choose to embrace my optimistic inner 5-year old: I don’t buy that result.
I’m not sure I agree with you –but maybe it’s just that I misunderstood your point.
On one hand, that 8-year-old little-scientist version of the Scientific Method is a simplification that works for elementary school lab assignments, but not for real acts of discovery, theory generation, or even validation of knowledge –it’s to science what arithmetic problems are to mathematical proofs. Scientists need to play dirty, and I don’t mean dirty in a ‘dishonest’ sense, but in a ‘think on your feet’ and ‘handle inconsistency’ sense. There is no theory that fully, completely explains an aspect of our experience without errors –tinkering with the theories and discriminating between signal and noise is part of the art of doing science. It makes science rather subjective, of course, which is a problem and a huge responsibility for us. But a naive-positivist epistemology hasn’t gotten us anywhere.
But perhaps we agree and you address that with your points 2 and 3. If that’s the case, I would only add that, in my experience, it’s rarely black-and-white: I don’t recall a single study, from anybody (including myself!), for which I completely buy all of its results. Some I dismiss entirely; an element of your points 2 and 3 appears in my judgment of all. The challenge, for myself as well as anyone else, is to learn to distinguish in ourselves between those concerns and the more convenient #4.
I think we mostly agree. The challenge for scientists is to design studies that allow interesting/useful conclusions (not necessarily practical ones) to be drawn and then to draw them without exceeding the limits imposed by the design of the study. And, to concisely sum up the thesis of this post, the challenge for scientists and non-scientists alike is to adopt a consistent view of science and not accept/reject a study on the basis of whether it does or does or does not conform to their beliefs.
The Strayer et al. paper bothers me. The sample size is quite small (41 people), with the only requirements being that they possess a valid drivers license and have corrected vision. There are millions of drivers in the state of Utah alone. I would have liked to see some discussion of why they designed the experiment as they did, and why they didn’t choose to control for manual-vocal multitasking ability, driving instruction, years of driving experience and driving record. Furthermore, their experiment only measures the ability of the drivers to mimic the behaviour of the pace car, not to avoid different types of potential accidents or drive in a safe or predictable manner. How many auto accidents are caused by a failure to avoid collision with another vehicle ahead on a highway? The experiment does not take into account any other types of driving accidents, such as vehicle-vehicle accidents, left-turn accidents, vehicle-pedestrian accidents, parking garage accidents, or single vehicle accidents. In their defense, the authors do admit a more holistic analysis of accidents in a related paper: Cell Phone-Induced Failures of Visual Attention During Simulated Driving.
As a thought experiment, I would very much like to have seen a third cohort where the driver was taking part in a discussion within the confines of the simulated vehicle. The authors previously reported that listening to the radio or a book on tape did not affect driving ability. If there were a significant difference between the conversing drivers (no phone) and conversing drivers (hands-free phone), then I’d be convinced that hands free cell phones should also be banned.
I’m not sure if you got a chance to read the Strayer and Johnston, 2001 paper that I included as a footnote; though it also has a sample size you may not like, it deals much more directly with the cognitive aspect through shadowing and generation tasks and I actually prefer it to Strayer et al., 2003. I’m not familiar with related epidemiological studies, so I can’t name any good ones, but I would assume they provide more direct evidence against using handsfree phones (I know they provide evidence against it, the question is just whether or not it is more direct).