One book down, only about 998 to go! Wrong – Why Experts Keep Failing Us And How To Know When Not To Trust Them was a really interesting a read, a good start to the project 🙂 What shocked me most in the book was the finding that 2 in 3 high-end research papers are later refuted by other papers! As someone who will probably cite other scientists work a lot and perhaps pursue time-consuming projects based on this work, I found this kind of terrifying.
The author also discusses many of the ways science can go wrong, including: measuring indirect indicators of the value you’re interested in (shrinking tumor size as a stand in for cancer going away for instance); assuming animal studies will translate to humans; using poorly understood programs and statistical methods written by other people; not publishing negative results; “moving the goal posts” by reporting an observed effect as though it was what you were looking for; and throwing out inconvenient data. While the last one, throwing out data, sounds like the worst offense to me, I could see that bias creeping in unnoticed, as scientists might be more likely to check for broken equipment and other flaws if the data don’t match their expectations.
But scientists aren’t the only ones to blame for public misconceptions about research. Those of us reading about results in the news often expect simple, universal and pleasing answers to complex questions. The media increases the problem by trying to deliver what readers want, often presenting results as facts and not listing any caveats the researchers may have given. It is also common to present results as though they are immediately applicable to readers (“diet soda increases caloric intake!”) even though only animal studies have been done or causation has not been proven.
Take-away for Readers – So, what should you as a reader do to avoid being taken in by dubious claims? The author makes several points I very much agree with. First, avoid research which claims a simple answer with no caveats. Most of today’s complex problems don’t have a clear-cut universal solution. Second, check where the data came from; was this just an animal study? And finally, if a link is presented, think about other possible explanations. For instance, if a study told you that it always snowed on Monday, you might realize this was because the study took place in December or in the arctic. Be similarly questioning about less obvious connections like the diet soda example listed above. Perhaps people who drink diet soda do so because they think it makes up for eating more, rather than eating more because of drinking diet soda.
Take-away for Scientists: Don’t have expectations of your results! Don’t throw away data unless there is a sound methodological reason for doing so (broken equipment, etc). Don’t be afraid t publish negative results somewhere like PLoS which accepts them (did you know PLoS takes all methodologically sound papers? I didn’t!). Have the guts to expose people intentionally massaging the data. Don’t accept other researchers results as gospel. Be aware a paper you are citing may already have been refuted. And never, never, never give the media soundbites which can be used to present your results as more definitive than they really are. Lets not add to the problem people!
Summary
Wrong – 3 stars – I think this book makes some great points, but I don’t know that it’s for everyone. As long as you’re not doing science or reporting on science, the only real message is to think critically about results. If you are in the first two groups, I think this is a really important read.
Doing Dewey 