Are You Cherry Picking Data? The Danger of Trying to Prove You're Right!

How do you research something on the internet? Do you start with an open mind or do you set out to try and prove that what you believe is the truth about a subject?  What you type into Google will give you a clue, as well as what you do with the results once they appear.

If you are in fact cherry picking data then there is a pretty good chance that your conclusions may be completely wrong.

Let me explain!

 
 

Welcome to another evidence matters video.  In this series I discuss the importance of having good quality evidence to support the way we treat our pets and how to know if you can believe the advice you are receiving so that you can be certain you are making the best decisions possible.


What is cherry picking data?

If you’ve spent any time online you will have seen plenty of people cry “do your research”. Well’, research is all well and good, but only if you are doing it properly.  This plea is normally made by someone with alternative views and I bet in most cases they are serial cherry pickers!

Let’s start with the Wikipedia definition of cherry picking:

“Cherry picking, suppressing evidence, or the fallacy of incomplete evidence is the act of pointing to individual cases or data that seem to confirm a particular position while ignoring a significant portion of related cases or data that may contradict that position”

Let me explain what this means with an example using my old friend homeopathy.

Underlying bias

Let’s say you come from a family that strongly believes in homeopathy but you want to research if it really works or not.  Chances are you will go into your research with the starting opinion that homeopathy does work, even if this belief is a subconscious one.

The worst thing you could google would be “what is the evidence homeopathy works” and leave it at that.  Looking at the first page of google you will find that 4 of the results are proclaiming the wonder of homeopathic treatment and making bold claims about how effective it is. 1 result sits on the fence and the other 4 results on the first page claim that homeopathy is completely ineffective.

If you are a believer looking at these results, in all likelihood you will just click on the 4 results that support your view, follow any links they contain, make copious notes proving what you already believed and claim with satisfaction that your own independent research confirms homeopathy is the best thing since sliced bread.

What you will have done is cherry pick the evidence that supports your view and disregarded any evidence that suggested otherwise.

Of course if all you did was read the evidence that proves homeopathy doesn’t work you’d be doing the same thing.

If you read all the results and then googled “what is the evidence homeopathy doesn't work” you’d be doing a much better job.  You’d be trying to find evidence both for and against to allow you to try and form an overall opinion. You’d read about the systematic reviews of all the evidence and see the many institutions that all draw the same conclusion. In all likelihood you’ll come to the same opinion.

There's "evidence" for everything

In reality, you can find “evidence” for anything you can imagine.  This evidence may be nothing more than anecdote, and I discuss why this isn’t real evidence in a future video out soon, but if it supports your viewpoint and is the only thing you look at.

Just take the belief that the earth is flat.  If your research simply meant you read the “evidence” on this page from the flat earth society you will do nothing but reinforce your belief and that is not research.

Remember as well that cherry picking isn’t always done consciously.  Making a conscious decision to avoid cherry picking is important whenever you are researching a particular idea or treatment.  And remember that “evidence” can be found and quoted to support almost every opinion.

Pharma issues

Another category of cherry picking to be aware of is that done by manufacturers of whichever drug or supplement you are researching.

Most products will often make bold claims like “proven to result in a 59% faster cure rate!”.  If you actually look at where this data comes from, the actual conclusions may be very different.  Equally, they may have just picked data from the one study that showed a positive result and ignored the majority that failed to demonstrate any benefit.

The actual publishing of research studies is another area with the potential for cherry picking.  Running studies that are conducted in a way so as to be able to draw conclusions takes a lot of money.  As a result drug companies will often be the ones funding studies into their drugs. It is quite possible that they will then only publish those studies which show the best results, leaving those with less enthusiastic outcomes unpublished and unseen.

That’s not to say that the positive studies have been carried out incorrectly or that all pharmaceutical funded studies should be ignores, but that the overall body of evidence may be subject to bias as a result of cherry picking which studies to publish.

A really good book that discussed this issue and more about the pharmaceutical industry in more detail is the very readable “Bad Pharma”.  While you’re at it pick up the book “Bad Science” by the same author which goes through a lot of the ideas I’m discussing in my evidence matters series that are key to evidence based veterinary medicine.

Cherry picking data is just one way to misinterpret evidence.  Make sure you know all about the placebo effect in pets as well as reduction to the mean in the same article.

Our Pets Health: because they're family