If you are trying to educate yourself about proper diet and fitness or even if you just scroll through your social media feed, it’s very likely that you’ll see information about the latest study. That can be “Study shows green tea is the best fat burner!” or other weird stuff.

It’s usually extremely simplified, out of context and in many cases absolutely wrong. Here’s what’s behind the “most recent studies”.

Researcher’s Dilemma

Put yourself in the shoes of a researcher. They constantly have to work on something interesting and new and also keep publishing new studies to secure their funding. That means they sometimes have to blow things out of proportion or get creative when it comes to drawing conclusions from the results.

One way of doing that is p-hacking. P-hacking is when researchers take a bunch of random variables and play with them so long until they come up with a statistically significant correlation that they can draw a conclusion from. It also means that you come up with your hypothesis after finding the correlation. In a normal scenario you come up with a hypothesis, a statistical model to prove it and then go about collecting your data. Then you analyze your data set and either prove or disprove your hypothesis. Doing it the other way around is not very scientific.

 

Sample size

It’s standard to use random samples in studies to avoid any bias. Another important factor though is the sample size. If for instance the group of people you are looking at is very small (e.g. 10 to 50 people), can you draw a conclusion from the data that is applicable to a whole population? No, you can’t. Deciding the appropriate sample size is a science in itself but just keep in mind that results from a study with a small sample size can never be generalized.

How come no one realizes it?

By outlining your scientific method, researchers make it possible for other researchers to replicate the original study. So you could assume that studies with surprising results motivate some researchers to replicate the study to see if it’s actually true. The problem though is that there is absolutely no glory in replicating studies and no one cares about a study that simply confirms what another study already found out.

 

Was it actually a human trial?

Depending on the study, it can be quite dangerous to conduct it on humans. Which is why a lot of studies are done on animals like rats, rabbits etc.

Take for example one of the key studies that linked cholesterol to cardiovascular disease. First of all it’s already over 100 years old and secondly, it was conducted on rabbits. Rabbits are herbivores and never consume any food that contains cholesterol. Humans on the other hand are omnivores and cholesterol is a natural part of our diet. Feeding rabbits something that’s not in their natural diet will obviously make them sick. But then drawing the conclusion that the same applies for humans is false.

 

Who funded the study?

This one is so basic but overlooked way too often. Researchers don’t work for free. They have bills to pay too. So, where is the money coming from? While many researchers are relying on scholarships or some other funding, a lot of researchers receive money from certain industries or even individual companies. The usual suspects are the tobacco industry, corn lobby, sugar related organizations (Kellog, Coca Cola etc.) and more. If the researcher is funded by a particular industry, it is extremely unlikely that he or she will publish any study that damages the industry. It would be career suicide.

Where’s the over simplification coming from?

Ok, so you got a decent study from independent researchers yet the information you are presented on TV seem strange. That could be because the scientific body summarized the study for their press release. The busy journalist was just looking at the headlines and brief summary without looking at the details. Without the details and knowing the subject matter the journalist takes it out of context intentionally or unintentionally and then ends up completely misinterpreting it.

Simple checklist

So, what can you do? Whenever you hear about a new study, check the following:

1. What are their sources?
2. Who funded that study?
3. What’s the actual outcome of the study (vs. what the media presented)?
4. If that still sounds too good to be true, look at the scientific method behind it

 

Lastly, here is an overview showing nicely that basically everything you eat or drink causes and cures cancer at the same time. That’s what crappy research leads to.