At least once every week I open the newspaper to see a headline trumpeting the results of the latest human health studies. Studies linking egg yolks, red meat and alcohol (to name but a few), with increased mortality and morbidity.
The headlines are purposefully provocative (“Egg yolks almost as unhealthy as cigarettes“), the articles replete with scientific and statistical terms that mean nothing to the reader and very rarely is there sufficient information about the original study to allow proper interpretation of the results.
Frequently, this week’s study contradicts the results of previous studies on the same topic, making it difficult for the educated lay person to know which study is correct and hence, whether they should reduce their consumption of eggs, eliminate red meat from their diet completely, drink more, drink less or not at all.
Long before I became a fitness writer, I studied the behaviour, ecology and evolution of fish. Data analysis and experimental design were things I excelled at. Much of what I learned is directly applicable to understanding the results of human health studies. Shall I share?
Rather than risk the ire of readers by using their favourite food as an example, let’s investigate the relationship between snorkleberries and snorkleberry-beri (a highly contagious disease that renders the fingers and toes a fluorescent pink, leading to social stigmatization and shunning).
When you read the results of human health studies you need to ask four questions:
1. Is the study observation or experimental?
Observational (or population) studies are those conducted on a large group (or population) of individuals. Data is often obtained via questionnaire. For example, individuals might answer questions about how frequently they eat snorkleberries , how often they exercise, whether there’s a history of heart disease in their family and whether they smoke. Measurements such as weight, height, body fat and blood snorkleberry levels might be taken as well.
Multivariate analysis of the data is conducted to identify relationships between variables; for example, individuals who ate more snorkleberries were more likely to suffer from snorkleberry-beri. Observational studies can identify correlations, but cannot conclude that consequence B is a direct result of action A.
Correlation does not equal causation. Why? It just might be that individuals who eat more snorkleberries also consume large amounts of enkelbird eggs.
Experimental studies are exactly what they sound like. Experiments.
Subjects were (or should have been) randomly assigned to test and control groups, assigned a particular treatment (for example, eat 100 snorkleberries per week, eat 50 snorkleberries per week, eat 0 snorkleberries per week) and monitored for adherence to the protocol (this is a huge challenge with human subjects ).
Data are analyzed by either comparing average responses between groups or, in the case of an experiment in which each subject experiences each treatment (again, in random order), the average of the difference between each subject’s response to the various treatments. Confusing, ‘eh?
Experimental studies can demonstrate a causal relationship between action A and consequence B, given appropriate experimental controls and data analysis. In this case, snorkleberries eaters suffered more bouts of snorkleberry-beri than those who abstained.
2. Is the sample size adequate?
In general, the more participants in the study, the better. Most data sets contain a lot of noise (variation between individuals). Biologically significant effects may not reveal themselves to the researcher if too few individuals are included in the study. Observational studies tend to have much larger sample sizes than experimental studies, in part, because it’s a lot easier to get people to fill out questionnaires than it is to get them to adhere to strict research protocols. But also, because of the ethics involved in conducting controlled health studies on humans. (Who would voluntarily sign up to eat 100 snorkleberries each week if they knew there was a possibility that they’d contract snorkleberry-beri?).
3. Is the effect revealed by the study biologically meaningful?
While having a large sample size is generally thought to be beneficial, due to the magic of statistics, it also makes very small effects more easily detectable. Just because a statistically significant effect is found does not mean that the effect itself is biologically important. A 300% increase in the risk of contracting snorkleberry-beri is not biologically significant if the initial risk itself was only 1 in one million.
4. Was the study free of bias?
While most scientific researchers aim to eliminate all forms of bias from their studies, human subjects are notorious for failing to follow experimental protocols to the letter (“I didn’t like the taste of the snorkleberries so I substituted blueberries instead”). When questionnaires are administered, we often (unconsciously) under-report activities that we perceive as negative and even slant our answers in the direction that we think the researcher wants to hear.
And of course, a study’s source of funding should always be considered when deciding how much value to give to its results. Naturally, the Snorkleberry Grower’s Association is happy to fund (and acknowledge funding) of studies indicating no relationship between snorkleberry consumption and snorkleberry-beri.
If you’ve made it to the end of the post, congratulations! You’ve just passed Stats 101. Enjoy some snorkleberry juice, on me; I promise, there’s no evidence that it causes snorkleberry-beri!
Are you frequently alarmed by the results of human health studies?
Do you greet their conclusions with skepticism or believe what’s written and immediately change your behaviour?