Reading Nutrition Research: Why Substances That Work Don't Always Get Credit

Interpreting nutrition research is always controversial. It’s not uncommon for studies to return with findings that puzzle both scientists and consumers alike—particularly when a study fails to detect an effect that might actually exist. This article aims to shed light on this phenomenon, exploring the nuances to consider when you follow any of our PubMed article links. This article gets into the weeds to cover some critical concepts to understand when reading research including things like Type I vs Type II errors, statistical significance, and confounding variables. But first, we’ll address the philosophy Supplement Sciences uses when deciding when the research is “good enough” to mention.

Supplement Sciences Has A Bias

Yes, we fully acknowledge a bias toward reporting positive findings. Even when the studies on a topic don’t all agree. Even though more research is needed. Even when the research is mixed.

Many skeptics seem to take great pride in rejecting the benefits of an intervention in their “more rigorous” interpretations of the research. Such skeptics rarely acknowledge the risk, often much larger, that they will falsely dismiss study interventions that are likely to have real benefit.

Supplement-Sciences.com articles are written with the perspective of 30 years of clinical experience as a registered dietitian nutritionist. Research is selected as worthy/unworthy of reporting based on this clinical perspective as well as a master’s degree that focused primarily on developing the skills needed to design, defend, and analyze research methodologies. Our philosophy is designed to serve the interests of people who would rather have potentially imperfect nutritional evidence they can use today rather than “more rigorous” research that will ultimately arrive years too late to be useful.

Science Takes Too Long for Those Who Need Answers Today

It often pays off to be an early adopter when research starts to show that nutritional discoveries can make a difference in health. Throughout over 30 years of my career as a dietitian, there has been a clear pattern of researchers revealing an understanding of a nutritional principle many many years before their research is widely acknowledged. Researchers estimate that it typically takes 17 years from the time that something is in the peer-reviewed literature until it shows up in clinical practice where your physician is likely to know about it. [PMID: 37018006]

Trans fats are a classic example. Here’s the timeline:

1981: Mensink and Katan Study: Identifies the negative impacts of trans fats on cholesterol levels, noting that they both raise LDL (bad cholesterol) and lower HDL (good cholesterol).
1985: Lipid Research Clinics Coronary Primary Prevention Trial (LRC-CPPT): A large-scale, long-term clinical trial provided indirect insights into the effects of trans fats on lipid levels and cardiovascular risk.
1987: Multiple Risk Factor Intervention Trial (MRFIT): This study on dietary patterns and heart disease, highlighted the negative impact of trans fats.
1988: Harvard Nurses’ Health Study (Initial Reports): This study began to provide critical data linking trans fat intake and heart disease risk.
1990: Harvard School of Public Health: Research demonstrates a strong association between trans fat intake and an increased risk of heart disease.
1993: Lancet Study: Study confirms the understanding of how trans fats increase the risk of heart disease.
1994: USDA Consideration: The U.S. Department of Agriculture begins to consider requiring trans fat labeling.
2002: National Academy of Sciences Report: The report declares that the only safe level of trans fats is zero.
2003: FDA Trans Fat Labeling Requirement: The FDA announces the requirement for trans fats to be listed on Nutrition Facts labels.
2006: Implementation of FDA Labeling: The FDA’s requirement for trans fat labeling officially takes effect.
2015: FDA’s Determination on Partially Hydrogenated Oils: The FDA determines that partially hydrogenated oils are not “Generally Recognized as Safe”.
2018: Complete Ban on Trans Fats: The FDA’s ban on partially hydrogenated oils becomes effective.

Today, there are parallels playing out when discussing glyphosate, atrazine, and GMO crops as well as a multitude of other nutritional questions.

“No Good Evidence…”

Even with the massive widespread health implications of trans fat intake for the entire population and the research funding that goes with that, studies on trans fats still took many years to come to a consensus. Is it any wonder, then, that you might read that “There’s no good evidence for XYZ supplement.” Some cite that at least with FDA-approved drugs, there’s evidence that they work, implying that anything that isn’t FDA-approved doesn’t have evidence.

It can be a shock to learn that even “evidence-based medicine” comes up short when it comes to having “good evidence” according to Cochrane Reviews, a global organization with researchers from over 130 countries. In fact, one Cochrane review finds that most healthcare interventions (94%) were not supported by high-quality evidence. Further, they found that 8.1% of medical interventions caused harm.

Objective: To estimate the proportion of healthcare interventions tested within Cochrane Reviews that are effective according to high-quality evidence.

Results: Of 1,567 eligible interventions, 87 (5.6%) had high-quality evidence supporting their benefits. Harms were measured for 577 (36.8%) interventions. There was statistically significant evidence for harm in 127 (8.1%) of these. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was the main potential limitation of our study.

Conclusion: More than 9 in 10 healthcare interventions studied within recent Cochrane Reviews are not supported by high-quality evidence, and harms are under-reported.
Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis [PMID: 37018006]

How To Read The Research

Below, we move beyond the debate about what is “good evidence” and get into some of the more technical aspects of nutrition research. These include statistical significance, positive and negative findings, and Type I vs. Type II errors.

This technical detail can help make it easier to understand the maddening back and forth of the media reports that overstate the conflicts and inconsistencies in nutritional studies. The media is motivated to get your eyes on their stories. The more sensational the headline, the better chance you’ll read it. Ad revenue and time on page is what makes the bottom line. Correct interpretation and reporting can’t match the importance of the bottom line.

What It Takes For Research To Report Positive Findings

It helps to understand what researchers need to accomplish in order to report that something appears to work. No one has a magic formula for designing research studies that are valid and reliable. Within their budgets, researchers do the best they can to design studies to find the truth. Here’s what researchers need in order to report that something probably works:

Statistics: A 19 out of 20 statistical chance that the result of the study wasn’t a fluke.
Peer Review: The reputation of the publishing journal is affected by the quality of the research they publish. By publishing the article, the researchers on the editorial board of that journal believe:
- That the study is designed well enough to eliminate any major risk of confounding factors invalidating the results.
- The study was able to reach valid and reliable results
- That the results are reported appropriately according to scientific standards
- That the researchers have a good reputation within their field
- That any conflicts of interest are not significant enough to invalidate the study.
Financial resources: Funding can be hard to come by in nutrition research because nutrients and natural plant molecules cannot be patented. As a result, many studies are very small and underpowered to pick up nuanced effect sizes.

Validity and Reliability

Validity and reliability are cornerstone concepts in research. Validity refers to how well a study measures what it is supposed to measure. For instance, if a study aims to evaluate the effect of a diet on blood pressure, validity would mean that the methods truly measure the impact of the diet on blood pressure and not something else.

Reliability, on the other hand, refers to the consistency of a measure—a reliable study design produces stable and consistent results over time and across different conditions. If a study’s findings can be consistently replicated under similar methodologies, the study is considered reliable.

Both validity and reliability are crucial for building a solid foundation of credible and dependable research, enabling scientists and practitioners to draw meaningful conclusions and make informed decisions.

Blinding in Nutrition Research

Blinding is a crucial methodological technique used in nutrition research to minimize bias and ensure objectivity in the study outcomes. In a blinded study, participants, and often the researchers themselves, are unaware of which participants are receiving the treatment or intervention and which are receiving a placebo. This can be single-blind, where only the participants are unaware, or double-blind, where both participants and researchers are kept in the dark. For example, in a study examining the effects of a new dietary supplement, neither the subjects nor the investigators know who is receiving the supplement and who is receiving a placebo. This prevents participants’ and researchers’ expectations or preconceptions from influencing the behaviors, treatment administration, and interpretation of results, leading to more reliable and valid conclusions.

Placebo Control in Nutrition Research

Placebo control is a fundamental component of rigorous nutritional studies, particularly in clinical trials. In nutrition research, a placebo must be indistinguishable in taste, texture, and appearance from the dietary product being tested. Using a placebo allows researchers to isolate the effects of the dietary intervention from psychological and physiological effects not attributable to the intervention itself. For instance, if a study aims to evaluate the effectiveness of a protein supplement on muscle growth, participants might receive either the protein supplement or a placebo with no nutritional value. Any differences in muscle growth can then be attributed to the actual effects of the protein supplement rather than participants’ beliefs or expectations about taking it, thus affirming the real impact of the nutritional intervention.

Statistical Significance

In research, the null hypothesis is the default assumption that “nothing special is happening” between the things we’re studying. The job of the researchers is to see if they can prove this assumption wrong. In other words, they set up the study so that they can prove whatever the relationship is, it is not “nothing”.

The p-value measures the probability that the results would occur by accident even if there were no actual effect. The standard p-value of less than 0.05, means there’s statistically less than a 5% probability that the results are due to random chance rather than a real effect. In other words, there is strong evidence to reject the null hypothesis.

Reaching “statistical significance” means that there is a 19 out of 20 chance that the effects are not due to chance. In nutrition research, this is a 95% confidence level that a relationship exists between the nutrient being studied and the health benefit, assuming the experimental design and analysis are correct. Reaching “statistical significance” means that the probability of the observed effect occurring by chance is less than 5%. It suggests there is a 95% confidence that the observed differences are not due to random variation, provided the study design and execution are sound.

The 0.05 level is a convention that has been historically established in the sciences. It strikes a balance between being too lenient and too strict. Setting the bar at 0.05 helps prevent too many false positives (claiming an effect exists when it doesn’t, a Type I error) without being so stringent that real effects are too often missed (Type II error). More on Type I and Type II errors below.

To take this further, it’s interesting to notice when a study reports a p-value of “p<0.001”. This means there is less than one chance in a thousand that the result was reached just by random chance. Pretty impressive.

Achieving statistical significance can be challenging for several reasons:

Small Sample Sizes: Studies with a small number of participants may not have enough power to detect an effect, even if one exists. The smaller the sample, the harder it is to achieve statistical significance.
Effect Size: If the true effect of a supplement is small, it requires larger sample sizes or more sensitive measures to detect it. Studies not designed to capture subtle effects might fail to reach the statistical significance threshold.
Variability in Data: High variability within the sample can obscure real effects. For example, if the response to a supplement varies widely among participants, it can be difficult to detect a consistent pattern that meets the criteria for statistical significance.

While the 0.05 threshold is a useful tool for assessing the likelihood that a study’s results reflect true effects, it is not a definitive measure of truth. It is merely a convention that shows the probability of the results occurring by chance. Researchers must carefully design studies to ensure adequate power (enough people in the study) to detect meaningful effects, consider the size and variability of the sample, and use appropriate statistical corrections when necessary. This approach helps in making robust and reliable conclusions from the data.

Confounding Variables

Imagine researchers are trying to figure out if vitamin D helps improve bone health. If they don’t consider how much calcium participants are consuming—Calcium is a confounding variable.

Now, for a more complex example, when researchers study the effects of multivitamins on overall health. If they are not controlling for a range of factors such as age, smoking, stress levels, diet quality, genetic predispositions, or socioeconomic status, the study’s results can become muddled. For instance, if a study finds that multivitamins do not significantly improve health, this conclusion overlooks that certain subgroups, such as older adults or smokers, might indeed benefit from multivitamins, while others may not.

People often think that the larger the number of participants in the study, the better the study. But, in this multivitamin study, including a large and diverse group of participants with these confounding variables is a drawback. The wide targeting means that specific beneficial effects for certain subgroups could be diluted by the non-effects in others. For example, if multivitamins help reduce stress-related health issues but only in individuals who have high-stress jobs, this effect might be lost in a large study that includes many people with low-stress lifestyles. Similarly, the benefits of multivitamins might vary greatly with diet quality; those already consuming a nutrient-rich diet might show little to no effect from additional vitamins, overshadowing potential benefits seen in those with poorer diets.

This kind of broad-brush approach in large studies can obscure the nuanced ways in which multivitamins might impact different groups of people, emphasizing the need for well-designed studies that account for these variables to truly understand who might benefit the most from multivitamin supplements.

Understanding Positive vs. Negative Findings

Results of nutritional research falls into two categories: positive findings and negative findings. Understanding these outcomes is important for interpreting the implications of research.

A positive finding means that the results show a clear connection between the nutritional component and a specific health outcome, such as improved heart health, increased bone density, or enhanced immune function. Positive findings mean that there definitely seems to be an effect.

Negative Findings are not as easy to interpret. Negative findings arise when the study does NOT find the supplement or nutrient has the expected health benefit under the study conditions. It is important to note that a negative finding does not necessarily prove that the supplement has no effect whatsoever. Rather, it shows that the study, with its specific conditions, did not observe an effect. The reasons for this could vary, including factors such as insufficient sample size, incorrect dosage, or inadequate study duration. For example, a short 6-week study of low-dose Vitamin D on bone health in subjects who are not deficient in the vitamin is going to have negative findings.

Understanding False Positives And False Negatives

Navigating the complexities of research often involves understanding the nuances of false positives (Type I errors) and false negatives (Type II errors). These are critical concepts that researchers and readers alike must grasp to accurately interpret findings.

A false positive (Type I error) happens when a study mistakenly indicates an effect is present when it isn’t. Imagine being told a medication works when it actually doesn’t; this is the essence of a false positive. Typically, researchers set a risk threshold of 5% for these errors, which means there’s a small chance that findings might be due to random fluctuations rather than a real effect.

On the flip side, false negatives (Type II errors) happens when a study fails to detect an effect that actually does exist. Often, this happens because the study doesn’t have enough participants or isn’t long enough to reveal a subtle effect or the methods used aren’t sensitive enough to detect it.

To Sum It Up

This article delves into the complex and often controversial world of nutrition research, addressing why studies sometimes produce perplexing or contradictory results. It explores critical concepts such as confounding variables, Type I and Type II errors, and statistical significance—key factors that can influence the outcomes of research studies.

We also discussed the philosophy used by Supplement Sciences in evaluating whether research is robust enough to cite. By unpacking these nuanced aspects of scientific studies of nutrients and supplements, we hope our readers will find it easier to understand and interpret research findings found in the PubMed article links.