Reading Nutrition Research: Why Substances That Work Don't Always Get Credit

Interpreting nutrition research often raises questions and invites debate. It’s not unusual for studies to produce results that seem uncertain, especially when a real effect goes undetected. This article takes a closer look at why that happens and what it means for making sense of the research you come across, including the studies we reference on PubMed. We’ll explore key ideas such as Type I and Type II errors, statistical significance, and confounding variables. But first, we’ll explain how Supplement Sciences evaluates whether the evidence behind a supplement is strong enough to include in our content.

Science Always Has A Bias: Choose One Or The Other

The scientific method typically begins with the assumption that nothing is happening, meaning the intervention being tested has no real effect. This assumption is called the null hypothesis. Researchers then collect and analyze data to determine whether that starting point holds true.

Science always involves a balance between two risks: finding an effect when none truly exists and failing to find an effect that really is there. These are known as Type I and Type II errors, which are explained in more detail later in the article. At Supplement Sciences, we are open about the fact that our coverage tends to highlight positive findings, even when studies on a topic do not all agree, when more research is needed, or when the evidence is mixed.

Many skeptics seem to take great pride in rejecting the benefits of an intervention in their “more rigorous” interpretations of the research. Yet they rarely acknowledge the equal or even greater risk of rejecting interventions that may in fact be beneficial.

Supplement-Sciences.com articles are written with the perspective of 30 years of clinical experience as a registered dietitian nutritionist. Research is selected as worthy/unworthy of reporting based on this clinical perspective as well as a master’s degree that focused primarily on developing the skills needed to design, defend, and analyze research methodologies. Our philosophy is designed to serve the interests of people who would rather have potentially imperfect nutritional evidence they can use today rather than “more rigorous” research that will ultimately arrive years too late to be useful.

Science Takes Too Long for Those Who Need Answers Today

It often pays off to be an early adopter when research starts to show that nutritional discoveries can make a difference in health. Throughout over 30 years of my career as a dietitian, there has been a clear pattern of researchers revealing an understanding of a nutritional principle many many years before their research is widely acknowledged. Researchers estimate that it typically takes 17 years from the time that something is in the peer-reviewed literature until it shows up in clinical practice where your physician is likely to know about it. [PMID: 37018006]

Trans fats are a classic example. Here’s the timeline:

1981: Mensink and Katan Study: Identifies the negative impacts of trans fats on cholesterol levels, noting that they both raise LDL (bad cholesterol) and lower HDL (good cholesterol).
1985: Lipid Research Clinics Coronary Primary Prevention Trial (LRC-CPPT): A large-scale, long-term clinical trial provided indirect insights into the effects of trans fats on lipid levels and cardiovascular risk.
1987: Multiple Risk Factor Intervention Trial (MRFIT): This study on dietary patterns and heart disease, highlighted the negative impact of trans fats.
1988: Harvard Nurses’ Health Study (Initial Reports): This study began to provide critical data linking trans fat intake and heart disease risk.
1990: Harvard School of Public Health: Research demonstrates a strong association between trans fat intake and an increased risk of heart disease.
1993: Lancet Study: Study confirms the understanding of how trans fats increase the risk of heart disease.
1994: USDA Consideration: The U.S. Department of Agriculture begins to consider requiring trans fat labeling.
2002: National Academy of Sciences Report: The report declares that the only safe level of trans fats is zero.
2003: FDA Trans Fat Labeling Requirement: The FDA announces the requirement for trans fats to be listed on Nutrition Facts labels.
2006: Implementation of FDA Labeling: The FDA’s requirement for trans fat labeling officially takes effect.
2015: FDA’s Determination on Partially Hydrogenated Oils: The FDA determines that partially hydrogenated oils are not “Generally Recognized as Safe”.
2018: Complete Ban on Trans Fats: The FDA’s ban on partially hydrogenated oils becomes effective.

Today, there are parallels playing out when discussing glyphosate, atrazine, and GMO crops as well as a multitude of other nutritional questions.

“No Good Evidence…”

Even with the massive widespread health implications of trans fat intake for the entire population and the research funding that goes with that, studies on trans fats still took many years to come to a consensus. Is it any wonder, then, that you might read that “There’s no good evidence for XYZ supplement.” Some cite that at least with FDA-approved drugs, there’s evidence that they work, implying that anything that isn’t FDA-approved doesn’t have evidence.

It can be a shock to learn that even “evidence-based medicine” comes up short when it comes to having “good evidence” according to Cochrane Reviews, a global organization with researchers from over 130 countries. In fact, one Cochrane review finds that most healthcare interventions (94%) were not supported by high-quality evidence. Further, they found that 8.1% of medical interventions caused harm.

Objective: To estimate the proportion of healthcare interventions tested within Cochrane Reviews that are effective according to high-quality evidence.

Results: Of 1,567 eligible interventions, 87 (5.6%) had high-quality evidence supporting their benefits. Harms were measured for 577 (36.8%) interventions. There was statistically significant evidence for harm in 127 (8.1%) of these. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was the main potential limitation of our study.

Conclusion: More than 9 in 10 healthcare interventions studied within recent Cochrane Reviews are not supported by high-quality evidence, and harms are under-reported.
Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis [PMID: 35447356]

How To Read The Research

Below, we move beyond the debate about what is “good evidence” and get into some of the more technical aspects of nutrition research. These include statistical significance, positive and negative findings, and Type I vs. Type II errors.

This technical detail can help make it easier to understand the maddening back and forth of the media reports that overstate the conflicts and inconsistencies in nutritional studies. The media is motivated to get your eyes on their stories. The more sensational the headline, the better chance you’ll read it. Ad revenue and time on page is what makes the bottom line. Correct interpretation and reporting can’t match the importance of the bottom line.

What It Takes For Research To Report Positive Findings

It helps to understand what researchers need to accomplish in order to report that something appears to work. No one has a magic formula for designing research studies that are valid and reliable. Within their budgets, researchers do the best they can to design studies to find the truth. Here’s what researchers need in order to report that something probably works:

Statistics: A 19 out of 20 statistical chance that the result of the study wasn’t a fluke.
Peer Review: The reputation of the publishing journal is affected by the quality of the research they publish. By publishing the article, the researchers on the editorial board of that journal believe:
- That the study is designed well enough to eliminate any major risk of confounding factors invalidating the results.
- The study was able to reach valid and reliable results
- That the results are reported appropriately according to scientific standards
- That the researchers have a good reputation within their field
- That any conflicts of interest are not significant enough to invalidate the study.
Financial resources: Funding can be hard to come by in nutrition research because nutrients and natural plant molecules cannot be patented. As a result, many studies are very small and underpowered to pick up nuanced effect sizes.

Validity and Reliability

Validity and reliability are two of the most important ideas in research. Validity is about whether a study truly measures what it intends to measure. For example, if researchers set out to study the effect of a diet on blood pressure, validity means their methods actually capture the diet’s impact on blood pressure rather than something unrelated.

Reliability is about consistency. A reliable study produces stable, repeatable results across time and under similar conditions. When findings can be reproduced using the same approach, the study is considered reliable.

Together, validity and reliability form the foundation of credible research. They give scientists and practitioners the confidence to draw meaningful conclusions and use the evidence to guide real-world decisions.

Blinding in Nutrition Research

Blinding is a crucial methodological technique used in nutrition research to minimize bias and ensure objectivity in the study outcomes. In a blinded study, participants, and often the researchers themselves, are unaware of which participants are receiving the treatment or intervention and which are receiving a placebo. This can be single-blind, where only the participants are unaware, or double-blind, where both participants and researchers are kept in the dark. For example, in a study examining the effects of a new dietary supplement, neither the subjects nor the investigators know who is receiving the supplement and who is receiving a placebo. This prevents participants’ and researchers’ expectations or preconceptions from influencing the behaviors, treatment administration, and interpretation of results, leading to more reliable and valid conclusions.

Placebo Control in Nutrition Research

Placebo control is a fundamental component of rigorous nutritional studies, particularly in clinical trials. In nutrition research, a placebo must be indistinguishable in taste, texture, and appearance from the dietary product being tested. Using a placebo allows researchers to isolate the effects of the dietary intervention from psychological and physiological effects not attributable to the intervention itself. For instance, if a study aims to evaluate the effectiveness of a protein supplement on muscle growth, participants might receive either the protein supplement or a placebo with no nutritional value. Any differences in muscle growth can then be attributed to the actual effects of the protein supplement rather than participants’ beliefs or expectations about taking it, thus affirming the real impact of the nutritional intervention.

Statistical Significance

In research, the null hypothesis is the default assumption that “nothing special is happening” between the things we’re studying. The job of the researchers is to see if they can prove this assumption wrong. In other words, they set up the study so that they can prove whatever the relationship is, it is not “nothing”.

The p-value measures the probability that the results would occur by accident even if there were no actual effect. The standard p-value of less than 0.05, means there’s statistically less than a 5% probability that the results are due to random chance rather than a real effect. In other words, there is strong evidence to reject the null hypothesis.

Reaching “statistical significance” means that there is a 19 out of 20 chance that the effects are not due to chance. In nutrition research, this is a 95% confidence level that a relationship exists between the nutrient being studied and the health benefit, assuming the experimental design and analysis are correct. Reaching “statistical significance” means that the probability of the observed effect occurring by chance is less than 5%. It suggests there is a 95% confidence that the observed differences are not due to random variation, provided the study design and execution are sound.

The 0.05 level is a convention that has been historically established in the sciences. It strikes a balance between being too lenient and too strict. Setting the bar at 0.05 helps prevent too many false positives (claiming an effect exists when it doesn’t, a Type I error) without being so stringent that real effects are too often missed (Type II error). More on Type I and Type II errors below.

To take this further, it’s interesting to notice when a study reports a p-value of “p<0.001”. This means there is less than one chance in a thousand that the result was reached just by random chance. Pretty impressive.

Achieving statistical significance can be challenging for several reasons:

Small Sample Sizes: Studies with a small number of participants may not have enough power to detect an effect, even if one exists. The smaller the sample, the harder it is to achieve statistical significance.
Effect Size: If the true effect of a supplement is small, it requires larger sample sizes or more sensitive measures to detect it. Studies not designed to capture subtle effects might fail to reach the statistical significance threshold.
Variability in Data: High variability within the sample can obscure real effects. For example, if the response to a supplement varies widely among participants, it can be difficult to detect a consistent pattern that meets the criteria for statistical significance.

While the 0.05 threshold is a useful tool for assessing the likelihood that a study’s results reflect true effects, it is not a definitive measure of truth. It is merely a convention that shows the probability of the results occurring by chance. Researchers must carefully design studies to ensure adequate power (enough people in the study) to detect meaningful effects, consider the size and variability of the sample, and use appropriate statistical corrections when necessary. This approach helps in making robust and reliable conclusions from the data.

Confounding Variables

Imagine a study looking at whether vitamin D improves bone health. If the researchers fail to track how much calcium participants are consuming, their results could be misleading because calcium also affects bone health. In this case, calcium acts as a confounding variable, meaning it influences both the factor being studied and the outcome, making it harder to see the true effect of vitamin D.

This issue becomes even more pronounced with complex questions such as the impact of multivitamins on overall health. If researchers do not control for factors like age, smoking habits, stress levels, diet quality, genetic background or socioeconomic status, the results can easily become unclear. For example, a study might conclude that multivitamins do not improve health, but that broad conclusion could mask benefits for specific groups, such as older adults or smokers, who might respond differently.

Many people assume that bigger studies automatically produce better answers. However, in the multivitamin example, having a large and diverse participant pool can actually dilute meaningful effects. If multivitamins reduce stress-related health problems only among people in high-stress jobs, that benefit might disappear when averaged with results from people living low-stress lifestyles. Likewise, those already eating nutrient-rich diets might see little effect from supplements, overshadowing benefits observed in people with poorer diets.

This highlights why well-designed studies must account for confounding variables. Without this attention to detail, the subtle ways supplements affect different groups can be overlooked, and valuable information about who benefits most can be lost.

Understanding Positive vs. Negative Findings

Results of nutritional research falls into two categories: positive findings and negative findings. Understanding these outcomes is important for interpreting the implications of research.

A positive finding means that the results show a clear connection between the nutritional component and a specific health outcome, such as improved heart health, increased bone density, or enhanced immune function. Positive findings mean that there definitely seems to be an effect.

Negative Findings are not as easy to interpret. Negative findings arise when the study does NOT find the supplement or nutrient has the expected health benefit under the study conditions. It is important to note that a negative finding does not necessarily prove that the supplement has no effect whatsoever. Rather, it shows that the study, with its specific conditions, did not observe an effect. The reasons for this could vary, including factors such as insufficient sample size, incorrect dosage, or inadequate study duration. For example, a short 6-week study of low-dose Vitamin D on bone health in subjects who are not deficient in the vitamin is going to have negative findings.

Understanding False Positives And False Negatives

Navigating the complexities of research often involves understanding the nuances of false positives (Type I errors) and false negatives (Type II errors). These are critical concepts that researchers and readers alike must grasp to accurately interpret findings.

A false positive (Type I error) happens when a study mistakenly indicates an effect is present when it isn’t. Imagine being told a medication works when it actually doesn’t; this is the essence of a false positive. Typically, researchers set a risk threshold of 5% for these errors, which means there’s a small chance that findings might be due to random fluctuations rather than a real effect.

On the flip side, false negatives (Type II errors) happens when a study fails to detect an effect that actually does exist. Often, this happens because the study doesn’t have enough participants or isn’t long enough to reveal a subtle effect or the methods used aren’t sensitive enough to detect it.

To Sum It Up

This article explores why nutrition research can feel complex and at times contradictory. It walks through key ideas such as confounding variables, Type I and Type II errors, and statistical significance, offering clarity on how these factors shape study outcomes.

It also explains how Supplement Sciences assesses the strength of evidence before weaving it into our content. By outlining these core elements of nutrition and supplement research, the article aims to help readers make better sense of the studies included in the PubMed references we cite.