A frankly shocking amount of research funding — perhaps as high as 85%, or $200 billion — is wasted on poorly designed studies. How can you improve your study design to avoid falling into this trend?
Each day scientists plan, conduct, analyze and debate study design, hoping their efforts will lead to a cure or a new understanding of a critical biological pathway. Is it easy? No way! Designing studies must take into consideration those things that will influence study quality.
There are both internal and external factors driving the design of ineffective research studies. The internal factors are most directly under a researcher's control, as they have to do with actually conducting a study. External factors, those influences imposed from outside sources, are somewhat more difficult to account for.
By internal, we mean all the components of the actual study under a researcher's control. Study design considers the necessary components needed to get from hypothesis to interpretation.
Each one of these components can and will influence the outcome of the study. Choosing the wrong strain or dosing regimen can lead to a failed study. By failure, we mean the results will not be valid. This contrasts with negative results, which don't support your hypothesis but are valid.
Throughout this process, attention to detail is imperative to assure quality, useful results. Without that, science does not move forward. A small diversion in quality may invalidate an entire study. Invalid studies result in wasted resources. Cures get delayed. In essence, we let down our community of patients.
Other internal considerations may include attrition. Running an Alzheimer's Disease study, for example, may require mice to be on study for twelve-to-eighteen months. Unfortunately, mice will die during this time. Sizing the study groups to take this into consideration will assure valid results.
In late 2016 VOX media sent out a survey to scientists across disciplines and geographies asking what problems they face. 270 scientists replied, many from the life sciences. They heard back from graduate students, professors and department heads. The authors boiled results down to seven problems scientist face. One of the problems described is publication bias (read as "publication is king"). This quality influencer means researchers design their studies in a way that improves chances of publication. We can infer that a scientist at a for-profit, may design studies to attract VC money or partnerships.
Michael Inzlicht, a University of Toronto psychology and neuroscience professor, writes: "I think the one thing that would have the biggest impact is removing publication bias: judging papers by the quality of questions, quality of method, and soundness of analyses, but not on the results themselves."
More journals now accept negative results and base acceptance more on study design, robust methods and valid results. PLOS One is a journal that was founded on those principles. They want to encourage good science and reward scientists for thorough study design.
Removing this negative quality influencer would free up researchers. They could design required studies with less fear of not getting published.
The table below comes from a presentation given by Gerald Carter from The Ohio State University. While a bit overstated, it demonstrates the conflict that researchers feel. This table is for academics, but we can imagine a similar table for those researchers in for-profit labs.
Early Career Researcher (ECR), Xenia Schmalz, a Post-doc at the University of Padova, wrote the following on her blog: "Most of us [ECRs] have short-term contracts, and we are forced to decide whether we want to use our limited time being good scientists, or good academics...I face daily reminders of publication pressure...Being a good scientist (as defined in the table above) requires thorough planning of experiments, testing of possibly hundreds of participants...careful data analysis, follow-up experiments to clarify messy findings, etc., etc. In the time that it takes to publish one good study, a peer can publish at least five sloppy studies."
The VOX article also points out the lack of study replication. Are published studies and their results repeatable in another lab? Replication helps to move from hypothesis, to theory, to law. "Testing, validating, retesting — it's all part of a slow and grinding process to arrive at some semblance of scientific truth...Scientists face few incentives to engage in the slog of replication." Scientists may be using unreplicated studies to build their own studies which could end with flawed results.
External influences on study design may be less tangible than internal, but they are real and must be confronted.
In the long view, quality studies will always win out. It is a win-win-win; for researchers conducting studies, for researchers replicating studies, and for patients whose quality of health will improve.
Watch the Taconic Biosciences' Webinar:
Read the Related Taconic Biosciences' Insight:
Each day scientists plan, conduct, analyze and debate study design, hoping their efforts will lead to a cure or a new understanding of a critical biological pathway. Is it easy? No way! Designing studies must take into consideration those things that will influence study quality.
There are both internal and external factors driving the design of ineffective research studies. The internal factors are most directly under a researcher's control, as they have to do with actually conducting a study. External factors, those influences imposed from outside sources, are somewhat more difficult to account for.
Internal Factors Influencing Study Design
| Study Design |
| Compose hypothesis |
| Identify mouse strain(s), sex and age |
| Define experimental and control groups |
| Number of mice to power the study |
| Dosing regimen |
| Sample collection and timeline |
| Sample prep |
| Sample test |
| Statistical Analysis |
| Interpretation |
Each one of these components can and will influence the outcome of the study. Choosing the wrong strain or dosing regimen can lead to a failed study. By failure, we mean the results will not be valid. This contrasts with negative results, which don't support your hypothesis but are valid.
Throughout this process, attention to detail is imperative to assure quality, useful results. Without that, science does not move forward. A small diversion in quality may invalidate an entire study. Invalid studies result in wasted resources. Cures get delayed. In essence, we let down our community of patients.
Other internal considerations may include attrition. Running an Alzheimer's Disease study, for example, may require mice to be on study for twelve-to-eighteen months. Unfortunately, mice will die during this time. Sizing the study groups to take this into consideration will assure valid results.
External Influences on Study Design
Gone are the days of science for science sake, working tirelessly toward a goal where passion was the main driver. That's not to say scientists don't have passion today, but they face many other competing influences within companies, academia, or governments.In late 2016 VOX media sent out a survey to scientists across disciplines and geographies asking what problems they face. 270 scientists replied, many from the life sciences. They heard back from graduate students, professors and department heads. The authors boiled results down to seven problems scientist face. One of the problems described is publication bias (read as "publication is king"). This quality influencer means researchers design their studies in a way that improves chances of publication. We can infer that a scientist at a for-profit, may design studies to attract VC money or partnerships.
Michael Inzlicht, a University of Toronto psychology and neuroscience professor, writes: "I think the one thing that would have the biggest impact is removing publication bias: judging papers by the quality of questions, quality of method, and soundness of analyses, but not on the results themselves."
More journals now accept negative results and base acceptance more on study design, robust methods and valid results. PLOS One is a journal that was founded on those principles. They want to encourage good science and reward scientists for thorough study design.
Removing this negative quality influencer would free up researchers. They could design required studies with less fear of not getting published.
The table below comes from a presentation given by Gerald Carter from The Ohio State University. While a bit overstated, it demonstrates the conflict that researchers feel. This table is for academics, but we can imagine a similar table for those researchers in for-profit labs.
| Divergent Interests | |
| To be a good scientist... | To be a good academic... |
| Be skeptical of your results | "Sell" your results |
| Interpret conclusions carefully | Highlight/exaggerate importance |
| Publish negative results | Publish "strategically" |
| Ignore social prestige | Use impact factors to make writing decisions |
| Challenge authority | Cite authority. Make friends |
| Replicate, Replicate | Replicate...if you must |
| Novel exciting results are less likely to be true. Double-check them | Publish novel exciting results before you get scooped. |
From: https://www.slideshare.net/geraldcarter1/academia-51559796
Early Career Researcher (ECR), Xenia Schmalz, a Post-doc at the University of Padova, wrote the following on her blog: "Most of us [ECRs] have short-term contracts, and we are forced to decide whether we want to use our limited time being good scientists, or good academics...I face daily reminders of publication pressure...Being a good scientist (as defined in the table above) requires thorough planning of experiments, testing of possibly hundreds of participants...careful data analysis, follow-up experiments to clarify messy findings, etc., etc. In the time that it takes to publish one good study, a peer can publish at least five sloppy studies."
The VOX article also points out the lack of study replication. Are published studies and their results repeatable in another lab? Replication helps to move from hypothesis, to theory, to law. "Testing, validating, retesting — it's all part of a slow and grinding process to arrive at some semblance of scientific truth...Scientists face few incentives to engage in the slog of replication." Scientists may be using unreplicated studies to build their own studies which could end with flawed results.
External influences on study design may be less tangible than internal, but they are real and must be confronted.
Improving Study Design Helps Everyone
No one ever said the life of a researcher was going to be easy. Research has become a complicated, multibillion-dollar enterprise. External study design pressures can be hard to navigate. The internal quality influencers are no less daunting, constrained by budgets and time. Publication is necessary for science to progress. The publications, though, are only useful when based on sound studies.In the long view, quality studies will always win out. It is a win-win-win; for researchers conducting studies, for researchers replicating studies, and for patients whose quality of health will improve.
Watch the Taconic Biosciences' Webinar:Read the Related Taconic Biosciences' Insight:
