Evidence is Evidence! Are You Sure about That?

The post below highlights one aspect of Chad Cook and Ken Learman’s Rehabilitation Research Boot Camp course series. Click here to start the nine part series. 

Let’s say you find an amazing manuscript that shows a marked clinical effect with a specific treatment intervention. Must you truly vet the information within the article before sharing with your colleagues? Should you even bother doing that? Should you identify the key features that could corrode the findings? Are there tangible components that we should all consider when consuming research?

The answer for all these questions is Yes.

Changing Evidence & Meaningful Information

Lakshmipati once wrote, “Half of what is true today will be proven to be incorrect in the next five years. Unfortunately we don’t know which half that is going to be.”

I would argue that the half that will change will contradict previous work. But don’t despair, unlike biology and physics, medicine is an applied science. An applied science discipline uses existing scientific knowledge to develop practical applications, for example: technology or inventions. This suggests an ongoing change in evidence as new information is produced. Our responsibility is making the appropriate change as meaningful information is produced.

Working with Bias

Bias is the systematic distortion of the estimated intervention effect away from the “truth”. Inadequacies in design, conduct, or reasons outside our control cause bias. All studies are biased, yet, some studies are so biased that the results reflect these design flaws.

Within research, we can evaluate risk of bias by using standardized tools. These tools are critical when discriminating the findings of a study. Understanding the right tools to use for the right study designs is a learned ability.

Consider the Source

Publishing a paper is no longer the challenge it once was. There are over 1,000 predatory journals that will publish any paper, regardless of the flaws, just for paying a processing fee.

A predatory journal bypasses the traditional peer-review process; the single safe guard for scientific veracity. A predatory journal exists only to make money. The predatory journal’s titles often mimic legitimate sources and readers may mistake the end product as a legitimate scientifically vetted paper.

Identifying Fake Research

You have likely heard about the preventative relationship between drinking red wine and heart disease, or the association of vaccinations and autism. Alternatively, perhaps you have heard that chronic fatigue syndrome is caused from a virus? All of this has been identified as “fake research”.

In fact, the number of manuscripts retracted by publications over the last several years has increased markedly. A retraction occurs when a journal publicly withdraws a paper it published because of concerns over the legitimacy of the information. Understanding fake research and how to identify it is crucial for the scientific consumer.

One Article is One Article

The multiplicity of options when designing a study make the transferability of one study’s findings quite challenging to others. Many factors can influence outcomes, including the following:

• Differences in patient sample
• Study analyses
• Dosage of intervention
• Interactions of researchers and subjects
• Fidelity of the care provided
• Timeframe of the study

Even the selection of the outcome can influence the study results. This is why the reader must understand the different nuances that make up a study, and why one should be cautious when assimilating results across the full population.

The Necessity of Critical Appraisal

Critical appraisal uses a careful and systematical process of examining research to judge its trustworthiness, and its value and relevance in a particular context. Critical appraisal provides analytical evaluations as to whether the results of the study can be believed, and transferred appropriately into other environments, like policy, further research, education or clinical practice. All work should be critically appraised, regardless of its publication, author(s), or affiliated institution.

Developing Independent Thinkers

Rogers’ diffusion of innovation curve discusses the adoption of new information among the population. Rogers defines a group known as the “late majority” who tend to be skeptical, and only adopt new ideas or approaches when a majority use them.

With research, a “late majority” conceptual approach is not a detrimental viewpoint. By remaining skeptical and fully vetting new tests, new treatments and new ideas, we reduce the risk of adopting care processes that are not necessarily beneficial, an approach that competes with a legitimate approach, or worse, harmful to the patient.

My new Rehabilitation Research Boot Camp course series is designed to improve the research consumer’s understanding of the information.