But It's Already Been Done! #ExpBio

Mar 29 2015 Published by under EB 2015

Despite the proliferation of journals and publications, our basic science efforts often do not pay off in new treatments. Part of this is time; it generally takes 15-20 years for a new science observation to be translated to the bedside.

The uglier part of the story is lack of reproducibility of preclinical studies.

In 2011 a pharmaceutical company published what proportion of preclinical studies could be replicated for drug development:

ReproFail

The bottom line: well under half! Inconsistent effects do not make good treatment prospects.

Saturday’s seesion, Reproducibility in Research: What are the problems? How can we fix them? What happens if we don’t?,   addressed these issues. Sponsored by the Policy Committee, the session brought together several speakers to address the issues.

Perhaps the most intriguing idea involved unconscious bias. This concept receives a lot of attention during discussions of diversity issues. Most of us have been conditioned to see a white male as the default for a professor or a leader. In science, the bigger problem is the way we do it. As humans, we are programmed to find evidence that supports our bias and, perhaps, minimize findings that contradict it. In forming a hypothesis, we develop a bias requiring support.

So what can we do about this unconscious process?

One though is to stop making hypotheses.

Now, before you start screaming about scientific sacrilege, just listen. The idea would be to have multiple potential thoughts about possible outcomes or to just define a scientific question without defining outcomes.

Obviously, eliminating our current well-defined hypothesis testing model will not solve all these issues, especially with such pressure for high-impact publications to get funding and keep jobs. A variety of other ideas came up, but the problem clearly does not have an easy fix.

This great session left us with more questions than answers. What a great way to start the meeting!

5 responses so far

  • Sarah says:

    I don't think your suggestion is at odds with having a hypothesis. Isn't it the way science is meant to work? You have a question. Several hypotheses are consistent with existing data. You design new experiments to tease apart these hypotheses, hopefully leaving one. Then you think some more if there are any other hypotheses that fit, and experiment some more. If you have one hypothesis left, you beat up on it and design experiments that test its unexpected predictions.

  • Pascale says:

    That is the way science is supposed to work; however, human nature is to give more weight to observations that fit our expectations and less to contradictory evidence. We do this unconsciously in many situations, not just the lab. If the hypothesis is not supported, then our data may go unpublished as negative studies, even though those data are just as important to teasing out the scientific truth.
    I don't expect to see stating a hypothesis go out of vogue in my lifetime, but it was an intriguing idea.

  • Sarah says:

    I must be narrow-minded, because I can't imagine starting a project without generating hypotheses.
    I agree unconscious bias is dangerous. There are definitely theories, data and techniques that I've been sad to abandon. But I feel like unconscious bias, at least in the lab, can be minimised by acknowledging it exists and encouraging dissent. I would have thought scientific meetings would also be a good place to get an independent critique, but it seems people are too nice. or maybe not really paying attention. And it's true, people like to present a story, rather than "here is some data that I can't get my head around". I might try submitting that as an abstract and see how it goes!

  • AScientist says:

    It's not the hypotheses that are the problem, it's expecting that the answer to the hypothesis should be yes (or no). I always flip the hypothesis (anything can be stated as an opposite) and try to argue that the data support the opposite. Like on a debate team, you should take the argument opposite to your own beliefs and fight for that position! If you can't, then you've got a result.

    • whizbang says:

      Once again, that's the way science should work. The problem lies with those who fall in love with their hypothesis and are unable to "kill their darlings." Some scientists may also fear that killing their hypothesis will be seen as negative in their careers.
      Like I said, this is clearly food for thought, and a discussion we need to have. How do we make absolutely sure that everything has been done, especially when authors may consciously or unconsciously cherry-pick their data? Peer review can only do so much, and right now funding for replication of the work of others is waaaaaaay down the list of probabilities.

Leave a Reply