Scientists Secret Recipe

“A protocol is a recipe,” said Jonathan Livny, a senior research scientist at the Broad Institute who has collaborated with Transfyr. “You can give somebody a recipe, and it’s not going to make them a great chef.”

Like apprentice chefs, scientists spend years in training, shadowing experts, asking questions and trying out procedures for themselves.

To make an experiment work, they may have to make thousands of minor decisions over the course of a day. A tube needs to be shaken — let it whir on a vibrating platform, or just flick it back and forth by hand?

“The really good people sometimes don’t know why they’re that good,” Dr. Livny said. “They don’t remember the 20 times they did something another way and it failed. They’ve blocked all that sadness out.”

In the 1960s, the Hungarian chemist and philosopher Michael Polyani gave this mysterious expertise a name: tacit knowledge. “We know more than we can tell,” he liked to say.

Scientists have generally come to agree with Polanyi that tacit knowledge is an essential part of research. But it can also slow down progress.

In theory, science moves forward as researchers build on previous work. When a new study comes out, other scientists will attempt to replicate the work. If the original result was correct, a replication ought to produce the same results.

Surprisingly often, it doesn’t. And tacit knowledge is part of the problem. Scientists can’t know exactly how to replicate a study simply by reading about it.

Renee Wegrzyn, a co-founder of Transfyr, became keenly aware of this gap while doing research as a postdoctoral researcher. She jotted down records of her experiments on proteins in a lab notebook, but it captured only a fraction of her efforts.

When Dr. Wegrzyn published her results, she had to distill her records even further. “I did a postdoc that was three years long, and then it got summed up in a three-page paper,” she said.

Missing from those notes and reports were all the little decisions Dr. Wegrzyn had made while performing her research.

“All of those details could matter, but we just don’t know if they matter,” said Brian Nosek, an expert on replication at the University of Virginia who is not involved in Transfyr.

When a replicated study fails to produce the original results, that may mean the original study was wrong. Or maybe it was right, and the replicating scientists made an unwitting blunder. It can be hard for teams of scientists to agree on what happened.

In 1993, for example, the psychologist Frances Rauscher and her colleagues reported that people who listen to Mozart get a temporary boost on reasoning tests. The buzz around the so-called Mozart effect led other researchers to run experiments of their own.

Many of them couldn’t find any significant benefit, but Dr. Rauscher brushed off these failures. The other scientists didn’t find the Mozart effect because they weren’t good enough at running experiments, she said — they were not good scientific chefs.

The search for the Mozart effect went on for years. In the end, other scientists found little evidence that it exists.

Conflicts like these led some scientists to search for new ways to communicate tacit knowledge. In 2006, the biologist Moshe Pritsker created a journal called JoVE, which published videos of scientists carrying out complex experiments.

In 2014, the biologist Lenny Teytelman created a place where scientists could share their protocols and talk about them. Researchers use the website, called Protocols.io, to help train new members of their labs when the seasoned scientists with magic hands have moved on.

With Transfyr, Dr. Wegrzyn and Ms. Wagner are building a system that can slurp up enormous amounts of data about what happens in a lab, in the form of video, audio and sensor logs from lab equipment. Transfyr even tracks the source of supplies, down to the lot numbers of glove boxes.

(Why? “Maybe there’s a fume coming off the glove that is changing your experiment,” Ms. Wagner said.)

Dr. Nosek said that this new approach might reveal some secrets about why experiments succeed or fail. “What I really like about them is they’re going gangbusters into trying to unpack every detail,” he said. “It makes a lot of sense to go all in, and then figure out what actually matters.”

- Author: Carl Zimmer, The New York Times

Read more...