Category: Methodology

Double-Blind Trials Are Not the Only Source of Sound Knowledge

(p. 1) . . . while all doctors agree about the importance of gauging the quality of evidence, many feel that a hierarchy of methods is simplistic. As the doctor Mark Tonelli has argued, distinct forms of knowledge can’t be judged by the same standards: what a patient prefers on the basis of personal experience; what a doctor thinks on the basis of clinical experience; and what clinical research has discovered — each of these is valuable in its own way. While scientists concur that randomized trials are ideal for evaluating the average effects of treatments, such precision isn’t necessary when the benefits are obvious or clear from other data.
Clinical expertise and rigorous evaluation also differ in their utility at different stages of scientific inquiry. For discovery and explanation, as the clinical epidemiologist Jan Vandenbroucke has argued, practitioners’ instincts, observations and case studies are most useful, whereas randomized controlled trials are least useful. Expertise and systematic evaluation are partners, not rivals.
Distrusting expertise makes it easy to confuse an absence of randomized evaluations with an absence of knowledge. And this leads to the false belief that knowledge of what works in social policy, education or fighting terrorism can come only from randomized evaluations. But by that logic (as a spoof scientific article claimed), we don’t know if parachutes really work because we have no randomized controlled trials of them.

For the full commentary, see:
PAGAN KENNEDY. “The Thin Gene.” The New York Times, SundayReview Section (Sun., NOV. 27, 2016): 1 & 6.
(Note: ellipsis added.)
(Note: the online version of the commentary has the date NOV. 25, 2016.)

The academic article calling for double-blind randomized trials to establish the efficacy of parachutes, is:
Smith, Gordon C. S., and Jill P. Pell. “Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge: Systematic Review of Randomised Controlled Trials.” BMJ 327, no. 7429 (Dec. 18, 2003): 1459-61.

Science Can Learn Much from Outliers “Who Are Naturally Different”

(p. 1) Abby Solomon suffers from a one-in-a-billion genetic syndrome: After just about an hour without food, she begins to starve. She sleeps in snatches. In her dreams she gorges on French fries. But as soon as she wakes up and nibbles a few bites, she feels full, so she ends up consuming very few calories. At 5 feet 10 inches tall, she weighs 99 pounds.
Now 21 years old, she is one of the few people in the world to survive into adulthood with neonatal progeroid syndrome, a condition that results from damage to the FBN1 gene.
. . .
(p. 6) Dr. Chopra told me that, as far as medical science is concerned, Abby Solomon is worth thousands of the rest of us.
. . .
“Nothing comes close to starting with people who are naturally different,” he said. This is why he searches out patients at the extreme ends of the spectrum — those who are wired to weigh 80 pounds or 380 pounds. He said, “We have the opportunity to help a bigger swath of humanity when we learn from these outliers.”
In 2013, after hearing about Ms. Solomon’s unusual condition from another patient, he asked her to visit his clinic. Ms. Solomon warned him that she would be able to carry on a conversation for only 15 minutes before she needed to snack on chips or a cookie. That remark inspired a revelation. Dr. Chopra realized that “she had to eat small, sugary meals all day to stay alive, because her body was constantly running out of glucose,” he said.
The clue led Dr. Chopra and his colleagues to their discovery of the blood-sugar-regulating hormone, which they named asprosin. Ms. Solomon’s natural asprosin deficiency keeps her on the brink of starvation, but Dr. Chopra’s hope is that an artificial compound that blocks asprosin could be used as a treatment for obesity. He and his team have already tested such a compound on mice, and found that it can reverse insulin resistance and weight gain.

For the full commentary, see:
PAGAN KENNEDY. “The Thin Gene.” The New York Times, SundayReview Section (Sun., NOV. 27, 2016): 1 & 6.
(Note: ellipses added.)
(Note: the online version of the commentary has the date NOV. 25, 2016.)

Jane Jacobs Studied the “Mess of Everyday Life”

(p. C6) The decidedly unpredictable and unscientific mess of everyday life was the passion of the urban theorist Jane Jacobs. For her, studying the street and the city was the key to understanding how things work. Robert Kanigel’s “Eyes on the Street: The Life of Jane Jacobs” has taken a place on my bookshelf right next to Robert Caro’s landmark biography of her nemesis, Robert Moses.

For Bierut’s full book recommendations, see:
Michael Bierut. “12 Months of Reading.” The Wall Street Journal (Sat., December 10, 2016): C6.
(Note: the online version of the review has the date Dec. 7, 2016, and has the title “Michael Bierut on Jane Jacobs.”)

The book recommended, is:
Kanigel, Robert. Eyes on the Street: The Life of Jane Jacobs. New York: Alfred A. Knopf, 2016.

“Celebration of Big Data” Becomes “Funeral”

(p. B8) DENVER — At the governor’s mansion here on Friday [November 11, 2016], past the columned entryway and the French chandeliers, Emmy Ruiz placed a hand on the shoulder of a fellow Hillary Clinton operative. “It’s like we’re at a funeral,” said Ms. Ruiz, dressed — perhaps coincidentally — in black.
Just days after Donald J. Trump’s surprise presidential victory, the nation’s professional political forecasters and persuaders — the pollsters, the ad creators, the campaign strategists — gathered in Denver for their annual convention. It was supposed to be a celebration of big data and strategic wizardry for a multibillion-dollar industry that has spent nearly a century packaging political candidates.
Instead, the conference of the International Association of Political Consultants felt like a therapy session for a business in psychological free fall.

For the full story, see:
JULIE TURKEWITZ. “At Conference, Political Consultants Wonder Where They Went Wrong.” The New York Times (Tues., NOV. 15, 2016): B8.
(Note: bracketed date added.)
(Note: the online version of the story has the date NOV. 14, 2016.)

Economist Removed from Plane for Scribbling Math

The seatmate was wrong to think the scribbling was Arabic, but was right to be alarmed.

(p. A13) In May [2016], an Italian economist from the University of Pennsylvania was removed from an American Airlines flight in Philadelphia after his seatmate became alarmed, thinking that the math he was scribbling on a piece of paper was Arabic, The Washington Post reported.

For the full story, see:
CHRISTINE HAUSER. “American Airlines Orders 2 Muslim American Women Off a Long-Delayed Flight.” The New York Times (Sat., AUG. 5, 2016): A13.
(Note: bracketed year added.)
(Note: the online version of the story has the date AUG. 5, 2016, and has the title “2 Muslim American Women Ordered Off American Airlines Flight.”)

Many Discoveries Take a Long Time Because “No One Really Looked”

Periods are a strange phenomenon. We don’t know why humans have them, or, to look at it another way, why most other animals don’t. Scientists say only 1.5 percent of mammal species have periods, and most of those are primates like us. The ranks of the menstrually afflicted grew a little bit recently, as researchers learned that female spiny mice have periods, too. They shared their findings on the bioRxiv preprint server.
. . .
Why did it take scientists so long to notice that these curious creatures were part of the period posse? “The answer, as with many discoveries in science, is that no one really looked,” said Hayley Dickinson, a reproductive physiologist and long-time spiny rat researcher at the University of Monash. “Everyone knew that rodents didn’t menstruate.”

For the full story, see:
Nowogrodzki, Anna. “First Rodent Found with a Humanlike Menstrual Cycle.” Nature (Fri., June 10, 2016).
(Note: ellipsis added.)

The preprint of the research mentioned above is:
Bellofiore, Nadia, Stacey J. Ellery, Jared Mamrot, David W. Walker, Peter Temple-Smith, and Hayley Dickinson. “First Evidence of a Menstruating Rodent: The Spiny Mouse (Acomys Cahirinus).” bioRxiv (June 3, 2016).

Majerus Did Not Need a Randomized Trial to Know that Aspirin Prevents Heart Attacks

(p. A21) Philip W. Majerus, a biochemist who was credited as being the first to theorize that taking small doses of aspirin regularly can prevent heart attacks and strokes in vulnerable patients, died last Wednesday [June 8, 2016] at his home in St. Louis. . . .
. . .
Even before his findings were confirmed in a study by other researchers a decade later, Dr. Majerus was taking aspirin daily.
“I was already convinced that aspirin prevented heart attacks,” he recalled in the journal Advances in Biological Regulation in 2014. “I was unwilling to be randomized into a trial where I might end up with the placebo. I refused to participate.”
Dr. Majerus recommended that “all adults should take an aspirin daily unless they are among the few percent of the population that cannot tolerate the drug.” The cardiovascular benefit of aspirin was fully achieved by 50 to 75 milligrams daily, he said, and “there is no evidence that branded aspirin, which is much more expensive, is in any way superior to the generic version.”
Later studies found that for people in their 50s who are vulnerable to heart disease, taking daily doses of aspirin reduces the risk of heart disease.
. . .
Investigating how aspirin inhibited clotting, Dr. Majerus concluded that the medicine modified an enzyme that leads to the formation of a platelet-made molecule that constricts blood vessels and aggregates platelets. The pills’ effect lasts for the platelets’ life span, typically about two weeks.
“Phil Majerus, more than any other individual, has produced the most original body of work on biochemistry of platelets as it relates to thrombosis,” Prof. Joseph L. Goldstein, a Nobel laureate at the University of Texas Southwestern Medical Center in Dallas, said when the Bristol-Myers Squibb Award was announced.

For the full obituary, see:
SAM ROBERTS. “Dr. Philip Majerus, Who Recognized Heart Benefits of Aspirin, Is Dead at 79.” The New York Times (Weds., JUNE 15, 2016): A23.
(Note: ellipses, and bracketed date, added.)
(Note: the online version of the obituary has the date JUNE 14, 2016, and has the title “Dr. Philip Majerus, Who Discerned Aspirin’s Heart Benefits, Dies at 79.”)

Half of Important Psychology Articles Could Not Be Replicated

(p. A1) The past several years have been bruising ones for the credibility of the social sciences. A star social psychologist was caught fabricating data, leading to more than 50 retracted papers. A top journal published a study supporting the existence of ESP that was widely criticized. The journal Science pulled a political science paper on the effect of gay canvassers on voters’ behavior because of concerns about faked data.
Now, a painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday [August 27, 2015] in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.
The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.

For the full story, see:
BENEDICT CAREY. “Psychology’s Fears Confirmed: Rechecked Studies Don’t Hold Up.” The New York Times (Fri., AUG. 28, 2015): A1 & A13.
(Note: bracketed date added.)
(Note: the online version of the story has the date AUG. 27, 2015 and has the title “Many Psychology Findings Not as Strong as Claimed, Study Says.”)

The Science article reporting the large number of psychology articles that proved unreplicable, is:
Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (Aug. 28, 2015): 943.

Many Empirical Research Results Are False

(p. B7) Research on 100 studies in psychology found in 2015 that more than 60% couldn’t be replicated. Similar results have been found in medicine and economics. Campbell Harvey, a professor at Duke University and president of the American Finance Association, estimates that at least half of all “discoveries” in investment research, and financial products based on them, are false.
. . .
Brian Nosek, a psychology professor at the University of Virginia and executive director of the Center for Open Science, a nonprofit seeking to improve research practices, has spent much of the last decade analyzing why so many studies don’t stand up over time.
Because researchers have an incentive to come up with results that are “positive and clean and novel,” he says, they often test a plethora of ideas, throwing out those that don’t appear to work and pursuing those that confirm their own hunches.
If the researchers test enough possibilities, they may find positive results by chance alone — and may fool themselves into believing that luck didn’t determine the outcomes.

For the full commentary, see:
JASON ZWEIG. “Chasing Hot Returns in ‘Smart-Beta’ Can Be Dumb.” The Wall Street Journal (Sat., Feb 13, 2016): B1 & B7.
(Note: ellipsis added.)
(Note: the online version of the commentary has the date Feb 12, 2016, and has the title “Chasing Hot Returns in ‘Smart-Beta’ Funds Can Be a Dumb Idea.”)

“Science Is Not a Body of Infallible Work, of Immutable Laws”

(p. 1) . . . , “Failure: Why Science Is So Successful” is a breath of contemplative fresh air. Stuart ­Fire­stein, a professor in the department of biological sciences at Columbia University, is best known for his work on ignorance, including inviting scientists to speak to his students about what they don’t know. In a tone reminiscent of Lewis Thomas’s “The Lives of a Cell,” the book is a collection of loosely interwoven meditations on failure and scientific method.
. . .
If we succeed by failing, then we should be freed from the monolithic road to academic tenure; science should be taught as an adventure in failure. With a delightful combination of feigned naïveté and keen eye for the messy ways that great discoveries occur, he goes so far as to suggest writing a grant proposal in which you promise to fail better. He knows this isn’t how the world works, but nevertheless argues that change will take place “when we cease, or at least reduce, our devotion to facts and collections of them, when we decide that science education is not a memorization marathon, when we — scientists and nonscientists — recognize that science is not a body of infallible work, of immutable laws of facts. . . . And that most of what there is to know is still unknown.”

For the full review, see:
ROBERT A. BURTON. “Error Messages.”The New York Times, SundayReview Section (Sun., Jan. 3, 2016): 8.
(Note: first two ellipses added; third ellipsis in original.)
(Note: the online version of the review has the date DEC. 29, 2015, and has the title “‘Black Box Thinking’ and ‘Failure: Why Science Is So Successful’.”)

The book under review, is:
Firestein, Stuart. Failure: Why Science Is So Successful. New York: Oxford University Press, 2016.