How do we get from reports of success to tragedy or disappointment?
- From “tPA (a clot-busting drug) saves in acute stroke” to “clotbusters increase death or have no effect on mortality.”
- From “antidepressants prevent suicides” to “antidepressants cause suicides.”
- From “lowering cholesterol reduces cardiovascular deaths” to “lowering cholesterol has no effect”?
We often get conflicting headlines and get whipped from one side to another. How does this happen?
One explanation is that reports of success may be inaccurate. Epidemiologist John P.A. Ioannidis reviewed 49 frequently cited clinical research studies, all published in well-respected journals, and found that subsequent investigations threw results of 14 into question. In 7 cases a supposedly effective intervention did not work; in 7 more the drug or procedure did not work as well as originally claimed.
(See http://jama.ama-assn.org/cgi/content/abstract/294/2/2180)
If a researcher wants to fool us, two ways are to hide data or suppress it.
Hiding [data] in plain sight
Take the example of the clot-busting drug, tPA (tissue plasminogen activator). Initially, there was a struggle between neurologists, who loved the drug, and emergency room doctors who weren’t so sure it was safe. Knowing this, I was surprised when the American Heart Association (AHA) in 2000 gave the drug its highest recommendation as an intervention for acute stroke, claiming it saved lives. Some emergency room physicians questioned that recommendation and there were rumored connections between AHA and Genentech, the company that makes the drug.
Five studies showed that tPA either increased mortality or had no effect on mortality. Results from a single study, conducted by the National Institute of Neurologic Disorders and Stroke (NINDS), were submitted to the Food and Drug Administration and used to gain approval. That trial showed that tPA had no effect on mortality following a stroke.
So, how did AHA get away with the claim that it saved lives? I talked to various doctors about this and it seemed that the use of the word “and” was the key. The NINDS study looked did not report on the drug’s impact on death, as a specific outcome, but at “death and disability.” So if tPA appeared to reduce post-stroke disability, but not the rate of death, then the researchers could truthfully report an improvement in “death and disability.” This type of combined endpoint is a brilliant marketing tool that allows drug companies to emphasize a trivial outcome and shift attention away from a more serious one.
Reporters should always look closely at combined endpoints.
It’s worth looking at the connection between Genentech and the AHA. The drug company had contributed more than $11 million to the AHA and specifically donated $250,000 to rebuild the organization’s headquarters in Houston. Moreover, seven of the nine AHA panelists that recommended tPA had ties to Genentech or its marketing partner.
As a result of the AHA recommendation, the “brain attack campaign” was launched and hospitals began advertising their certified, specialized stroke centers. And this has happened despite the fact that the AHA has withdrawn its claim that tPA saves lives.
The lesson here is that it’s important for journalists to look at the entire body of evidence, both published and unpublished. Don’t just accept the “good news” from a single study.
As reporters, it is important to acknowledge controversy when it exists and search out sources without ties to industry. We need to ask doctors about potential conflicts of interest.
Replication of results is also important. Should there be another study to determine if the drug really works? The manufacturers of tPA said another study might be a good thing for American, but not necessarily a good thing for the company.
It’s also important to look at the term “significant.” Results can be statistically significant but not necessarily clinically significant. There even may be a small significant benefit but also negative side effects. So, always look for results that have clinical significance.
(Dr. Gregory Harshfield commented that journals allow scientists to note the statistical significance of findings only in an article’s abstract—the part most likely to be read. He advised always looking at an article’s last paragraph to see that is said about clinical significance.)
So, the real question is, did the marginal changes of tPA translate into real clinical benefit?
If we are looking at clinical benefit, we could ask: why tPA, why not gyms? Clotbusting drugs benefit only 3 to 4 of every 1,000 stroke patients, yet the “brain attack campaign” has spent millions of dollars promoting its use.
What if the same amount of money had been invested in workplace gyms to promote exercise? They are a proven way to improve employee health, and the AHA itself says that regular exercise reduces the likelihood of having a stroke by 27%.
Net benefit and antidepressants
It’s important to recognize that “benefit” does not necessarily mean “net benefit.”
Journalists must note how benefits are expressed, and there is cause for concern if they are reported:
- On one scale, but negative impacts are reported on a different scale,
- From an “unblinded” arm, meaning that study participants knew what treatment they were receiving,
- Without regard to harms—separate clinical trials are conducted to test efficacy versus safety.
[A major study of depression among adolescents (Treatment for Adolescents with Depression, or TADS) illustrates how failure to report negative data can allow enthusiastic news about a treatment’s success to run wild. The study unequivocally showed that a combination of talk therapy and Prozac aided depressed young people. Later, however, concern grew that Prozac might actually increase suicidal behavior in depressed teens. For a detailed account, see http://www.pubmedcentral.nih.gov/articlereader.fcgi?artid=518915]
How do we get from Prozac as a huge success to Prozac as a link to suicide? Several issues in the study are worth considering:
- Safety signals and antidepressants
- Net benefit—balance risk and benefit.
- Problem of underpowering (multiple efficacy studies showed a ‘signal” of increased suicidality, but because the studies were “underpowered” for means of the safety analysis, they were dubbed nonsignificant)
Basically, a much larger study was needed to capture net benefit.
When challenged about the way the study reported positive benefits but no negative outcomes, the principal investigator refused to provide all the data needed to review negative effects.
Unfortunately, two methodological problems—“underpowering” and publication bias—obscured what the FDA eventually found in a pooled analysis of randomized controlled trials: the suicide rate were doubled.
The underlying problem was that, with Prozac, as with most psychotropic drugs, many people respond really well, but some do really badly. Among those youth who took Prozac, about 38% became “activated,” which means very anxious, and 18% had same effect without Prozac.
This lead to an important tip for reporters: Always ask what happens to people who were not treated with anything, because taking that into account often shrinks the net benefit of an intervention.
Reporters sometimes doubt that we can’t really analyze studies and identify methodological flaws because we’re not epidemiologists or statisticians.
My goal is to shake your faith in the big studies—not to make you epidemiologists. Take time to talk to doctors and researchers until you understand what was done and why and how it worked. Don’t let doctors shoot you down or intimidate you.
When studies are too small to detect dangers associated with a drug, researchers sometimes combine results from a bunch of small studies. But this doesn’t necessarily yield good information.
And when the results of a study show that an intervention doesn’t work, that study may be suppressed simply because journal editors would rather publish studies showing positive results.
When researchers make clinical recommendations based on a new study, reporters need to consider where the study fits in the hierarchy of evidence. The best kind of evidence comes from randomized, controlled double-blind studies. Experiments done in test tubes or in animals, opinions voiced by experts, accounts of individual cases, or comparisons of cohorts of people should not be the basis for clinical decisions.
The cholesterol debate
Lipitor and other drugs in the so-called “statin” family, all of which lower serum cholesterol levels to varying degrees, are among the best-selling drugs in the world. And their story offers cautionary tales for reporters.
When we consider how beneficial statins really are, it’s important to look at how their benefits have been framed in terms of
- Relative or absolute risk reduction
- Cause-specific vs all-cause mortality
- Which populations are studied
- Use of surrogate markers
One statin study, for example, reported a 32% reduction in the relative risk for cardiovascular mortality among treated patients. Translated into absolute numbers, however, that meant that one person in 715 would benefit. Relative risk inflates impact of benefit.
If researchers want to minimize negative effects, which every drug or intervention has, they are likely to use absolute risk because it makes effects look smaller.
So, again, it’s important to look at the net benefit—overall mortality, not just cardiovascular benefit. Studies of statins may focus solely on reductions in cardiovascular mortality, giving the impression that everyone lived happily ever after. In fact, though, the treated group died at the same rate as other people – but they died of non-cardiovascular causes such as kidney failure or cancer.
As one researcher (Beatrice Golomb (MD, PhD) put it, “In all groups except middle-aged men with heart disease—or at high risk for it—whatever benefits statins have to the heart are completely offset by [an] increase in other causes of death and serious problems.”
|
Dying with a corrected cholesterol level is not a successful outcome in my book—John Abramson, M.D., author of Overdosed America: The Broken Promise of American Medicine
|
Statins are the biggest sellers of all time. However, they are sold through advertising not science.
Simply having an elevated cholesterol level, without heart disease or diabetes, is not a death sentence. In fact, its importance may be marginal. The same is true for high blood pressure. These are surrogate markers for disease, but they are not diseases.
Niche marketing: the “race drug”
V-HEFT was a large clinical trial designed to see whether a combination of two generic anti-hypertension drugs would benefit patients with congestive heart failure (CHF). The idea was that if this combination was effective, its makers could obtain a patent to market it as a new agent. In a large and varied sample, however, the benefits were negligible for the whole study population. A subgroup analysis, however, suggested statistically significant benefits among participants who identified themselves as African American.
So the company decided to market the drug to black people and sponsored a study (A-HEFT) that “proved” benefit, but did not include white participants as a comparison group.
In June 2005, the FDA approved BiDil for use in African Americans only, and the drug thus became the first “race drug.”
What is BiDil? Essentially, it combines two established drugs, isosorbide dinitrate and hydralazine. A 30-day course of those drugs – taken as separate tablets – costs about $52, whereas 30 days of BiDil costs about $195. This is an example of how drug companies try to exploit our anxieties about race, gender and social group.
An editorial in The New England Journal of Medicine summarized the situation:
“A-Heft has not established that adding isosorbide dinitrate and hydralazine to conventional therapy for heart failure yields greater benefits for blacks than for other racial or ethnic groups…[This] race-specific drug was driven in large measure by regulatory and market incentives.”
What journalists can do
How can journalists make sense of medical studies?
- Think devious (it’s a must)
- Read the following books:
- Evaluating Clinical Research by Furberg and Furberg
- Epidemiology in Medicine by Hennekens and Buring
- Medical Use of Statistics by Bailar and Mosteller
- Consult independent experts; contact Brownlee.Lenzer@gmail.com or www.overtreated.com. for a free (and growing) list of experts who have no conflicts of interest and are helpful to reporters.
- Don’t let experts intimidate you
- Always ask about financial conflicts of interest
- Review the 10 criteria used by experts who evaluate news stories for www.healthnewsreview.org
|
return to Knight Health home 