It's tough to make predictions

“Generic HIV drugs 'less effective',” said the BBC headline. The article continued: “Any rise in the use of cheaper, non-branded HIV drugs could see more patients with treatment failure, doctors warn.”

This news looked worrying, and as soon as it was posted on the BBC website, people flagged it up on social media websites.

In the last issue of HTU (see The generic generation, HTU 213), we looked at how many people’s HIV drug regimens will probably soon include cheaper generic versions of some antiretrovirals. Potentially, first-line regimens may be entirely made up of generic drugs within five years.

We discussed the cost advantages, but also whether this might reduce the efficacy of people’s meds. This might happen if, say, a double- or triple-drug fixed-dose combination (FDC) pill such as Atripla (tenofovir, FTC and efavirenz) or Kivexa (abacavir and 3TC) was easier to take than the drugs as separate pills.

We also looked at whether there might be pressure to substitute a slightly less-effective drug, available as a generic, for a slightly more effective one that was still on-patent.

But the BBC report looked like it was saying generic drugs were worse because they were generics. This is still a widely held misunderstanding.

In fact, in order to sell their drugs, generic manufacturers have to show that the active ingredients in their products are chemically identical to, and reach levels in the body which are no more than 5% above or below, branded-drug concentrations.

The BBC report seemed to have found a study that said the opposite. When we looked into it, however, the study they cited¹ – in the prestigious Annals of Internal Medicine journal – seemed familiar.

We realised we’d reported on it last summer, when its lead author Rochelle Walensky, of Harvard Medical School, presented it at the International AIDS Conference in Washington.

The study did not say generics were inferior. In fact, the news it gave was almost entirely good. Using generic HIV drugs would save the US health system nearly a billion dollars a year.

However, the researchers went further, and this is where the confusion started. They asked: let’s just suppose that generics are somewhat inferior. What if adherence is a bit poorer to separate pills? What if 3TC is a little less potent than FTC? What if people just don’t like getting different-coloured medicines?

So they fed into their model a worst-case scenario, within the boundaries of what’s at all likely. What if generic FTC and efavirenz (plus branded tenofovir) were 7% less potent than the three drugs put together in Atripla?What if the generics failed for one in 20 patients in the first year, rather than one in 40, as is the case with Atripla?

If you made those assumptions, the mathematical model they used predicted that you’d lose 4.4 months from the average HIV-positive person’s lifespan. Those 4.4 months were what the BBC report seized on.

People consistently misunderstand mathematical models. We treat them as infallible oracles and then feel lied to when they turn out to be wrong. But a mathematical model is not intended to be reality, which is far too complex to model.

Furthermore, people tend to confuse input with output, which is what happened here. The BBC report assumed the reduced effectiveness of generics was a finding, the sausage that came out of the machine. It wasn’t; it was one of the assumptions used as input, the meat fed into the mincer.

This wasn’t helped by Harvard Medical School putting out a press release that made it look as if the lower efficacy of generics was a finding.

“The switch from branded to generic antiretrovirals would place us in the uncomfortable position of trading some losses of both quality and quantity of life for a large potential dollar savings,” Walensky is quoted as saying. ‘Would place us’ sounds like a prediction, doesn’t it? Not ‘would place us if our assumptions turn out to be true’.

The meat in this case was also of decidedly dodgy quality. There’s only one study referenced in the paper² that found better clinical outcomes in people given a combination pill than separate drugs. The difference was quite significant in this case, but the study group – homeless people in San Francisco – may have very specific difficulties with adherence. Other studies reported on aidsmap.com either found no difference³ in clinical outcomes in people taking more pills or didn’t take into account the fact that people on more pills are likely to be on second-line HIV regimens and may be sicker.⁴

They also cite three studies suggesting that FTC could be slightly more effective than 3TC, but this is disputed by a World Health Organization (WHO) analysis.⁵

All these factors could mean that taking people off combination pills could impact on treatment success, and it’s perfectly valid to calculate what the impact could be. But if the assumption that using generics might impact on adherence or viral load doesn’t stand up, then you save your $1 billion with no shortening of life expectancy.

WHO wrote to the Annals journal, criticising the way the research was done and adding that adherence was only a problem if all-generic FDCs were forbidden by patent law. License generic tenofovir, and you solve the problem.

In the end, the BBC changed its headline to ‘Study questions generic HIV drug use’, which it did. But they still write “[Researchers] say, based on modelling and trial data, that generic medicines may be slightly less effective.”

Enough, still, to worry some patients who find a generic pill in their clinic prescription. But like most predictions, it’s dependent on the assumptions we make. Garbage in: garbage out. As the immortal baseball player Yogi Berra said: “It’s tough to make predictions, especially about the future.”