The modest protective effect of the HIV vaccine combination used in the Thai vaccine trial announced last month is statistically significant and not the result of massaging the statistics to produce a positive result, study investigators said at the AIDS Vaccine 2009 conference in Paris today.
The protective effect of the vaccine had been called into question following the release of headline results from the study in late September, which showed that the vaccine reduced the risk of HIV infection by 31%, a result that was statistically significant.
However, at the time of the announcement attention was drawn to the very wide confidence interval of this estimate. Statisticians estimated that the result lay between 1.1% and 51.6% with 95% confidence; in other words, there was a 5% possibility that the result lay outside these bounds.
Subsequently, news leaked on other data from the trial, which suggested that if analysed in other ways, the results were not statistically significant. In particular, concerns were raised about why the results of the less encouraging per-protocol analysis were not released to the world’s press at the same time as the headline result.
Today Dr Nelson Michael of the US Military HIV Research Program explained the three analyses of the study and why the investigators chose to release one particular analysis last month, ahead of the AIDS Vaccine 2009 conference.
Three analyses of the study were planned:
- An intent-to-treat analysis, which included all trial participants randomised, regardless of whether they received the vaccine or not.
- A modified intent-to-treat analysis, which excluded any randomised participant who was discovered to have evidence of HIV infection through viral load testing prior to vaccination. This analysis includes study participants who may have missed some of the vaccinations, and goes some way to estimating the real-life efficacy of the vaccine.
- A per-protocol analysis which excluded all participants who missed any doses of the vaccine, or who received vaccinations on days other than scheduled study visits. A per-protocol analysis in a vaccine study is designed to test the efficacy of a particular vaccination schedule, and is most useful in a study that is intended to lead to product licensing.
The headline results of the study, released last month, indicated that the vaccine reduced the risk of infection by 31%, but the statistical confidence intervals were very wide (between 1.1% and 51%). Nevertheless the result was statistically significant (p=0.04).
Other results were not released to the press in September, said Dr Jerome Kim of the US Military HIV Research Program, because they were already embargoed prior to the AIDS Vaccine 2009 conference. But researchers had to honour their commitment, made at the outset of the study, that the Thai people would be the first to hear the result of the trial, so the modified intent-to-treat analysis was revealed in September.
The strict intent-to-treat analysis, however, showed that the reduction in the risk of infection in the vaccine group was not statistically significant (-26.4%, 95% confidence interval -4% to 47.9% (p=0.08). This analysis included seven participants who turned out to have been infected at the time of randomisation, prior to the first shot of vaccine. These infections were identified when stored samples taken at baseline were tested after the participants tested positive at the time of their fourth vaccination.
Similarly, the per-protocol analysis, which excluded around 25% of study participants and thus failed to capture around 31% of infections that occurred during the study (just over half of them in the first six months), showed a reduction in risk of 26.2% (95% confidence interval -13.3% to 51.9%, p=0.16).
The full results were also released online today by the New England Journal of Medicine and, in their report, the investigators note that the result of the modified intent-to-treat analysis remained statistically significant regardless of the statistical method used to test significance. The investigators used no fewer than six different tests to query the magnitude and robustness of the effect observed, and found that all produced results within the same range, a p value lying somewhere between 0.03 and 0.05.
The researchers concluded that the modified intent-to-treat analysis, which had always been planned as part of the study and which was used throughout the study as the primary analyses to determine if the trial should be stopped on the grounds of futility, gave the most clinically useful information for making future decisions about how to take the vaccine forward, since it most closely reproduced the likely conditions in which a vaccine’s effectiveness would be judged in the field.
Vaccine more effective in lower-risk participants – effect may wane
Perhaps of greater long-term significance for the HIV vaccine field, Michael Nelson presented details of sub-group analyses which suggested that the vaccine exerted a greater protective effect in people with fewer sexual partners, although the study was not powered to produce definitive evidence on this question.
47.5% of participants were classified as low-risk, reporting one or no sexual partners in the six months preceding entry to the study and judging themselves to be at low risk. This judgement was corroborated by their answers to questions about their sexual partners; individuals with partners who were commercial sex workers, injecting drug users, HIV-positive or men who have sex with men were classified as medium or high risk. Individuals who themselves fell into any of these categories were classified as high risk. Only 24% of participants were classified as high risk.
When the vaccine efficacy was compared between the high-risk and the low- and medium-risk groups, there was a suggestion that individuals in the low- and medium-risk groups experienced a greater reduction in the risk of infection if they received the vaccine. Whereas the vaccine efficacy was 40% in the low-risk group and 46% in the medium-risk group, it was only 3.7% in the high-risk group. (Confidence intervals overlapped for all three estimates.)
Dr Jerome Kim of the US Military HIV Research Program stressed that the finding was not statistically significant and that the study was not powered to look at these questions, but said the results were intriguing.
Similarly intriguing was the trend towards a greater protective effect in the first year, indicating that the effect of the vaccine combination may wane over time.
Although strong cell-mediated immune responses were observed against HIV envelope and gag proteins in a modest number of participants, it is still impossible to determine if any immunologic parameters correlate with protection from infection, or to unpick the relative contributions of the two vaccines used in the study.
“These data need to be ripped apart by many, many people. More questions raised by the data will help the field focus on the most important studies that need to be done as we go forward,” Mitchell Warren of the AIDS Vaccine Advocacy Coalition told aidsmap.
The investigators are due to launch a website submission system soon at www.aidsresearch.gov where investigators anywhere in the world who wish to carry out research on stored samples from the study can propose experiments that may shed light on what happened to trial participants and how the vaccine may have worked.
The trial investigators have also asked the HIV Vaccine Enterprise and the World Health Organization to convene a meeting with ethicists, researchers and advocates to determine whether future vaccine studies can use a placebo group or whether, given the modest efficacy of the vaccine combination used in the Thai trial, it is no longer ethical to do so in some or all future studies.
Supachai Rerks-Ngarm et al. Vaccination with ALVAC and AIDSVAX to prevent HIV infection in Thailand. New England Journal of Medicine, online advance publication, October 20, 2009.