Gay men recall seeing mass media campaign, but impact on behaviour remains unclear

A media campaign promoting condom use and sexual health check-ups to gay and other men who have sex with men in Scotland was recalled by 60% of the target group, but there is no evidence that it had an impact on risky sexual behaviour. While the data suggest some associations between having seen the campaign and HIV testing behaviours, they are difficult to interpret.

Writing in BMC Public Health, the researchers draw attention to the methodological challenges of evaluating mass media campaigns and call for alternative approaches to be employed.

The campaign

The Make Your Position Clear campaign was delivered in 2009 and 2010 in Glasgow and neighbouring regions of Scotland. Its aims were to promote the use of condoms and water-based lubricant during all instances of anal intercourse, and to promote regular sexual health check-ups, including HIV testing.

The campaign posters – which can be viewed online – are comparable to many other health promotion campaigns for gay men in the UK, with images of attractive men, a somewhat cryptic slogan, a simple health message and a link to a campaign website. The evaluators note that there was no explicit attempt to use any theoretical behaviour change techniques during the campaign development.

The materials were distributed more widely than for many other campaigns, including on the Glasgow subway, on buses, in GP surgeries, in public libraries, and in gyms. Materials were also distributed in the more ‘traditional’ settings of gay bars, clubs and saunas; online dating websites and apps; and Glasgow’s Pride festival.

The evaluation

With the resources available, the researchers were able to conduct a post-intervention survey in seven Glasgow gay bars in July 2010. While this method can quite straightforwardly tell us how many men recalled seeing campaign materials, the researchers also attempted to discern whether there was any association between remembering the campaign and health behaviours. But they warn from the outset that an association cannot prove that the campaign caused people to change their behaviour.

A total of 784 HIV-negative men who have sex with men were included (men with diagnosed HIV were excluded). Most respondents were in their twenties or thirties.

Sixty per cent could remember seeing the campaign, although most men needed to be shown some campaign visuals in order to distinguish this intervention from other safer sex materials they had seen. (It is typical for published evaluations to report that between 50 and 80% of the target audience can recall seeing a mass media HIV prevention campaign.)

Men living in Glasgow and men using the gay scene regularly were more likely to have seen the campaign. Of all respondents, 302 recalled seeing campaign images on the gay scene and 234 in a more mainstream setting, but only 68 participants had picked up a campaign leaflet, 27 had talked to an outreach worker about the campaign, and 45 had visited its website. (A separate report found that over a five-month period, 9557 people visited the website, although only one in four were from the targeted region of Scotland.)

The researchers wished to see if men who could remember the campaign had behaviours that differed from other men.

Moreover, they differentiated men with ‘high’ campaign exposure (e.g. saw the campaign several times, could accurately recall its messages) from men with ‘medium’ exposure, men with ‘low’ exposure (e.g. could only vaguely remember it), and men with no exposure at all. It could be hoped that increasing levels of exposure to the campaign would be associated with increasingly healthy behaviour.

The analyses included statistical adjustment for known confounding factors (age, use of the gay scene, place of residence), and looked at the following behavioural outcomes.

• Number of unprotected anal intercourse partners in the past year – no difference according to degree of campaign exposure.
• A more complex measure of sexual risk, taking into account the perceived HIV status of partners and avoidance of anal sex – no difference according to degree of campaign exposure.
• Correct use of lubricant during anal sex – men with ‘no’ exposure to the campaign were less likely to correctly use lube than those with ‘low’ exposure. But men with ‘medium’ or ‘high’ exposure were no more likely to use lube correctly.
• HIV testing in the last six months – men with ‘medium’ or ‘high’ exposure to the campaign were more likely than other men to have tested for HIV recently.
• Intention to test for HIV in the next six months – first analyses suggested an association. But after adjusting for whether men had previously tested for HIV, there was no difference according to degree of campaign exposure.
• STI testing in the last six months – no difference according to degree of campaign exposure.

The challenges of demonstrating effectiveness

“Whilst the results are intriguing they are also frustrating,” write Paul Flowers and colleagues, who conducted the evaluation. Their survey found some associations between exposure to a mass media campaign and some sexual health behaviours, but the cross-sectional nature of the survey cannot demonstrate causality, the results are inconsistent and do not suggest a dose-response relationship.

The researchers note the methodological challenges of evaluating the effectiveness of mass media campaigns, despite the centrality of campaigns to HIV prevention practice. Whereas randomised controlled trials are normally conducted with clearly delineated groups of people, mass media campaigns aim to maximise spread and saturation within a population. Evaluations are often subject to compromise from the outset, given problems with funding and pressure to deliver campaigns quickly, leaving researchers unable to collect data before a campaign is delivered, or in another part of the country where the campaign was not delivered.

With weaker methodologies, it is hard to disentangle the effect of the campaign from other explanations. For example, rather than the campaign influencing men to test for HIV, it may be that men who test regularly are more likely to notice campaigns on the topic. An unidentified and unmeasured factor (such as perception of risk or ‘social capital’) may influence both the likelihood that a person sees the campaign and takes a test.

Moreover, as an increasing number of behavioural outcomes are examined, there is a greater risk that some observed differences have occurred by chance and would not be found if the study were repeated. While it is tempting to compare the results of surveys conducted in different years, with limited consistency in the people taking part, there is the potential for differences in the samples to skew results.

“The current economic climate could result in a lack of funding to support gold standard evaluations (i.e. RCTs) of mass media interventions promoting health behaviour change,” despite pressure for interventions to be based on reliable evidence, the authors say. It is therefore likely that increasing numbers of researchers will be working with similar constraints, using “pragmatic” but “inherently weak” evaluative designs.

Instead, they call for a greater examination of theories of behaviour change, the manner in which interventions are delivered and the processes through which mass media are consumed.