Viewpoints |
‘More research needed’: plugging gaps in the evidence base on health inequalities
Mark PetticrewMRC Social and Public Health Sciences Unit, University of Glasgow, G12 8RZ
Correspondence: email: Mark{at}msoc.mrc.gla.ac.uk
 |
Introduction |
|---|
 Top Introduction What sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
Policymakers and public health researchers have demanded better evidence of the effects of interventions on public health and health inequalities and have pointed to the relative absence of rigorous outcome evaluations in this area. While most of these commentators agree that health inequalities are a problem, the means of tackling them often seems to rest on poor evidential foundations. This is no longer surprising, as the problem has been well-documented and the solution probably seems clear: if there is not enough evidence, then obviously we need more; and if the evidence is ‘weak’, well, then, it just needs to be ‘stronger’ in future. The funding environment in the UK, at least, is supportive of the production of this new, stronger evidence, with several new funding initiatives which are aiming to foster the production of new evaluations of public health interventions. However, it might be useful to consider further what ‘more evidence’ might look like (that is, what sort of evidence is needed) and how might it be ‘stronger’ than in the past and also to consider what additional challenges collecting this new evidence might bring.
|
What sort of evidence is needed? |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
It is clear that evidence on the effectiveness and cost-effectiveness of public health interventions is often missing. Sometimes this is because policies are insufficiently subjected to outcome evaluation, perhaps because it is assumed that they are mostly beneficial and any positive outcomes can be taken as read. Randomized controlled trials of policies are particularly uncommon and there are a variety of real and perceived barriers to trials. This means that the public are frequently ‘enrolled’ in real-life policy ‘experiments’ without giving their explicit consent,or indeed without any real prospect of anyone learning anything substantial about the effects of those interventions. Although the experimental nature of most policies is generally unacknowledged, this is not always the case; in a few cases in the UK in the 1990s it was acknowledged explicitly that government policies really were a form of ‘experiment’, though not in the epidemiological sense of the word. For example, the former Health Secretary Kenneth Clark famously called the NHS ‘a great experiment’, and he also referred to his health service reforms in the early 1990s in these same terms. His fellow minister, Nigel Lawson, also described his macroeconomic policies as ‘The British Experiment.’ However policy interventions like these are only ‘experiments’ in the weakest sense of the word – in the words of a recent report, ‘experiment’ often just means ‘Trying It Out’ 1. ‘True’ experiments – that is, RCTs, remain rare in most countries, though they are more common in the US.2
There is no lack of non-experimental evidence, however. There is, for example, a wealth of aetiological evidence,and we also have several influential empirically based theories about the societal, social, psychosocial and other causes of inequalities. However, it often appears to be difficult to translate this information into new interventions and even when the interventions are implemented, their evaluation is often problematic. Some of the major social determinants of health and health inequalities are difficult to randomize for practical or political reasons; one's social class for example, is not amenable to experimental manipulation and some forms of major structural intervention may be similarly problematic. The relative paucity of experimental evidence in the social sectors compared to healthcare can be seen by comparing the number of RCTs and possible RCTs in the Campbell and Cochrane databases: about 11 000 trials appear in the Campbell database, compared with over 400 000 in CENTRAL – the Cochrane Central Register of Controlled Trials. This disparity is only partly accounted for by the difference in age and resources between the Campbell and Cochrane Collaborations.
Producing more and better, evidence will however take time and while new studies are being commissioned, it is likely that the evidence base about effects of interventions is likely to remain ‘weak’ for some time to come. This suggests that while waiting for the gaps to be filled, we need to make better use of some of the evidence we do have – that is, observational epidemiological research on the social determinants. Such studies can be employed in the cause of evaluation (monitoring of changes in outcomes over time), in directly identifying points where new interventions may be directed and in suggesting which new interventions may be appropriate. However, such studies are probably under-used for these purposes and are certainly under-synthesized.
Discussions of public health evidence often tend to be quite fragmented and we usually focus on intervention studies (and their absence) to the exclusion of the rest of the evidence base. We also tend to overlook the extent to which these different types of evidence inter-relate (or should inter-relate), for example, how observational studies may be carried out as a step towards the design of new RCTs or are sometimes conducted where RCTs are not feasible. Focusing on the lack of RCTs however means that researchers tend to err on the side of scientific conservatism, where the evidence is simply never strong enough. Policymakers however are probably less interested in the evidence we don’t have, than in which direction the evidence is pointing (with suitable caveats). To quote one policymaker: "Rather than admitting to politicians that the studies are ‘poor,’ [researchers] have to say that we have accumulated clear and consistent evidence within limits that points to certain definite impacts".3
|
Ranking evidence |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
One reason why we focus on the gaps in the evidence base is that we tend to separate it into mutually-exclusive categories, with an implicit ranking – sifting it into ‘wheat’ and ‘chaff’ in one widely-used metaphor. Sorting evidence in this fashion is useful when there is a lot of evidence to sift through; given the choice between a trial and an uncontrolled study of the effects of an intervention, we would probably prefer the trial and in turn we would prefer the uncontrolled study to the unsupported anecdote. However – to stretch the winnowing metaphor further – when wheat is in scarce supply, no amount of sifting makes it better or any more abundant. Similarly, given a weak evidence base, ranking evidence does not tell us how to integrate different sorts of evidence, nor does it recognize the fact that different types of evidence are generally collected for different purposes. In public health we need to be more concerned with integrating evidence and using the evidence we already have, to identify plausible new evaluative studies, than with ‘ranking’ evidence.
|
Problems with outcomes |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
The problems of the patchy evidence base are often discussed, but there are other more basic problems which generally receive less of an airing. One such problem relates to the choice of appropriate outcomes to measure. In public health we are sometimes guilty of a sort of naïve ‘healthism’ in talking about our need for ‘better evidence’. For most of the main social determinants of health and health inequalities, ‘health’ is not the primary intended outcome and so, by default, evaluations which measure change in health status are uncommon. To take one example: housing is widely assumed to be a major determinant of health and health inequalities. However, the primary intended outcome of investment in new housing is not ‘health’. It is the provision of new, warm, dry, safe houses. The evidence that new houses were actually provided is easily obtained and a ‘good enough’ answer to the question requires evidence from pretty near the bottom of the hierarchy of evidence (i.e., the evidence of ones’ eyes). At most, it requires ascertaining that the houses were built and asking tenants if they are more satisfied and happier with their houses. This sort of evidence is indeed already collected routinely by most housing providers, whereas evidence on less obvious outcomes – the health impacts of their activities is generally not collected. The same issue applies to many other social interventions. It is not surprising that there is little robust evidence on the health impacts of social policies, because health improvement is often an unintended, spillover effect. Identifying meaningful primary and secondary outcomes and powering studies to detect them in evaluations of social interventions can be difficult and risks taking a narrow view of the intended purposes of the intervention in question.
So, while it is widely accepted that the main determinants of inequalities are social determinants, ‘health’ and ‘health inequalities’ may understandably not be at the top of most pressured policymakers list of priorities and developing the evidence base means making public health and health inequalities part of their business and this remains a major challenge. Health Impact Assessment (HIA) is intended to help with this task by ensuring that all major policies are assessed with respect to their impact on health and inequalities. However, the Catch-22 is that HIA depends on the provision of evidence about those effects – and we already know that this evidence is often absent.
|
Ecologies of evidence? |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
McLaren and Hawe suggest that ‘ecological thinking’ may be helpful in public health. Such an ecological perspective takes into account the individual and environmental determinants of health and other behaviours.4 Perhaps such ecological thinking can also be applied to public health evidence; it may be useful, for example, to consider the wider ‘ecology’ within which public health evidence exists and in which different sorts of evidence occupy different niches (i.e., where different study designs have different degrees of ‘fitness’ for different purposes). To pursue the analogy, in a food chain certain animals interact with, feed on and share space with other organisms. In a linear food chain polar bears eat seals, which eat fish, which eat shrimp, which in turn eat plankton. More complex organic relationships are described as ‘food webs’. A similar ‘ecology of evidence’ would highlight the relationships between different types of evidence and what they each contribute over time to the evidence base: for example, observational studies are used to inform the development of RCTs, which may themselves involve parallel qualitative studies, all of which are eventually subsumed within new systematic reviews, which inform the development of new observational and evaluative studies ...
This does not deny the strength of experimental designs and the fact that they provide strongest evidence of effectiveness, but recognizes that they may not always be possible and that we need to use what evidence is available. In some cases, Wanless and others (including some health inequalities researchers) have suggested that the gaps in the evidence base may be partially filled by exploiting the opportunities offered by ‘natural experiments’, such as changes in employment opportunities, housing provision or cigarette pricing.5 Natural experiments in the case of health inequalities can help investigate the determinants of those inequalities as well as assisting in the identification of potentially effective interventions. They can also indicate the scale and nature of impacts in situations where randomized controlled trials are not available. However, they bring their own challenges, chief among them being potential problems with internal validity, which makes drawing strong, unbiased causal inferences from such difficult studies and this has led to the occasional suggestion that such non-randomized studies should carry a ‘health warning’. This does not mean however, that they cannot provide valuable and valid information.
|
Conclusion |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
Implicit in debates about public health evidence is the assumption that the evidence base is something which needs to be assembled, as quickly as possible, in order to tell us definitively what to do. Of course, it is never that easy. The amount of evidence we ‘need’ cannot easily be ascertained. There will never be ‘enough’ evidence in absolute terms and the evidence base will probably always be criticized as being ‘weak’. Deciding how much evidence is ‘enough’ may just be another way of deciding how much uncertainty we are prepared to live with.5
A gap – in an evidence base or anywhere else – seems a simple enough problem to deal with – you just need to close it. However, not all gaps are easily closed and part of the challenge is to correctly identify those gaps and then use that information to identify where more evaluations of the effects of public policies on human health and wellbeing are most needed. The results of those studies may take time to emerge. In the interim it therefore seems particularly important to learn to make better use of whatever evidence is already available and whatever its limitations.
|
Acknowledgements |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
I would like to thank Matt Egan and Hilary Thomson for their advice. The idea from the article came from discussions in Calgary in 2006 at the International Collaboration on Complex Interventions funded by the Canadian Institutes of Health Research.
|
References |
|---|
|
TopIntroductionWhat sort of evidence...Ranking evidenceProblems with outcomesEcologies of evidence?ConclusionAcknowledgementsReferences |
|---|
1 "Trying it out": the role of pilots in policymaking. (2003) December. UK Government Chief Social Researcher's Office. Report of a review of Government Pilots.
2 Bozeman B, Scott P. Laboratory experiments in public policy and management. J Publ Adm Res Theor (1992) 3:293–313.
3 Petticrew M, Whitehead M, Graham H, Macintyre S, Egan M. Evidence for public health policy on inequalities: 1: The reality according to policymakers. J Epidemiol Commun H. (2004) 58:811–16.
4 McLaren L, Hawe P. Ecological perspectives in health research. J Epidemiol Commun H. (2005) 59:6–14.
5 Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute for Clinical Excellence. Lancet (2002) 360:711––15.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  E. Waters, M. Petticrew, N. Priest, A. Weightman, A. Harden, and J. Doyle Evidence synthesis, upstream determinants and health inequalities: the role of a proposed new Cochrane Public Health Review Group Eur J Public Health, June 1, 2008; 18(3): 221 - 223. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||