The European Journal of Public Health Advance Access originally published online on May 12, 2006
The European Journal of Public Health 2006 16(4):341-343; doi:10.1093/eurpub/ckl068
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Viewpoints |
Learning from Policy Failure? Health Action Zones in England
Ken Judge1 and Linda Bauld2
1 Professor, Department of Public Health & Health Policy, University of Glasgow, Scotland
2 Senior Lecturer, Department of Urban Studies, University of Glasgow, Scotland
Correspondence: Professor Ken Judge, Department of Public Health & Health Policy, University of Glasgow, 1, Lilybank Gardens, Glasgow, G12 8RZ, e-mail: k.judge{at}clinmed.gla.ac.uk
It is widely recognized that the evidence base for reducing health inequalities is relatively weak. Although many countries have invested in major policy interventions that might be expected to have an impact on the distribution of health outcomes, policymakers and researchers have not always found effective ways of learning from major social initiatives. In an ideal world, interventions would be designed and implemented in a way that allowed robust scientific findings to be produced. But while attempts to move in that direction should continue, UK experiences also suggest that greater efforts have to be made to learn from new policies and initiatives that are often conceived and delivered with evaluation well down their list of objectives.
When New Labour was elected in 1997 it promoted the belief that comprehensive and purposive public action would make a real difference to the growing social injustices that were evident across all aspects of British life. New Labour quickly introduced a wide range of initiatives to tackle social exclusion and poverty. But many of them failed to live up to expectations. Nevertheless, there is important learning to be gleaned from early attempts by New Labour to deliver on its election promises.
 |
Health action zones |
|---|
 Top Health action zones Learning from experienceHow can we do...ConclusionReferences |
|---|
One strand of New Labour's initial strategy involved a focus on area-based initiatives to reduce the effects of persistent disadvantages in neighborhoods blighted by generations of poverty and neglect. Health Action Zones (HAZs)1 were the first example of this type of intervention, and their focus on community-based activities to tackle health inequalities excited great interest both nationally and internationally.
HAZs were multi-agency partnerships located in 26 areas of England. The first wave of HAZs was launched in 1998 (15 areas) followed by a second wave (11 areas) in 1999. They varied significantly in terms of their population size and organizational configuration, ranging from large conurbations such as Merseyside and Tyne and Wear to largely rural areas such as Cornwall and North Cumbria. They were provided with fairly modest resources (
£4–5 million per year per zone at 2004 prices) but expected to develop local programs and activities to improve health and reduce inequalities during a 7-year lifespan. Many of the HAZs set themselves very ambitious goals in terms of reducing health inequalities or improving population health. For example, one of the largest aimed to transform the health prospects of its conurbation such that within <10 years its overall life expectancy would be amongst the highest in Europe.
The three broad strategic objectives of HAZs were (i) to identify and address the public health needs of the local area; (ii) to increase the effectiveness, efficiency and responsiveness of services; and (iii) to develop partnerships for improving people's health and relevant services. In response to these central government expectations, HAZs chose to invest in a wide range of projects and programs. The emphasis changed over time, but a flavor of the range of activities is given by an analysis of the 214 ‘programs’ that were initially identified across both the 1st and 2nd wave HAZs. The largest proportion of initial programs addressed the determinants of health by promoting healthy lifestyles, improving employment, housing, education and tackling substance abuse. Another important set of activities focused on the health of particular population groups and/or specific health problems. But there was hardly any aspect of population health improvement or community regeneration that at least one of the HAZs was not concerned with in one way or another.
HAZs were born at a time when anything seemed possible for a New Labour Government desperate to make things work and work quickly. But the tide of enthusiasm for change outran the capacity to deliver it. Too many hugely ambitious, aspirational targets were promulgated. The pressure put on local agents to produce ‘early wins’ was debilitating. A sense of disillusionment began to set in relatively early in their lifespan, and HAZs soon lost their high profile as the policy agenda filled with an ever-expanding list of new initiatives to transform public services and promote social justice. By the beginning of 2003, much earlier than expected, they were to all intents and purposes wound up.
The national evaluation of HAZs1 focused on monitoring activity in all 26 zones as well as examining three specific themes within different samples of HAZs: (i) building capacity for collaboration both amongst statutory agencies and with the community; (ii) developing the capacity for whole systems change; and (iii) tackling health inequalities. One of the main findings was that, although HAZs made little impact in terms of measurable improvement in health outcomes during their short lifespan, they did make a valuable contribution to building partnerships and raising awareness regarding inequalities in health. The study found that the sheer complexity of the initiative and the extent of policy change that HAZs experienced meant that drawing simple conclusions about impact was difficult. Not surprisingly, therefore, both the national1 and most of the local health2 evaluations focused on studies of the process.
Part of the problem for HAZs was that they were encouraged to set themselves impossibly ambitious goals to transform the health of their communities. Not surprisingly, they did not succeed. Although modest progress was made with individual programs and projects, there is no escaping the fact that HAZs did not—probably could not—do what they set out to achieve. But that does not mean that there is nothing of value to learn from their experience.
|
Learning from experience |
|---|
|
TopHealth action zonesLearning from experienceHow can we do...ConclusionReferences |
|---|
A major review of population health improvement commissioned by the Treasury in the UK concluded that although ‘there is often evidence on the scientific justification for action and for some specific interventions, there is generally little evidence about the cost-effectiveness of public health and preventative policies or their practical implementation’ [p. 5 in Ref. (3)]. One of the consequences is that policy initiatives are often not thought through with sufficient care, and researchers attempting to evaluate them often spend a great deal of time and effort supporting their development and implementation.4
The HAZ experience supports this view. It clearly demonstrates that there is a need to think more carefully about the focus of such initiatives, their objectives, their timescales, the support that they need both locally and nationally and the space, trust and time that is required to make any kind of sustainable change possible.
The notion that an injection of relatively modest resources accompanied by guidance—more evangelical than practical—from the central government might result in the speedy resolution of major social problems, that had proved largely intractable for generations, would not find many advocates today even among New Labour enthusiasts. But HAZs were put under pressure to demonstrate that they were ‘making a difference’ within a relatively short time period even though ‘early hits are not always evidence of accurate shooting’ [p. 95 in Ref. (5)].
The overwhelming problem—evident in much contemporary policy research—is that the universally voracious appetite for intelligence encourages the production of simple descriptions of activity without adequate discussion of the strengths and weaknesses of what is being presented. While undertaking the evaluation of HAZs we had serious concerns about the pressure to generate and use learning at too early a stage in the cycle of data collection, analysis and reflection. Simply documenting activity, which is frequently demanded and regularly served up, is not evidence of good practice and the growing tendency to pretend that it does yields little more than propaganda. On the other hand, it is plain that different customers for policy research need different products. More dialogue with key stakeholders about what a research team can produce, as well as what is not possible, and explicit agreements about what are the key priorities, are important and virtually continuous parts of the research process. Too many users of policy research still expect clear answers about impact when a more realistic product of evaluations is that they contribute to a process of enlightenment about highly complex processes that are interpreted by different actors in multiple ways.
|
How can we do better in future? |
|---|
|
TopHealth action zonesLearning from experienceHow can we do...ConclusionReferences |
|---|
In many important ways HAZs were victims of what Pawson describes as `the utter and appalling intricacy of social interventions' [p. 472 in Ref. (6)]. The more important and challenging the problem to be faced and the more ambitious the intervention that is proposed to deal with it the greater the degree of complexity that has to be faced by both implementers and evaluators. The precise form of that complexity differs from one initiative to another but there are a number of common strands, which include: lengthy implementation chains; negotiation—often protracted—about the way in which the resources for the intervention should be used; the adaptation of theories of change from similar initiatives; and, embracing the lessons of previous attempts at reform. ‘And evaluators are always left with the same question—complexity is inescapable, what can be done in the face of it?’ [p. 486 in Ref. (6)].
There is no single or easy answer. As experience in undertaking evaluations of complex initiatives grows and is shared, a growing number of practical suggestions are emerging. We follow the logic of Pawson's ‘tips on getting to grips with intricacy’ [pp. 486–8 in Ref. (6)] as a basis for outlining our own recommendations about key elements of an evaluation framework for complex policy initiatives.
Paint a picture
Many policy initiatives lack clarity about the desired outcomes and/or change mechanisms that they purport to promote. In these circumstances, logic models, theories of change and conceptual maps of what an initiative is trying to do can provide an invaluable starting point. The process of encouraging stakeholders to articulate their thoughts and assumptions about key parts of the postulated change process, even when significant gaps remain, is essential. But this should not be done to exhaustion. Consensus though desirable is not essential and too much emphasis on seeking a single theory of change may be more harmful than helpful. The key is to produce a recognizable narrative picture that is meaningful and recognizable to the key actors involved. This process is not an easy one, but skilled and experienced researchers can make a very useful contribution.7
Be selective
The principal purpose of the map for the evaluator is to facilitate decisions about where the major research effort should be concentrated. It is unlikely that it will ever be possible to investigate every interesting aspect of complex initiatives. Choices have to be made about where and how to deploy scarce research resources. This selection has to be negotiated with research commissioners and local implementers but the potentially successful evaluator must persuade everyone concerned that only so much can be examined properly and that the emphasis must be on the critical change mechanisms. In making the selection of which mechanisms to choose for close scrutiny it is useful to remember the value to be obtained by combining horizontal and vertical perspectives about different aspects of an intervention. But whatever choices are made about research priorities, researchers should retain a focus on what works, for whom, and in what circumstances.
Look beyond the initiative
Really good policy research has to be dynamic and well connected with developments in other areas. Even though an initiative might look like a significant innovation and its ambition and scope appear new and exciting it will have many precursors. Many of its building blocks will have been evaluated in some form before. Policy research is not a good setting for reinventing the wheel. Systematic reviews of existing knowledge can help with the process of concentrating scarce evaluation effort so that it is used to maximum effect. Scholarship is as much an integral part of good quality policy research as in any other field of inquiry. The policy impact of any particular study will be greatly enhanced if findings can be linked to wider experience.
Maximize comparisons
Too many policy studies do not go beyond rich descriptions of processes and experiences. Valuable though these often are they are no substitute for analyses of impact, which require some basis for comparison. Even when multi-site initiatives are designed and launched in a great hurry there is enormous value in persuading commissioners that implementation agencies must agree to collect a certain amount of data in common. Ideally such data should provide information about what are thought to be the key outcomes and processes. One of the great weaknesses of the HAZ evaluation was the failure to achieve this.
Although it is often expensive, time consuming and difficult to collect, it is important that community-based initiatives should strive to establish a robust baseline with which to measure changes through time. But that is a necessary rather than sufficient condition for a high-quality evaluation. More qualitative indicators of success, ideally collected from a range of perspectives, are also needed. It is only through an assessment of both forms of evidence that researchers, practitioners and policy makers can learn more about the different elements of ‘what works’, and to use this knowledge to frame future policy and practice development.
Remember your limitations
Complex community-based initiatives are just that. They do not lend themselves to simple calculations of good or bad, right or wrong, successful or not. The job of the policy analyst is to illuminate the processes of change and experience that s/he observes. In doing so, s/he can contribute to a collective process of ‘enlightenment’ about complex change processes in modern social welfare systems. The policy researcher should, therefore, use every evaluation opportunity to throw light on some important aspect of the process either of her/his own choosing or that which can best be negotiated with the program sponsors and research commissioners. The ethical obligation of the policy researcher is to make a contribution—albeit a modest one—to the social process of understanding and promoting change. It is not to award or withhold a seal of approval to this or that ‘flavor of the moment’ initiative that might have provided the pretext for a new investigative opportunity.
|
Conclusion |
|---|
|
TopHealth action zonesLearning from experienceHow can we do...ConclusionReferences |
|---|
Almost a decade after the landslide election victory that brought New Labour into power at least some of the initial enthusiasm for a centralist approach to ‘big’ government has waned. But the commitment to promoting social justice remains. What has changed is that there appears to be a more sophisticated recognition of the complexity, pervasiveness and durability of the social problems that have to be confronted. At the same time there is greater awareness of the need to learn lessons from past attempts to promote social change. But a huge amount still needs to be done to ensure that learning is maximized from expensive efforts to promote social change such as a fairer distribution of population health improvement. In this respect, HAZs should provide a salutary rather than forgotten example of what not to do in the future.
It is often argued that the really significant contributions to strengthening the evidence base to tackle health inequalities will come from well-designed studies that set out to answer clearly defined questions. But this will not be possible unless the interventions are themselves designed with more rigor than is commonplace. How realistic is this? It seems more likely that the frequent urge to intervene in response to the latest crisis or problem to reach the political hit list will continue to be part of the public policy landscape in modern democracies. There is a real danger in these circumstances, especially when the imperative to intervene quickly is combined with unrealistic expectations about the speed with which research findings can be generated, that valuable learning opportunities will be missed. The challenge when evaluation opportunities arise in this way is to negotiate the best possible research approach that acknowledges inter alia that incontrovertible measures of impact are not the only useful products that can be generated. The value of throwing light on complex processes in reflective and scholarly ways should not be underestimated even if it falls short of what is ideally required.
|
Acknowledgments |
|---|
This paper is based on research funded by the Department of Health, London. The views expressed are those of the authors and not necessarily those of the department of health.
|
References |
|---|
|
TopHealth action zonesLearning from experienceHow can we do...ConclusionReferences |
|---|
1 Barnes M, Bauld L, Benzeval M, et al. Health Action Zones: Partnerships for Health Equity. London: Routledge, 2005.
2 Bauld L, Judge K, editors. Learning from Health Action Zones. Chichester: Aeneas, 2002.
3 Wanless D. Securing Good Health for the Whole Population. London: The Stationary Office, 2004.
4 Mays N, Wyke S, Malbon G, Goodwin N, editors. The Purchasing of Health Care from Primary Care Organisations. Buckingham: Open University Press, 2001.
5 Alcock P. Participation or pathology: contradictory tensions in area-based policy. Soc Pol Soc 2004;3:87–96.[CrossRef]
6 Pawson R. Nothing as practical as good theory. Evaluation 2003;9:471–90.[CrossRef]
7 Mackenzie M, Blamey A. The practice and the theory: lessons from the application of a theories of change approach. Evaluation 2005;11:151–68.[Abstract]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  H Thomson A dose of realism for healthy urban policy: lessons from area-based initiatives in the UK J Epidemiol Community Health, October 1, 2008; 62(10): 932 - 936. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||