The European Journal of Public Health Advance Access originally published online on May 12, 2006
The European Journal of Public Health 2006 16(4):343-344; doi:10.1093/eurpub/ckl069
Viewpoints |
Learning from Policy Failure and Failing to Learn from Policy
Karien Stronks, Onyebuchi A. Arah and Thomas PlochgCorrespondence: Karien Stronks, PhD, Department of Social Medicine, Academic Medical Center, University of Amsterdam, PO Box 22700, 1100 DE Amsterdam, The Netherlands, tel: +31 20 5664892, e-mail: k.stronks{at}amc.uva.nl
Evaluation studies provide a key source of learning from policy success and failure. Policy interventions and their evaluation are, however, drenched in inescapable complexity. This makes it more difficult to evaluate this kind of intervention in the highly regarded randomized experimental design. Based on the experiences of the Health Action Zones (HAZs), Judge and Bauld outline key elements of a more realistic evaluation framework, which might contribute to a further understanding of complex policy initiatives in the field of public health.1 Their recommendations provide a good basis for the further development of the methodology of evaluation studies. Three additional issues should be mentioned, however. First, what is evidence in health policy? Second, we want to emphasize the importance of the evaluator having an open mind during the evaluation process. Third, we believe that the ultimate goal for us as evaluators is to influence health policy, in addition to understanding a policy intervention.
 |
Different types of evidence |
|---|
 Top Different types of evidence Theory guided but still...How can evaluators make...More art than science?References |
|---|
In line with Judge and Bauld, we must consider the question ‘what is evidence in health policy?’ Their paper can be read as a plea for the use of another type of evidence than that obtained in experimental studies. Implicitly, however, they seem to restrict themselves to evaluation studies that are based on new data collection. But must we always collect fresh data given the associated costs and burden? There is room, we believe, for the intelligent use of routine data such as linked administrative or monitoring data.2 Moreover, we believe that, in addition to evaluation studies, other types of studies might also contribute to the evidence base for effective health policy. These include the re-analysis of existing epidemiological datasets with the aim of assessing the cause—effect relationships between determinants and health. This knowledge is necessary for evaluating the potential impact of a specific intervention.3
|
Theory guided but still open minded |
|---|
|
TopDifferent types of evidenceTheory guided but still...How can evaluators make...More art than science?References |
|---|
We agree with Judge and Bauld that a conceptual framework of what a policy intervention aims to achieve is extremely useful as a starting point for both the intervention and its evaluation. On the other hand, the evaluator should continue to approach this with an open mind in order to anticipate results other than those originally aimed for. In this respect, it is debatable whether the HAZs should be considered a policy failure. Ultimately, before we can label policies as failures, we must learn to appreciate the fact that policy interventions in themselves are political tools which are both means and ends in a society.4,5 HAZs could be seen as a means used by the New Labour government to place public health high up on its political agenda. This might have led to increased public health awareness among policy makers and communities and, therefore, their readiness to pursue effective policies given favorable sociopolitical structures. HAZs could also been seen as an end characterized by increased government—community partnerships, where the community is a cornerstone of the ‘third way’ governance in the UK.5 Therefore, HAZs might be an attempt to reinstall ‘the social’ in public and social policy for health by way of community partnership and public participation. This political reasoning should partly determine how evaluation end-points are defined and calls for a complementary evaluation of the implicit and explicit political processes involved in such policy interventions.
|
How can evaluators make a difference? |
|---|
|
TopDifferent types of evidenceTheory guided but still...How can evaluators make...More art than science?References |
|---|
Given the complexity of health policy, evaluators would probably be satisfied if they succeeded in understanding the policy and in helping the implementation of the intervention policy from the outset. However, we believe that the aspirations of an evaluator should be on a higher level. The ultimate aim, in our view, should be to make a difference in terms of influencing health policy. From this viewpoint, the following few additional recommendations come up.
Be honest about the feasibility of a policy: Theoretically, a reflection on the value of a proposed policy or intervention might start before the intervention takes place. Such a reflection can be fed by an a priori impact assessment of (potential) policy interventions, using interdisciplinary knowledge, theory and thorough-going techniques.6 This is not an easy job, given, for example, the inherent long lag times for population effects of interventions. In addition, it requires courage on the part of the evaluator to tell policy makers that certain policies might not make a difference. If evaluators want to contribute to evidence-based policy, we feel it is their duty to be realistic about the anticipated effects and value of proposed interventions, and not to be overly optimistic.
Operate in close interaction with policy makers: If the evaluator is to influence health policy, a close interaction between the evaluator and the policy makers is necessary. This increases the chances that the evaluation study meets the expectations of the policy maker and will be used as an input for policy.7 The metaphor of a zip might be used to indicate the added value of such a linkage. The two rows of teeth, representing policy and research are independent parts, and both have their own expertise and agenda. They do have, however, a common starting point. In addition, they also have a common end point but only if they succeed in being pulled together during the whole process.
|
More art than science? |
|---|
|
TopDifferent types of evidenceTheory guided but still...How can evaluators make...More art than science?References |
|---|
We agree with Judge and Bauld that complexity is inescapable in policy evaluation and that there are no easy answers when facing this complexity. It is useful to search for meaningful and measurable proxies in the complexity. Complexity reduction will be inevitable and we must continue to search for ways to achieve reductions that are meaningful and realistic without being overly simplistic. Policy evaluation may therefore turn out to be more of an art than a science, requiring skilled, experienced, open-minded and courageous evaluators.
|
References |
|---|
|
TopDifferent types of evidenceTheory guided but still...How can evaluators make...More art than science?References |
|---|
1 Judge K, Bauld L. Learning from policy failure? Health Action Zones in England. Eur J Pub Health 2006; Advance Access published May 5, 2006, doi:10.1093/eurpub/ckl068
2 Sibthorpe B, Dixon J. Rethinking evaluation for policy action on the social origins of health and well-being. In: Eckersley R, Dixon J, Douglas B, editors, The Social Origins of Health and Well-Being Cambridge: Cambridge University Press, 2001.
3 Kindig D, Day P, Fox DM, et al. What new knowledge would help policymakers better balance investments for optimal health outcomes? Health Serv Res 2003;38:1923–37.[CrossRef][ISI][Medline]
4 Stone D. Policy Paradox. The Art of Political Decision Making. Revised edition. New York: W.W. Norton & Company.
5 Newman J. Modernising Governance. London: Sage, 2001.
6 Dunn WN. Public Policy Analysis: An Introduction. New Jersey: Pearson Education Inc., 2004.
7 Denis JL, Lomas J. Convergent evolution: the academic and policy roots of collaborative research. J Health Serv Res Policy 2003;8:1–6.[Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  H Thomson A dose of realism for healthy urban policy: lessons from area-based initiatives in the UK J. Epidemiol. Community Health, October 1, 2008; 62(10): 932 - 936. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||