The European Journal of Public Health Advance Access originally published online on December 2, 2006
The European Journal of Public Health 2007 17(4):375-380; doi:10.1093/eurpub/ckl256
Infectious Diseases |
Benchmarking national surveillance systems: a new tool for the comparison of communicable disease surveillance and control in Europe
Ralf Reintjes1,2, Martina Thelen1, Ralf Reiche3 and Ágnes Csohán4
1 Department Public Health, Faculty Life Sciences, Hamburg University of Applied Sciences Germany
2 Tampere School of Public Health, University of Tampere Finland
3 Scientific Institute of the German physicians (WIAD), Bonn Germany
4 Béla Johan National Centre for Epidemiology, Budapest Hungary
Correspondence: Prof. Dr. (med.) Ralf Reintjes, MSc (P.H.), MSc (Epi.), Department Public Health, Faculty Life Sciences, Hamburg University of Applied Sciences, Lohbrügger Kirchstrasse 65, D-21033 Hamburg, Germany, tel: +49 40 428 75 6106, fax: +49 40 428 75 6129, e-mail: ralf.reintjes{at}rzbd.haw-hamburg.de
Received July 28, 2006, accepted October 24, 2006
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionREFERENCES |
|---|
Background: Communicable diseases do not respect national boundaries and are important challenges to health internationally. Considerable variation exists in the structure and performance of surveillance systems for communicable disease prevention and control. European Union (EU) countries should share ideas to improve the quality of surveillance systems. The study aims to support the improvement and integration of surveillance systems of communicable diseases in Europe while using benchmarking for the comparison of national surveillance systems. Methods: Surveillance systems from England and Wales, Finland, France, Germany, Hungary, and The Netherlands were described and analysed. After comprehensive data collection and validation by several European public health (PH) experts, a descriptive data analysis was carried out. Benchmarking processes were performed with selected criteria (e.g. case definitions, early warning applications, and outbreak investigations). After the description of benchmarks, best practices were identified and described. Results: Benchmarking of national surveillance systems is applicable as a new tool for the comparison of communicable disease control in Europe. The countries included in the study have in general well-functioning communicable disease control and prevention systems. Nevertheless, there are different strengths and weaknesses in various countries. Practical examples from the various surveillance systems were demonstrated and recommendations were given to policy makers. Conclusion: A gold standard of surveillance systems in various European countries is very difficult to achieve because of heterogeneity (e.g. in disease burden, personal, and financial resources). However, to improve the quality of surveillance systems across Europe, it will be useful to benchmark the surveillance systems of all EU member states.
Keywords: benchmarking, communicable disease surveillance systems, European Union
Communicable diseases do not respect national boundaries. Outbreaks thus have the potential to involve more than one country.1,2 Based on these facts diseases like SARS, nv-CJD, and HIV/AIDS, as well as, re-emerging diseases like tuberculosis or diphtheria are important challenges to health internationally. It is obvious that the efficiency and effectiveness of national surveillance systems for communicable diseases and their control is the basic requirement to ensure safety at national and European level. The decreasing role of borders within the European Union (EU) will necessitate a more coordinated approach to the surveillance of communicable diseases. Free movement of goods and people creates a need for national surveillance institutes to communicate events to each other regularly, sometimes rapidly, and to use similar surveillance components and case definitions.3 Therefore, a number of European surveillance networks have been developed between the countries of the EU over recent years.4 Some of these networks had a strong influence on the harmonisation of the related disease specific surveillance systems in Europe. For example, for influenza networks are being evaluated according to a standard protocol.5,6 EuroHIV and EuroTB are other examples where EU standards have modified national surveillance systems. For example, EuroTB has adopted reporting standards, which have also had major impacts on the harmonisation of the TB data collection in Europe. In addition, the work of the European Centre for Disease Prevention and Control (ECDC), which started operating in 2005, is based on national and common European surveillance systems. However, the way surveillance of a number of diseases is undertaken in European countries varies greatly.7,8
In the context of the EU Project (HU02/IB/SO/02-TL),9 which aims to support Hungarian public health (PH) experts during the integration process of Hungary as new EU member state, the current study examines whether benchmarking of national surveillance systems for communicable disease control in Europe is applicable. Benchmarking is a systematic process, with which products, services and methods are judged and compared with those of competitors.10 Points of reference are created by such a procedure, in order to set performance-goals and to align activities of organisations to best practices.11 Benchmarking was developed in the field of industrial production12 and recently started to play an important role in research projects12 also, within the EU.13
The current benchmarking study compares national surveillance systems in six EU member states in order to identify strengths and weaknesses of these surveillance systems. The objective is to give recommendations to policy makers to improve the quality of national surveillance systems for communicable disease control.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionREFERENCES |
|---|
A benchmarking cycle generally includes planning, analysis, and implementation.15 Although benchmarking has significant methodical demands,16 a comprehensive procedure with seven key elements for this cycle was selected for this project (see table 1).17
Step 1: Selection of the process or function to be benchmarked
The basis for the first selection of benchmark processes were components of PH surveillance systems described by the Guidelines for Evaluating Public Health Surveillance Systems18 as well as by the Communicable Disease Control Handbook.1 Ten criteria for surveillance systems were identified in a consensus decision making process and transformed into benchmarks (see table 2). Results of the five most relevant criteria are presented here.
|
|
Step 2: Understanding the existing process or function
Numerous round-table discussions in the introduction period of the project were used to develop a uniform ‘language’ between the partners and establish common descriptions for the identified objects and criteria (see table 2).
Step 3: Identification of benchmark partners
The surveillance systems of England and Wales, Finland, France, Germany, Hungary, and The Netherlands were selected as benchmarking partners. The selection was made on the basis of best practices in different areas of communicable disease surveillance and control and because of existing contacts to experts at national public health institutes of these countries.
Step 4: Collection of data and information
Success or failure of the benchmarking essentially depends on the acceptance of the procedure and the database. It is very important to collect comparable data.19 Thus, a questionnaire regarding the necessity of benchmarking of national surveillance systems, possible indicators, evaluations of surveillance systems, strengths and weaknesses of components of surveillance systems, and available data sources for benchmarking was designed and used.
Step 5: Identification of gaps and reasons for them
The selected measurements quantify differences in execution and the purposeful questions identify the reasons of the differences. In order to analyse the data, the key features for each component were brought into a format, which allows a descriptive assessment of the contents and use of data.
Step 6: Development of programmes to implement findings
The proceedings, which take place during the conversion and/or the implementation of findings and recommendations, are not to be easily generalized because they depend for a large extent on the culture within the organisation. We looked for ‘best practices’ in the six countries under investigation, in order to give policymakers a short description of good performances in different areas of communicable disease surveillance and to develop a set of recommendations to improve the quality of the system.
Step 7: Implementation of changes and monitoring of results
The recommendations include suggestions for comparative evaluations of the surveillance systems of all EU member states.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionREFERENCES |
|---|
The benchmarking identified remarkable differences between the components under consideration and described good/best practices in the six national surveillance systems.
The number of ‘notifiable diseases’ in the six countries ranges from 26 (France) to 82 (Hungary). In each country, the extent of notifiable diseases depends, among other factors, on the historical background and the various disease burdens in the individual country. Notification of a certain disease is important to one country and might be less important to another.20 In England and Wales, financial incentives for notifiers are given to physicians for the notification per case. In five countries comparable ‘case definitions’ for all notifiable diseases are in use. In Finland, all cases are laboratory-confirmed except a part of tuberculosis cases. In Hungary, the implementation of EU case definitions is in preparation. France, Germany, and The Netherlands have a three-tiered classification within their case definitions similar to the recommended EU case definitions21 (see table 3).
|
‘Timeliness of reporting’ in Germany shows that reports reach the regional level within up to the third working day of the first week after the detection of the case. The national level is reached 1 week later. In Finland, laboratory notifications are sent directly to the national level with an average of 5 working days from sampling date. Physician notifications are sent through the regional level to the national level in
2 weeks. In England and Wales, the statutory notifications reach the national level within 7 days, laboratory reports reach the local level within 1–2 days, and the regional level within 7 days. In France the estimated time to inform national level varies between 1 and 5 days. Hungary has a timely reporting following disease onset (2–3 days to national level). In most countries timeliness was assessed using audits, and the question of reporting to local, regional, and national levels depends on the systems of governance. In all the six countries, paper ‘notification forms’ and/or phone calls are used by the notifier to report cases to the local PH system, while electronic case reporting systems are used from the local PH level upwards. In Hungary for example, the electronic software, EFRIR, is used to report cases from local to regional and national level. As soon as the data is entered in the PH intranet, it is accessible at the local, regional and national level.
In Hungary, ‘information generated from the data set’ is not automatically available, but can be analysed from the available data on request. In contrast, the five other countries regularly process their data to generate information. Automatic ‘early warning applications’ are presently not in use in Hungary. In Germany, automatic early warning is implemented in some states. An example is the bar chart called ‘Barometer’, which represents trends for notifiable diseases in North Rhine–Westphalia. In Finland, France, and The Netherlands, different early warning applications are in place and functioning (see table 4). Experience shows their value for PH practice. Nevertheless, formal evaluations are rarely published.
|
‘Outbreaks’ are investigated in all the selected countries. In Hungary the investigation is mainly concentrated on a descriptive analysis of the outbreak. Analytical studies are not a common practice. In the other five countries analytical outbreak investigations (e.g. case–control studies) are conducted for significant outbreaks (see table 5).
|
Regarding ‘data dissemination’, all selected national bulletins are accessible via the Internet. The frequency is either weekly or monthly. In Hungary, the National Centre of Epidemiology provides the printed weekly, Epinfo. This bulletin provides standard data tables, articles on communicable disease problems, relevant international incidents and outbreaks. In Germany, the weekly Epidemiologisches Bulletin is published at national level. The bulletin includes data, text, discussions, graphics, and description of effected population groups, regional distribution, and time series analysis. The data can also be accessed and used via an internet application. Other good examples are the data and information dissemination tools in Finland and The Netherlands. In Finland, a weekly updated internet version is available for the public which includes comments and epidemiological observations. The data format allows compilations of tables and trend analysis by the user. Actual figures and summary comments are also routinely reported in the monthly bulletin, the Kansanterveys-lehti. In The Netherlands, the Infectieziekten Bulletin is published at national level every 4 weeks. Up-to-date feedback on the situation is updated daily and accessible via the internet.
Generally, countries ‘evaluate’ their surveillance systems using different objectives, attributes and purposes. In 2002, the Hungarian national surveillance system was evaluated on behalf of the EU Commission. In Germany, the Robert Koch Institute evaluated the surveillance system after the new Infectious Disease Control Law was introduced in 2001.23 In The Netherlands, the electronic reporting system OSIRIS was evaluated for timeliness and completeness of infectious disease notification in 2003.24 Another example is the current evaluation of the surveillance system in France. It is a comprehensive study including field exchange of expert opinions with other EU member states (e.g. Germany and The Netherlands).
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionREFERENCES |
|---|
The results of the present study show that benchmarking of national surveillance systems is applicable as a new tool for the comparison of communicable disease control in Europe. The selected benchmarking partner countries for this study proved to be useful because of best practices in different areas of their surveillance systems for communicable diseases and the availability of data in these countries.
The study demonstrates that different case definitions and classifications still exist amongst EU member states. This hinders international comparisons of epidemiological data (e.g. burden of disease and disease trends). The introduction of comparable case definitions, according to the EU case definitions,21 the most fundamental criterion is in communicable disease control at the national and international levels and is therefore recommended for all EU member states. This will improve comparability of notification data with other member states and thus enhance the integration of national systems in European networks of communicable disease surveillance systems.
Completeness or sensitivity is an important issue regarding the quality of a surveillance system. Nevertheless, this is also very difficult to be compared. One widely accepted method to estimate the completeness is the so-called capture–recapture method. Even though it is used in various settings, currently only for a few diseases in a selected number of countries data are available. Based on the fact that the sensitivity of reporting for a given disease varies between different surveillance systems (even within one country) and that the sensitivity of reporting for various diseases within the same surveillance system varies greatly22 it is currently not realistic to compare national surveillance systems for this component. Being very well aware of these issues we decided not to use completeness or sensitivity as a benchmark.
Timeliness of reporting was one of the most difficult criteria to be investigated because of differences between and within national surveillance systems. In this study, average values were identified, which can only serve as rough indicators to demonstrate the heterogeneous timeliness of reporting. Nevertheless, the timeliness depends on the format of reporting. To be effective, reporting must be timely and accurate. Previous studies have demonstrated the benefits of electronic case reporting systems.24,25 For example, in The Netherlands, it was observed that timeliness and completeness was improved after the introduction of the internet based reporting system OSIRIS.25 In Hungary, timely and accurate data transfer from local to regional and national level is well established and based on an application of the central database EFRIR using an intranet. An introduction of early warning applications, including the use of algorithms, and time series analysis is suggested.
As timely surveillance data are the basis for early PH action, early warning applications should be used for timely visualisation of data. Good/best practices for timely data use and early warning applications are in use in several countries. For example, the Infectious Disease Surveillance Information System (ISIS) in The Netherlands is responsible for timely distribution of descriptive epidemiological reports and early warning signals to professionals in PH. It provides the infrastructure for the collection of incidence data for data analysis, interpretation and systematic reporting. ISIS allows the assessment of the present situation within a few seconds and is updated on a daily basis.26
The control of an outbreak depends on early detection, followed by a rapid structured investigation.1 The growth in international trade and travel has increased the likelihood that outbreaks will involve more than one country.2 For these reasons, effective outbreak investigation in each member state is an important need for national and European-wide communicable diseases safety. As it is the core tool to control, manage, and prevent outbreaks, spreading the knowledge, practice, and experience of analytical epidemiological outbreak investigations is crucial to uncover the source of infections and the route of transmission effectively. As the performance of analytical studies in outbreak investigation is rare in some countries, the setting up of training and regular performance of analytical epidemiology in outbreak investigations is needed. Training and practice of leading PH experts in analytical epidemiology for outbreak investigations is a necessary component for disease control, especially for new or unknown health problems. In cooperation with European networks (e.g. EPIET) and partners in other member states, establishment of training is recommended.
Although benchmarking has been shown to be suitable to answer the research question, some limitations have to be considered. The current study tried to address all relevant factors for the benchmarking process although the available resources were limited. Because of these limitations, the data analysis could only be short and descriptive. The study started with the identification of applicable and internationally accepted criteria of surveillance systems that can be suitable for benchmarking. During the data collection and analysis process, all selected benchmarking criteria were operationalised into a set of specific benchmarks. This modification process helped to get information that is comparable. During this process, the investigators had to rely on available information, publications, expert opinions, and professional experience. Since some data was heterogeneous and only of qualitative nature, not all benchmarks could be standardised and validated. In general, a benchmarking study should not aim to detect the mean. Benchmarking should search for best practices in order to adopt them at a larger scale. For the first study of this kind, this could only be achieved to a limited extent. Based on this several recommendations were given to Hungarian stakeholders via a benchmarking report in order to improve the national surveillance system in Hungary. Nevertheless, the current study provides several operationalised benchmarks that should be the basis for benchmarking processes of surveillance systems in the future. A gold standard for surveillance systems in various European countries is very difficult to achieve, because of their heterogeneity in many domains including disease burden, personal and financial resources. Often a West–East difference is discussed and questions, like ‘Are Western surveillance standards appropriate for all EU countries?’, are raised. Despite these heterogeneities in surveillance systems, all countries need to share ideas on how to increase the quality of national and common European surveillance systems. In the coming years the European Centre for Disease Prevention and Control (ECDC) will play a more important and central role in this process. Their work in supporting EU member states in improving national and consequently the European surveillance can build on lots of experiences from all over Europe. As this study shows, benchmarking national surveillance systems could be a useful tool for this process.
|
Acknowledgments |
|---|
We thank Horst-Gerhard Baumeister, György Berencsi, Dounia Bitar, Arnold Bosman, Àgnes Dencs, Miklós Fuzi, Susan Hahne, Douglas Harding, Jeremy Hawker, Katalin Krisztalovics, Peter Kreidl, Florence Lot, Márta Melles, Zsuzsanna Molná, Richard Pebody, Petri Ruutu, Zita Schillmöller, Johanna Takkinen, Ulrich van Treeck, Mary Ward, and Julius Weinberg for valuable information and support of this project. We also thank Nolene Sheppard for her help in improving the language of the paper. This project was funded by the EU (HU02/IB/SO/02-TL).
| Key points
|
|
REFERENCES |
|---|
|
TopAbstractMethodsResultsDiscussionREFERENCES |
|---|
1 Hawker JBN, Blair I, Reintjes R, Weinberg J. Communicable Disease Control Handbook (2005) 2nd edn. Blackwell Publishing.
2 MacLehose LBB, Camaroni I, Fulop N, et al. Communicable disease outbreaks involving more than one country: systems approach to evaluating the response. Br Med J (2001) 323:861–3.
3 Giesecke J. Surveillance of infectious diseases in the European Union. Lancet (1996) 348:1534.[CrossRef][Web of Science][Medline]
4 Sprenger MJW, Bootsma PA, Reintjes R. Infectious disease surveillance in Europe (in Dutch). Nederlands Tijdschrift voor Geneeskunde (1998) 31:2418–23.
5 Aguilera JF, Paget WJ, van der Velden J. Development of a protocol to evaluate the quality of clinical influenza data collected by sentinel practitioners in Europe. Eurosurveillance (2002) 7:158–60.[Medline]
6 Kyncl J, Paget WJ, Havlickova M, Kriz B. Harmonisation of the acute respiratory infection reporting system in the Czech Reublic with the European community networks. Eurosurveillance (2005) 10:5–6.
7 Desenclos JC, Bijkerk H, Huisman J. Variations in national infectious diseases surveillance in Europe. Lancet (1993) 341:1003–6.[CrossRef][Web of Science][Medline]
8 Fenton KA, Lowndes CM. The European Surveillance of Sexually Transmitted Infections (ESSTI) Network: Recent trends in the epidemiology of sexually transmitted infections in the European Union. Sex Transm Infect (2004) 80:255–63.
9 Reintjes R, Thelen M, Reiche R. Benchmarking national surveillance systems. In: Final Report of Twinning Project between Germany and Hungary. —HU02/IB/SO/02-TL. EU Commission 2005.
10 Camp RC. Benchmarking: The Search for Industry Best Practices That Lead to Superior Performance (1989) MilwaukeeWI: ASQC Quality Press.
11 Töpfer A. Benchmarking. In: Benchmarking. Der Weg zu Best Practice (1997) 31–75.
12 Zairi M, Leonard P. Practical Benchmarking: The Complete Guide (1994) London: Chapman & Hall.
13 Tintelnot C. Benchmarking in Forschung und Entwicklung, Benchmarking—der Weg zu unternehmerischen Spitzenleistungen (1997) 43–56.
14 O'Mahony M. EU productivity and competitiveness: an industry perspective; can Europe resume the catching-up process? Amt für Amtliche Veröff. der Europ. Gemeinschaft 2003.
15 American Productivity and Quality Center. Planning, Organizing and Managing Benchmarking Activities. A User's Guide (1993) Houston: TXAPQC.
16 van Sutum A. Methodische Probleme des Benchmarking: der Bertelsmann-Benchmarking-Index (2004) Münster: Centrum für Angewandte Wirtschaftsforschung.
17 Ransley DL. Do's and Don'ts of R&D Benchmarking. Res Tech Manage (1994) 37:50–6.
18 Centers for Disease Control and Prevention (CDC). Updated guidelines for evaluating public health surveillance systems: Recommendations from the Guidelines Working Group. MMWR (2001) 50:1–35. (RR-13).[Medline]
19 Bichler K. Benchmarking—Die richtigen Zahlen beschaffen. (1993).
20 Buehler JW, Rothman KJ, Greenland S. Modern Epidemiology. (1998) 2n edn. Philadelphia: Lippincott-Raven. 435–57.
21 Commission Decision 2002/253/EC laying down case definitions for reporting communicable diseases to the Community network under Decision No. (2002) 2119/98/EC of the European Parliament and of the Council. OJL86/44, 3.4.2002. Luxembourg: Official J Eur Comm, 2002.
22 Reintjes R, Termorshuizen F, van de Laar MJ. Assessing the sensitivity of STD surveillance in The Netherlands; an application of the capture-recapture method. Epidemiol Infect (1999) 122:97–102.[CrossRef][Medline]
23 Krause GAD, Claus H, Hellenbrand W, et al. First evaluation of the surveillance systems of notifiable diseases under the infectious disease control law in Germany. Gesundheitswesen (2003) 65:8–12.[Web of Science][Medline]
24 Ward M, Brandsema P, van Straten E, Bosman A. Electronic reporting improves timeliness and completeness of infectious disease notification, The Netherlands 2003. Eurosurveillance (2005) 10:27–30.[Medline]
25 Jajosky RA, Groseclose S. Evaluation of reporting timeliness of public health surveillance systems for infectious diseases. BMC Public Health (2004) 26:29.
26 Widdowson MA, Bosman A, van Straten E, et al. Automated, laboratory-based system using the internet for disease outbreak detection, The Netherlands. Emerg Infect Dis (2003) 9:1066–52.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  B. K. Bernitz Communicable disease policy development in response to changing European political frontiers in Finland, Norway and Sweden Scand J Public Health, November 1, 2008; 36(8): 875 - 878. [Abstract] [PDF]
|
|
|
|
 |
|
 M. Valimaki, P. Makkonen, M. Blek-Vehkaluoto, V. Mockiene, N. Istomina, U. Raid, M.-L. Vanska, and T. Suominen Willingness to Care for Patients With HIV/AIDS Nursing Ethics, September 1, 2008; 15(5): 586 - 600. [Abstract] [PDF]
|
|
|
|
 |
|
 A. Kroneman, L. Verhoef, J. Harris, H. Vennema, E. Duizer, Y. van Duynhoven, J. Gray, M. Iturriza, B. Bottiger, G. Falkenhorst, et al. Analysis of Integrated Virological and Epidemiological Reports of Norovirus Outbreaks Collected within the Foodborne Viruses in Europe Network from 1 July 2001 to 30 June 2006 J. Clin. Microbiol., September 1, 2008; 46(9): 2959 - 2965. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||