The European Journal of Public Health Advance Access originally published online on July 13, 2005
The European Journal of Public Health 2005 15(4):368-371; doi:10.1093/eurpub/cki013
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Health Services Research |
Implementing clinical protocols in oncology: quality gaps and the learning curve phenomenon
Simos Kedikoglou1, Konstantinos Syrigos2, Yannis Skalkidis3, Fani Ploiarchopoulou2, Nick Dessypris1 and Eleni Petridou1,4
1 Department of Hygiene and Epidemiology, Athens University Medical School, Athens, Greece
2 Third Department of Internal Medicine, Sotiria Hospital, Athens University Medical School, Athens, Greece
3 Second Department of Surgery, Aretaieion Hospital, Athens University Medical School, Athens, Greece
4 Department of Epidemiology, Harvard School of Public Health, Boston, MA, USA
Correspondence: Simos Kedikoglou, MD, MPH, Research Associate, Department of Hygiene and Epidemiology, Athens University Medical School, 75 Mikras Asias, Goudi, Athens 115-27, Greece, tel: +30 210 746 2201, fax: +30 210 746 2105, e-mail: sjk{at}otenet.gr
Received January 15, 2004, accepted May 5, 2004
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionReferences |
|---|
Background: The quality improvement effort in clinical practice has focused mostly on ‘performance quality’, i.e. on the development of comprehensive, evidence-based guidelines. This study aimed to assess the ‘conformance quality’, i.e. the extent to which guidelines once developed are correctly and consistently applied. It also aimed to assess the existence of quality gaps in the treatment of certain patient segments as defined by age or gender and to investigate methods to improve overall conformance quality. Methods: A retrospective audit of clinical practice in a well-defined oncology setting was undertaken and the results compared to those obtained from prospectively applying an internally developed clinical protocol in the same setting and using specific tools to increase conformance quality. Results: All indicators showed improvement after the implementation of the protocol that in many cases reached statistical significance, while in the entire cohort advanced age was associated (although not significantly) with sub-optimal delivery of care. A ‘learning curve’ phenomenon in the implementation of quality initiatives was detected, with all indicators improving substantially in the second part of the prospective study. Conclusions: Clinicians should pay separate attention to the implementation of chosen protocols and employ specific tools to increase conformance quality in patient care.
Keywords: clinical protocols, implementation, learning curve, lung cancer
The amount of published health care research data has increased exponentially. During the period 1975–1980 about 500 new studies were added annually to Medline; this figure is currently greater than 10 000, including more than 450 clinical guidelines.1 The multitude of scientific data has led to joint efforts aimed at developing a set of criteria for a more standardized process and assessment of the quality of clinical guidelines (e.g. the Scottish Cancer Therapy Network).2
The vast majority of these initiatives have focused exclusively on ‘performance quality’, that is, on building the most comprehensive evidence-based guidelines and clinical protocols3 under the assumption that once published, research will automatically be applied. In contrast, ‘conformance quality’, i.e. the extent to which these guidelines or protocols are consistently applied is often taken for granted and the issue has not gathered adequate attention. This is despite the fact that the few existing studies point to significant gaps in the extent and consistency of guideline application.4,5 The gap in this aspect of health care quality has led to a suggestion that evidence-based guidelines have to be complemented by evidence-based implementation.6 This aims to ensure that clinicians implement guidelines in a consistent way, often using supportive tools such as electronic protocols or computer-assisted clinical order entries for medications.
The inadequate focus on ‘conformance quality’ refutes the theoretical base of continuous improvement by breaking its feedback cycle, a necessary premise of the continuum. Indeed, the improvement process starts with a protocol that standardizes the clinical management process. The protocol's application should then decrease the variation in clinical practice and this reduction of variability in turn allows the identification of deficiencies previously hidden as ‘normal variation’ around the mean. The increased conformance means that these deficiencies may now become the target of further investigation7 and allow the retrospective improvement of the protocol itself in a theoretically never-ending cycle. Conversely, a decreased conformance quality (i.e. a lack of adherence to the protocol) fails to improve clinical practice and makes the documentation of further needs for improvements impossible or impracticable, breaking the continuum cycle.8 In view of the need to study and document conformance quality, we undertook a study to measure the existing variation in the use of resources for a specific diagnostic category and to investigate potential methods for enhancing conformance quality in clinical management.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
Our study was performed in an academic medical centre in Athens, Greece and focused on lung cancer management. The study involved two phases: in the first, retrospective, phase of the project, a clinical protocol for the staging and evaluation of patients with lung cancer disease was developed in collaboration with the clinical staff of the Oncology Unit, 3rd Department of Medicine at the Sotiria General Hospital in Athens, Greece. Lung cancer disease was chosen because of the lack of satisfactory treatment, the significant problems with conformance quality in existing guidelines9 and the magnitude of the epidemic.10 The specific department chosen featured stability in the clinical staff over time that allowed for easier intertemporal comparisons, and a strong interest in quality improvement evidenced by its participation in other similar efforts.
The protocol was developed specifically for the needs of the study and engulfed all the laboratory and imaging tests needed for the diagnosis, histological confirmation and staging of patients with a suspicious lesion in the lungs. In terms of imaging tests, the protocol included a thoracic CT, an abdominal CT and a brain CT as well as a bone scan, while the laboratory tests included whole blood count and liver function tests. Furthermore, all patients should have confirmation of the diagnosis by cytological examination of a specimen taken either by bronchoscopy or by fine needle aspiration of a peripheral lesion, with CT guidance.
Consequently, the medical records of 60 patients with suspected lung cancer who were hospitalized in the department during the 8 months prior to the implementation of the protocol were accessed and evaluated. To enhance consistency, every second patient admitted in this interval was included in the retrospective study, while the same resident (F.P.) extracted all data from their medical records and completed the protocol, recording documentation on the performance of the examinations included in the protocol.
In the second, prospective phase of the project, the clinical protocol was computerized using Microsoft Access 2000® and was installed in the electronic database of the department. The program allowed for real-time completion of the data and all clinicians were instructed to complete it during the patient's hospitalization in conjunction with the manual completion of the medical record. The purpose of the electronic protocol was to act as a continuous reminder to the clinical staff of the missing steps so as to increase adherence to the protocol and hence, conformance quality.
After the protocol was installed, all patients with suspected lung cancer admitted to the department had their record completed both manually and in electronic format by the resident responsible for their care. The enrolment continued until the prospective phase of the study had reached an equal number of 60 patients, a process that took 
4 months. All clinicians, residents and attending physicians participating in the protocol were blind to the outcomes of the first phase and there were no changes in the medical staff between the two phases of the study.
An effort was also made to detect the existence of a possible learning curve phenomenon, i.e. whether the implementation of the guideline improved as the clinicians of the department familiarized themselves with the process. For this purpose, the records of the last 30 patients prospectively enrolled were separately compared with the records from the retrospective phase of the study. Statistical comparisons between the two groups were performed using Fisher's exact test.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
Table 1 provides the distribution of the patients enrolled in the prospective and retrospective phase of the study by demographic, anthropometric and lifestyle data. There were 51 male and nine female patients, whose records were accessed in the retrospective first phase of the study and 47 male and 13 female patients, enrolled in the prospective phase of this study. Furthermore age, somatometric characteristics, as well as the smoking consumption of the two cohorts were not significantly different and hence the groups were deemed comparable for the purpose of this study.
|
Table 2 presents the summary of the findings for the prospective phase of the study. In terms of the imaging testing performed, all 60 (100%) patients in the retrospective phase had undergone thoracic CT, 56 patients (93.3%) had undergone abdominal CT and 48 patients (80.0%) had undergone a brain CT. It should be noted that in local practice thoracic CT does not include the adrenal glands. All patients had complete laboratory evaluation (whole blood count, liver function tests). It merits reference that, of the three patients who were admitted with central nervous system (CNS) symptoms documented in physical examination, only one (33%) had received a brain CT. Also, only 35 patients (58.37%) received a bone scan and among the nine patients with CNS symptoms for more than 12 months, only five (55%) received a bone scan.
|
Table 2 further provides the same data for the prospective phase of the study. Overall, an improvement was observed in four out of the eight components of care examined in this study. More specifically, in terms of the imaging testing performed, all 60 patients (100%) had a thoracic CT documented in their medical record. Fifty-nine patients (98%, P = 0.18) had an abdominal CT, while 56 patients (93%, P = 0.03) had a brain CT. All five patients admitted with CNS symptoms had a brain CT documented (100%, P = 0.11). Finally, 47 patients (78%, P = 0.001) received a bone scan, including all seven patients who had symptoms for more than 12 months (100%, P = 0.07). There were no differences in the laboratory examinations, as all patients enrolled in both phases had complete testing in all parameters measured. Lastly, the interesting results from the comparison of the medical records from the last 30 patients of the prospective phase of the study with the records of patients from the retrospective phase are illustrated. It is evident that the last 30 medical records of the prospective study were complete in all examined components in which improvement could be expected.
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
This research project is one of the few that have tried to measure conformance quality in the delivery of health care. It is evident from the results that there is considerable variation in the diagnostic procedures used to evaluate patients with lung cancer. The use of newer tools to improve quality aspects of care, such as electronic reminders in the medical records seems to improve conformance quality by decreasing variation. Our results are also compatible with a learning curve phenomenon since conformance quality seemed to improve as the study progressed and the clinical management of the last 30 patients seems to have totally conformed to the protocol.
One additional aspect of this study is the documentation of the learning curve phenomenon. Practitioners interested in the improvement of health care delivery should pay particular attention to this finding. In the phase of an increasingly cost-conscious environment every new improvement initiative has to prove its incremental benefit in vigorous testing. However, the design and powering of such testing should take into account the learning curve phenomenon. To illustrate this concept, in the retrospective arm of our study only 48 out of 60 patients (80%) received a brain CT. In the prospective arm of the study 56 out of 60 (93%) of patients received this testing and the result, although not ideal, reached statistical significance (P = 0.03). However, the four patients who did not receive this test in the prospective phase occurred in the first 30 patients of the study; therefore if the study were to end with only 30 patients enrolled, then the brain CT performance (26 out of 30 patients or 86%) would not have reached statistical significance (P = 0.4) and would have complicated the subsequent implementation of quality improvement, especially if the clinical environment were hostile to change.
This protocol was designed based on the widely accepted guidelines of diagnosing and staging malignant lung lesions.11 Lung cancer staging prior to treatment aims to determine the location of malignant lesions, the choice of optimal treatment schedule and the attainment of valid prognostic information. Histological confirmation and staging represent one inseparable process and thus tumour resectability assessment entails the risk of requesting unneeded tests, especially for patients that turn out to have small cell lung cancer. The protocol applied reflects what clinicians of this department considered at the time as the standard approach to all patients admitted with the suspicion of malignant lung cancer lesion.
An alternative choice to using an internally derived protocol would be the application of an externally developed one. However, this would have made it impossible to measure the unassisted degree of conformity in the retrospective phase of the study, since the external protocol wouldn't necessarily correspond to the practice patterns that were followed by these specific clinicians and in the clinical care under scrutiny. In addition, the relatively much longer interval required for an external protocol to reach consensus and measure performance across different clinical settings may have introduced biases in the form of changes in the medical staff and, potentially, in practice pattern within the department.
The use of an internally developed protocol rather than an external one should also be considered in view of the following facts: first, there is variability in current external protocols, such as, for example, noted differences between US and European literature in the need to detect distant metastases. Second, and most important, our goal was to measure the degree of conformity with a protocol devised by the department's clinical personnel rather than to implement a new protocol. A plausible argument can be made that conformance would be even lower in the case of an externally developed protocol as clinicians might resist change conceived elsewhere. In this case, our data may underreport quality problems that arise in the cases of implementing externally developed protocols.12
A possible limitation of our study is the fact that the documented quality gaps have not been linked to adverse final outcomes. In order to establish any existing difference a larger, randomized study, with recording of Quality of Life questionnaires and overall survival is needed. This link between processes and outcomes has long been the subject of debate in the quality of care research community.13,14 It is worth noting however that all costly and inconvenient therapeutic interventions are undertaken under the assumption that the process–outcome link does exist, i.e. that had these tests been omitted, a negative final outcome would have occurred.15
An important step in increasing conformance quality is to explicitly consider the tools used to ensure adherence to existing guidelines. Research has shown that the existence of specific processes to record performance and provide feedback is much more effective than the simple acknowledgement of the existence of a certain clinical protocol.3,16 Our study has used an electronic tool to increase adherence. It is of course difficult to disentangle the effect of the protocol itself from the effect of the electronic implementation. However, given that medical records are gradually being converted to electronic format, it seems that such disentanglement is not necessary since protocols would be set up and used electronically anyway. In this case, current practice would increase conformity in comparison to our study, which included a requirement for duplicate recording in both paper and electronic format.
In conclusion, our study provides evidence that there may be quality problems in the diagnosis and treatment of lung cancer disease. Clinicians should pay attention to conformance quality as one of the most important aspects of the quality of care delivered. This is particularly so for debilitating diseases such as advanced lung cancer, where major new treatment advances have not been achieved thus far. Thus, reducing the variation in the implementation of current treatment options might be one of the few steps available to provide an immediate incremental benefit for patients.
Key points
|
|
References |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
1 Chassin MR. Is health care ready for Six Sigma quality? Milbank Q 1998;76:565–91.[CrossRef][Web of Science][Medline]
2 Stroner PL, Brewster DH, Dewar JA, et al. In pursuit of excellence for patients with cancer: the Scottish Cancer Therapy Network model. Br J Cancer 1999;79:1641–5.[Medline]
3 Smith TJ, Hillner BE. Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways. J Clin Oncol 2001;19:2886–97.
4 Vanderpump MP, Alexander L, Scarpello JH, Clayton RN. An audit of the management of thyroid cancer in a district general hospital. Clin Endocrinol (Oxf) 1998;48:419–24.[Medline]
5 Hutchinson A, McIntosh A, Cox S, Gilbert C. Towards efficient guidelines: how to monitor guideline use in primary care. Health Technol Assess 2003;7:1–97.[Medline]
6 Grol R. Implementation of evidence and guidelines in clinical practice: a new field of research? Int J Qual Health Care 2000;12:455–6.
7 Hashimoto H, Bohmer RM, Harrell LC, Palacios IF. Continuous quality improvement decreases length of stay and adverse events: a case study in an interventional cardiology program. Am J Manag Care 1997;3:1141–50.[Web of Science][Medline]
8 Juran J, Godfrey Blanton A. Juran's Quality Handbook, 5th edition. New York, NY: McGraw Hill Publications, 1999;4.16–4.20.
9 Smith TJ, Hillner BE. Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways. Clin Oncol 2001;19:2886–97.
10 Howe HL, Wingo PA, Thun MJ, et al. Annual report to the nation on the status of cancer (1973 through 1998), featuring cancers with recent increasing trends. J Natl Cancer Inst 2001;93:824–42.
11 Solan MJ, Werner-Wasik M. Prognostic factors in non-small cell lung cancer. Semin Surg Oncol 2003;21:64–73.[CrossRef][Medline]
12 Harteloh PP, Verheggen FW. Quality assurance in health care. From a traditional towards a modern approach. Health Policy 1994;27:261–70.[Medline]
13 Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care 2001;13:475–80.
14 Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care 2001;13:469–74.
15 Mandelblatt JS, Ganz PA, Kahn KL. Proposed agenda for the measurement of quality-of-care outcomes in oncology practice. J Clin Oncol 1999;17:2614–22.
16 Palmer RH, Louis TA, Peterson HF, Rothrock JK, Strain R, Wright EA. What makes quality assurance effective? Results from a randomized, controlled trial in 16 primary care group practices. Med Care 1996;34 Suppl 9:SS29–39.[Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C.-R. Chien and M.-S. Lai Trends in the Pattern of Care for Lung Cancer and Their Correlation With New Clinical Evidence: Experiences in a University-Affiliated Medical Center American Journal of Medical Quality, November 1, 2006; 21(6): 408 - 414. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||