OUP user menu

Made in the USA: the import of American Consumer Assessment of Health Plan Surveys (CAHPS®) into the Dutch social insurance system

Diana M. J. Delnoij, Guus ten Asbroek, Onyebuchi A. Arah, Johan S. de Koning, Piet Stam, Aldien Poll, Barbara Vriens, Paul Schmidt, Niek S. Klazinga
DOI: http://dx.doi.org/10.1093/eurpub/ckl023 652-659 First published online: 8 March 2006

Abstract

Background: In the Netherlands, managed competition between health plans has been introduced. For Dutch health plans this implies that they need to collect data about their own performance and that of the care providers they contract. To that end, Consumer Assessment of Health Plan Surveys (CAHPS) instruments have recently been adopted by a large Dutch health plan. Objectives: This paper presents the results of a validation study of the Dutch version of the CAHPS Adult Commercial questionnaire. The questions addressed are as follows: Can this questionnaire be adapted for use in the context of the Dutch insurance system? and Can it generate valid information about the quality of health care and the performance of Dutch health plans? Methods: The translated questionnaire has been mailed to a sample of 977 enrollees. The psychometric properties of the translated instrument have been studied, and the results have been compared with those of other Dutch and American studies. Results: The net response rate was 51% (n = 500). In general, the questionnaires were filled out completely and consistently. Principal component analyses revealed a factor that can be labelled as patient-centredness in the primary process. It contains the domains that in the CAHPS literature are described as ‘courteous/helpful staff’ and ‘doctors communicating well’. Conclusions: The translated version of the CAHPS Adult Commercial questionnaire is a promising tool for Dutch health plans. More research is needed on the external and the content validity of these questionnaires in the Dutch context.

  • health services research
  • patient satisfaction
  • questionnaire development
  • social insurance

In a number of countries, e.g. Germany, Switzerland, and the Netherlands, managed competition between health plans has been introduced.1 In 1992, in the Netherlands free choice of health plans has been introduced along with a system of regulated price competition between health plans. It is expected that enrollees will choose the cheaper insurance with the better service, thus inducing health plans to increase their efficiency and service orientation.2 For Dutch health plans this implies that they need to collect data about their own performance and that of the care providers they contract, as perceived by their enrollees. They can use this information in their negotiations with contracted providers, and as consumer information on which enrollees can base their choice of health plan.

In their search for useful tools that measure providers' performance, health plans can choose from a variety of ‘families’ of questionnaires that measure the quality of care from the patient's perspective or the patient-centredness of health care.315 All these questionnaires measure patients' experiences rather than patient satisfaction.16 The domains covered are also comparable to a certain extent.17 All the instruments focus on different aspects of the patient-centredness of health care such as respect, dignity, prompt attention, autonomy, etc.

These domains are also addressed by CAHPS® (Consumer Assessment of Health Plan Surveys).1821 CAHPS consists of a set of standardized questionnaires and reporting methods, aimed at providing consumers with the information they need when choosing a health plan.22 CAHPS focuses on the service of the health plan itself and the relation between health plan performance and access to and quality of care. CAHPS is therefore particularly suitable for use by Dutch health insurers that have to act as strategic purchasers of care.

CAHPS instruments have been adopted by a large Dutch health plan (Agis) that has a major market share in the city of Amsterdam and the central regions of the Netherlands. Agis has field-tested a translated version of the CAHPS Adult Commercial questionnaire, version 3.0. This paper presents the results of a validation study of that Dutch version of CAHPS. The research questions are as follows:

  1. Can the American CAHPS Adult Commercial questionnaire be adapted for use in the context of the Dutch insurance system?

  2. Can this questionnaire generate valid information about the quality of health care and the performance of Dutch health plans?

The second question could be addressed in the validation study only to a limited extent, through comparison of the test results with data from other Dutch studies and with the American benchmark data.

Methods

Translation of the CAHPS Adult Commercial questionnaire

The 67 items in our questionnaire (see Appendix) stem mostly from the American CAHPS Adult Commercial questionnaire and the supplementary CAHPS questions. These supplementary questions also addressed the fluency of the respondent in—in our case—the Dutch language and the use of interpreters. The questions were added because ∼10% of the Dutch population, and close to 50% of the Amsterdam metropolitan population covered by Agis, consists of migrants. All the selected questions were translated forward and backward by a total of four interpreters, using the standard procedure for translating survey instruments (figure 1).

Figure 1

Steps in the translation process

Fielding the questionnaire

From its general database of the insured population, Agis selected a representative sample of 1000 adult insured (aged 18 or older) who were insured on 1 April 2003, i.e. 6 months prior to the sample selection. A check was conducted whether these people had not recently deceased. After this first cleaning of the data, 977 insured remained in the pilot and received a first postal questionnaire in the beginning of November 2003. After three reminders, a total of 544 respondents returned their questionnaire. Fresh questionnaires were sent with the second reminder. Figure 2 shows the number of questionnaires returned by date. The peaks correspond with the reminders. Late responders (defined as respondents whose questionnaire was received after 15 December 2003) are significantly more often male (χ2 = 6.71, P = 0.01) and tend to be younger, though this relationship is not statistically significant. Among the respondents, people over 55 were overrepresented (χ2 = 47.9, P = 0.00).

Figure 2

Number of questionnaires returned by data

The respondents are also older than the general Dutch population (7% is over 80, compared to 4.5% of the adult Dutch population23) and on a 5-point scale more often report a fair or poor (as opposed to good, very good, and excellent) health status (32% compared to 19.4% in the Dutch population24). This should be kept in mind when comparing the results with those of other Dutch studies.

Validation of the questionnaire

In the validation process, the subsequent steps were followed that are described in the CAHPS guidelines T4: Preparing the Data for Analysis, Document Number 14 from the CAHPS® Kit (http://www.cahps-sun.org/Cahpskit/KitsLogin.asp). This means that respondents were excluded who did not confirm that they were insured with Agis (n = 17); who had not been insured with Agis for at least 12 months (n = 9); and who stated that someone else had filled out the questionnaire for them (n = 17). After that, we checked whether the questionnaire was filled out completely. This was done by looking at a set of 19 so-called key questions, of which at least 10 should be answered. Incomplete questionnaires (n = 1) were excluded from further analyses.

The database that had thus been cleaned consisted of 500 cases and was used to look at the following:

  • The consistency of the data: We checked whether respondents followed skip-rules and whether they gave consistent answers.

  • Frequencies and missing values: Questions with more than 5% missing values or extremely skewed items, with more than 90% of the respondents choosing the most positive answer, were considered to be candidates for exclusion from a next version of the questionnaire.

  • Underlying factors, scales, and reliability: Principal component and reliability analyses were conducted on the questions with respect to patients' experiences with their health care in the past 12 months.

  • Comparability of the findings with results from other Dutch patient surveys and the American National CAHPS Benchmarking Database: The results of the Dutch pilot were compared with the American data and it was tested whether relations that are reported in the American literature can be reproduced in the Dutch data. This was be done by looking at the relation between respondents' evaluations of care and their age, level of education, and general and mental health status. Multiple regression analyses were carried out on the general ratings of the personal doctor, the specialist, the health care in the past 12 months, and the health plan.

Results

Consistency of the data

In general, the questionnaires were filled out completely and consistently. This became evident from examining the completeness with which the 19 so-called key questions were filled out (only one case had to be excluded because of incomplete data). But it is also the conclusion that was drawn after examining whether respondents followed skip-rules and whether they gave consistent answers. There was always a limited number of respondents who did answer questions that they were supposed to skip, but this number was generally between 20 and 30 respondents (so ∼5% of the respondents); with a minimum of 4 (respondents giving a score between 0 and 10 for their specialist, whereas they had previously answered that they had not visited a specialist) and a maximum of 49 (respondents stating to have had problems contacting the customer service of the insurance company, whereas they had previously stated not to have tried contacting the customer service). An inconsistency not related to skip instructions in the questionnaire, was found for eight respondents who said that their specialist was their personal doctor, whereas earlier they had answered that they had no personal doctor. Other major inconsistencies were not detected in the data.

Frequencies and missing values

Table 1 shows the frequencies of respondents using health care and services offered by the insurance company. In table 1, it can be seen that in the past 12 months only nine respondents said they had needed the help of an interpreter in their contact with health care providers. Some other services too were used by a relatively small percentage of respondents. This is the case for out-of-hours care (n = 81). Also, comparatively few people had sought information about the insurance company (n = 76) or had filed a complaint (n = 60).

View this table:
Table 1

Utilization in the past 12 months of care and of services offered by the insurance company by respondents (n = 500)

QuestionYesNoMissing
Do you/did you…n%n%n%
Have a personal doctor or nurse?30661.218136.2132.6
Think a visit to a specialist was necessary?21843.627154.2112.2
Call for help during office hours?27054.022545.051.0
Call for help outside office hours?8116.241082.091.8
Need emergency care?11422.439276.940.8
Make an appointment to get care?22545.026853.671.4
Need an interpreter?91.847995.8122.4
Get information when you entered the insurance company?18136.229559.0244.8
Seek information about the insurance company?7615.241482.8102.0
Call customer services?12725.435871.6153.0
Call or write to the insurance company to file a complaint?6012.042685.2142.8

In the past 12 months, about one-third of the respondents used the out-of-hours services of family physicians (n = 169, not in table), and about two-thirds had seen a family physician or specialist in the past 12 months not counting emergency or out-of-hours care (n = 313, not in table). Comparatively few people (14%, n = 70, not in table) visited an emergency department of a hospital.

The respondents, who had used care, were asked about their experiences with respect to access to services, prompt attention, communication, quality of information, etc. As expected, the distribution of scores on the frequency with which quality standards are met are skewed towards positive answers. But none of the items violated the practical rule set in advance, that no more than 90% of the answers should be in the most positive category.

Factors, scales, and reliability

The questionnaire contains a series of questions about the patient-centredness of care received in the past 12 months. Those were as follows:

In the past 12 months, how often

  • Were you taken to the exam room within 15 min of your appointment?

  • Did office staff at a doctor's office or clinic treat you with courtesy and respect?

  • Was office staff at a doctor's office or clinic as helpful as you thought they should be?

  • Did doctors or other health providers listen carefully to you?

  • Did doctors or other health providers explain things in a way you could understand?

  • Did doctors or other health providers show respect for what you had to say?

  • Did doctors or other health providers spend enough time with you?

  • Did you find it difficult to talk to a doctor or other care provider or to understand them, because you spoke different languages?

On these items, we performed a principal component analysis with varimax rotation. This resulted in a two-factor solution. Item ‘i’ (taken to the exam room within 15 min) had a factor loading of <0.40 on both factors. In the American data too, this item did not fit into the observed factor structure.25 The items that did have a factor loading of over 0.40 on factor 1 were items ‘(ii)’ through ‘(vii)’. Factor 2 consisted of items ‘(viii)’ (spoke different languages) and ‘(vii)’ (spend enough time with you). Because the latter item ‘(vii)’ also loaded high on factor 1, we decided to run another factor analysis only on items ‘(ii)’ through ‘(vii)’. This resulted in a one-factor solution. Table 2 shows the factor loadings of the different items on that factor. This factor can be labelled as patient-centredness in the primary process, and it contains the domains that in the CAHPS literature are described as ‘courteous/helpful staff’ and ‘doctors communicating well’. In American validation studies too, these two domains can be forced into one factor. But in the American data, the items ‘(ii)’ and ‘(iii)’ about office staff then have factor loadings of below 0.70.25 This is not the case in the Dutch data. Here, item ‘(v)’ (explain things in a way you could understand) has a comparatively low factor loading.

View this table:
Table 2

Factor loading of items concerning the patient-centredness of care received in the past 12 months (n = 331)

ItemFactor loading
How often did doctors listen carefully?0.85
How often did doctors show respect for what you had to say?0.83
How often did doctors treat you with courtesy and respect?0.78
How often did doctors spend enough time with you?0.77
How often did staff treat you with courtesy and respect?0.71
How often did doctors explain things in a way that you could understand?0.56

Subsequently, we calculated the reliability of the scale for patient-centredness of the primary process. Cronbach's alpha was 0.83. So the six items form a reliable scale. The scale score correlates significantly and positively with respondents' ratings as follows:

  • General rating of the quality of health care on scale of 0–10 (Pearson's r = 0.51, P = 0.00).

  • General rating of their personal doctor or nurse (Pearson's r = 0.47, P = 0.00).

  • General rating of their specialist (Pearson's r = 0.38, P = 0.00).

  • General rating of their health plan (Pearson's r = 0.30, P = 0.00).

This indicates that enrollees with better experiences in the primary process tend to give higher ratings. Significantly, better ratings for the personal doctor and for the health plan were also given by respondents who had been enlisted longer with Agis (the health plan under study).

Comparability with other Dutch and American data

The final step in our validation study consisted of a comparison of our findings with those of other Dutch studies and with the American benchmark data.

Since CAHPS questionnaires have not been used before in the Dutch health care system, comparison with other Dutch data is rather difficult. However, some observations can be made. In a large national study of general practice in the Netherlands, a series of generic items from QUOTE questionnaires were included in the face-to-face interviews conducted with 12 699 respondents from the general population.26 Some of these items are quite comparable with CAHPS items. For instance, questions referred to the degree with which people felt that their family physician spent enough time with them and the degree to which the family physician explained things well. The percentages of respondents who had positive and very positive experiences (measured on a 4-point scale) with these aspects of family doctors' care varied between 90% and 95%.27 Those scores are comparable with the percentage of respondents in our validation study who usually or always had positive experiences.

Other checks that were conducted relate to the utilization data that the Dutch translation of the CAHPS generated. We found that the number of people who said that they had visited a specialist was higher in our sample than in the general Dutch population (44.6% compared to 37.8%).28 Also, the percentage of respondents who had used the out-of-hours services of family physicians is higher than the percentage reported for the general Dutch population (34.3% compared to 26.4%).29 These differences may be caused by the fact that respondents in our study are older than the general Dutch population and more often report a bad health status.

A global comparison with the American CAHPS benchmark data is possible for scores such as the percentage of respondents experiencing problems or the percentage that usually or always has positive experiences and by testing whether relations that are reported in the American literature can be reproduced in the Dutch data. In the American Medicare population, the younger, sicker, and more educated patients give lower ratings.30 Therefore, we checked the relation between respondents' evaluations of care and their age, level of education and general health status via a multiple regression analysis. Controlling for education and health status, a higher age is associated with more positive ratings of the personal doctor (beta = 0.17, P < 0.05), health care in the past 12 months (beta = 0.30, P < 0.05), and the health plan (beta = 0.36, P < 0.05). Respondents who have finished no or primary school only give a higher rating to their health plan (beta = 0.13, P < 0.05). So, do respondents with a better general health status (beta = −0.16, P < 0.05; health status measured on a 5-point scale were 1 is ‘excellent’ and 5 is ‘poor’). To get an idea of the strength of the associations, in table 3 the mean ratings for the different subgroups are presented.

View this table:
Table 3

Mean ratings on a scale from 0 (worst possible) to 10 (best possible) per subgroup of respondents, divided by age, education, and health status

SubgroupPersonal doctorSpecialistHealth care <12 monthsHealth plan
Age
 18–247.37.77.06.4
 25–347.77.37.46.9
 35–447.57.17.47.2
 45–547.37.57.67.2
 55–648.07.78.27.9
 65–748.08.08.08.0
 75–798.68.48.79.0
 >808.37.98.68.4
Education
 Primary school only8.38.18.08.2
 Higher than primary school7.67.57.77.3
Health status
 Excellent7.97.48.07.6
 Very good7.88.07.97.5
 Good7.67.67.67.5
 Fair7.97.78.07.6
 Poor7.87.27.27.0

A comparison of the Dutch outcomes with the American benchmark data suggests that respondents in the US system experience more problems with the accessibility of care than do their Dutch counterparts, especially when it comes to finding a personal doctor or nurse and waiting for consent from the insurance company. However, there are no large differences in the extent to which the two populations get care quickly. Also, the frequency with which one experiences patient-centred care does not seem to differ profoundly between the two systems. The largest differences can be observed with respect to the general ratings of personal doctors, specialists, health care in the past 12 months, and health plans (figure 3). The percentage of people giving ratings between 0 and 6 is somewhat higher in the Netherlands than in the United States. But particularly the percentage of respondents giving ratings between 9 and 10 is much higher in the United States than in the Netherlands.

Figure 3

Rating on a scale from 0 (the worst possible) to 10 (the best possible) of their personal doctor, their specialist, their health care in the past 12 months and their health plan by respondents in the National CAHPS Benchmarking Database (Data for 2003) (USA) and in the Dutch pilot (NL)

Discussion

The purpose of this paper was to describe the results of a validation study of a Dutch version of the CAHPS Adult Commercial questionnaire, which was ‘imported’ from the United States. The comparison with American studies suggests that questions referring to the primary process in health care generate scores that are quite comparable between the American and the Dutch health system. American respondents have more problems getting access to care, but once they manage to get into the system the patient-centredness of the two health care systems is remarkably similar. Interestingly, the percentage of respondents giving ratings between 9 and 10 is much higher in the United States than in the Netherlands. Given the overall similarities in experiences, these differences probably reflect cultural differences rather than differences in the quality of care. A possible explanation could be that the Dutch respondents relate a rating between 0 and 10 to the grading systems that is used in the Dutch school system. The grade 10 reflects a flawless performance, which is why Dutch respondents may be more hesitant than Americans to give this rate.

Despite the similarities in experiences, we could not completely reproduce the factor structure of the American data and the relations between respondents' characteristics and ratings:

  • In the Dutch data, items referring to courtesy and helpfulness of the staff in doctors' offices load high on the factor that also contains the items about doctor–patient communication. Whereas in the American data, experiences with office staff form a separate factor. Perhaps this can be explained from the relatively small scale of doctors' offices in the Dutch health care system. For example, almost half of the Dutch family physicians work in a single-handed practice with the help of one practice assistant.31 In such a setting, the distinction between courtesy of the staff (being one practice assistant, who also provides services) and the communicative skills of the doctor may be difficult to make.

  • Zaslavsky et al.30 report that, in the Medicare Managed Care CAHPS data, health status and age are positively related to the ratings of the personal doctor, the specialist, the health care in the past months, and the health plan. And controlling for health status and age, respondents with a higher education rated health plans, health care, the personal doctor, and the specialist lower than those with less education. Except for the ratings of the health plan, we could not fully reproduce these findings in our study. We did find a positive relation between age and the ratings of personal doctors, specialists, and health care in the past 12 months. But controlling for age, in our study the level of education and the perceived general health status were not associated with the ratings of doctors and health care.

Our study also revealed some practical improvements that can be made in next versions of a Dutch questionnaire. At the time when the questionnaire was translated, it had seemed obvious that the questions with respect to the ‘personal doctor’ would be interpreted by a majority of respondents as referring to their family physician. In the Dutch health care system, family physicians have personal lists and almost every Dutch citizen is enlisted with his/her own (personal) family doctor. Apparently, for a substantial number of respondents this is an administrative reality only, because in our sample 36.2% stated that they do not have a personal doctor or nurse. In a future Dutch survey, we suggest to add two questions asking (i) if the respondent is enlisted with a family physician (the percentage stating ‘yes’ should be between 95% and 100%) and (ii) if their personal doctor is the same as their family physician (analogue to the CAHPS question whether the specialist is the same as the personal doctor).

Apart from this improvement, future surveys should either exclude certain questions or increase the sample size. Because in the sample that we studied some items were relevant to a limited number of respondents only. For example, a very low number of people had needed an interpreter. Perhaps this is not surprising, considering that the questionnaire was in Dutch. Those who are not fluent in Dutch at all will probably not have returned the questionnaire. Also, few people had sought information about the insurance company or had filed a complaint. If a sample size of 1000 or fewer is used, these questions can be excluded from the survey. If these topics would be of specific interest for health plans, we recommend increasing the sample size.

The crude response rate in this study was 56% (net response rate 51%). Compared to other consumer surveys that have recently been conducted in the Netherlands this is relatively high, considering the fact that the general population is addressed. The response rate in disease-specific surveys tends to be higher (around or over 70%). But previous surveys of consumer experiences with health insurance had a much lower response rate (25–30%).32 The fact that three reminders were sent in this study contributed to the relatively high response (cf. figure 2). The third reminder had a relatively small effect on the number of questionnaires returned. However, it does seem to result in getting somewhat more (young) men to participate in a study in which women are slightly overrepresented still. Without the third reminder this overrepresentation would have been larger.

In conclusion, the translated version of the CAHPS Adult Commercial questionnaire is a promising tool that can enable Dutch health plans to disclose transparent information about their own service and client orientation and about the quality of the care they contract. In the summer of 2005, NIVEL conducted a large survey aimed at comparing the performance of all the Dutch health insurance companies. In that survey, we used an improved version of the questionnaire that has been validated here. The data are currently analysed and will be presented on http://www.kiesbeter.nl, the consumer information portal of the Dutch Ministry of Health, Welfare, and Sports.

This more recent survey will provide ample opportunity to further develop the CAHPS questionnaire for the Dutch market. More research is needed on the external and the content validity of these questionnaires in the Dutch context. The CAHPS questionnaires have been developed on the basis of focus group discussions with American people.33 We have to study more deeply what it is that we are measuring with these questions when we pose them to Dutch respondents.

Apart from that the role of CAHPS in the Dutch health care reforms needs to be monitored. This role is potentially 2-fold: CAHPS information can be used by health plans in negotiations with contracting providers, and it can be used by consumers in choosing between health plans. Health plans increasingly have a business case to engage in strategic purchasing of care. Time will tell to what extent CAHPS information is useful for them.

As yet, it is unclear to what extent consumers in the future will choose between competing health plans on the basis of CAHPS information. Currently, Dutch consumers perceive few differences between the quality and services of health plans. For that reason, they see little benefits in going through the administrative turmoil of changing to another health plan.34 Reports based on CAHPS questionnaires may enable health plans to establish more distinctive profiles on the basis of which clients can choose. This can help to break the vicious circle in which insured do not switch to another health plan because they perceive no differences, and health plans do not bother to excel in performance because their clients do not change to a competitor anyway.

Key points

  • This article focuses on the possibilities of ‘importing’ an American CAHPS questionnaire into the Dutch health care system.

  • Using forward and backward translation, the CAHPS 3.0 Adult Commercial questionnaire has been adapted for use in the Dutch system.

  • The Dutch version of the questionnaire has been pilot tested in a sample of 977 enrollees of Agis Zorgverzekeringen (response rate 51%).

  • The translated questionnaire has been filled out completely and consistently, and its psychometric properties are good.

  • CAHPS is a promising tool for preparing consumer information about the performance of Dutch health insurance companies.

APPENDIX

View this table:
Table A1

Summary of the Dutch questionnaire and the translation process

SectionQuestionsDomains
Introduction1–3Check on insurance status (which company, how long, etc.)
Your personal doctor or nurse4–7Having a personal doctor or nurse
Problems finding a personal doctor or nurse (big problem, small problem, not a problem)
Rating of the personal doctor or nurse on a scale from 0 (worst possible) to 10 (best possible)
Getting health care from a specialist8–13Utilization of specialist care
Problems getting a referral (big problem, small problem, not a problem)
Rating of the specialist on a scale from 0 (worst possible) to 10 (best possible)
Your health care in the past 12 months14–42Utilization of care (e.g. visits to doctors, emergency room)
Problems in getting care from doctors, emergency rooms, etc. (big problem, small problem, not a problem)
Frequency with which experiences with prompt attention, respect and dignity, information, shared decision-making, etc. meet quality standards (always, often, sometimes, never)
Rating of health care providers on a scale from 0 (worst possible) to 10 (best possible)
Your health plan (insurance company)43–55Problems in getting information, contacting customer service, filing complaints, etc. (big problem, small problem, no problem)
Frequency with which experiences meet quality standards (always, often, sometimes, never)
Rating of insurance company on a scale from 0 (worst possible) to 10 (best possible)
About you56–67Age (eight categories from younger than 18–24 years of age to 80 and older)
Gender
Health status; general and mental on 5-point scale ranging from excellent to poor
Ethnicity and language
Level of education (11 categories)
Receiving help with filling out questionnaire

References

View Abstract