OUP user menu

Physicians’ knowledge, attitudes and professional use of RCTs and meta-analyses: A cross-sectional survey

Corrado De Vito , Carmelo G. Nobile , Giacomo Furnari , Maria Pavia , Maria De Giusti , Italo F. Angelillo , Paolo Villari
DOI: http://dx.doi.org/10.1093/eurpub/ckn134 297-302 First published online: 7 January 2009

Abstract

Background: Familiarity with Randomized Control Trials (RCTs) and meta-analyses is essential to practice Evidence Based Medicine (EBM). The aims of this study were to describe knowledge, attitudes and professional behavior of physicians towards methods to evaluate the efficacy of health interventions, particularly meta-analysis and to find out their possible associated factors. Methods: A cross-sectional survey was carried out on a random sample of Italian physicians through a self-administered questionnaire. Results: A response rate of 70.1% was achieved (654 questionnaires). Despite satisfactory knowledge and substantial positive attitudes, Italian physicians have not integrated the use of RCTs and meta-analyses into their practice to a large extent, because they infrequently read and use RCTs and meta-analyses to make decisions in clinical practice. There is a high correlation between knowledge, attitudes and professional use of RCTs and meta-analyses. Moreover, the results of our multivariate analysis show that the probability of an appropriate professional use, that is higher for doctors who know the English language, have internet access and dedicate a proper amount of time to continuing medical education, increases significantly with a previous exposure to meta-analysis during graduate/post-graduate training (OR 2.25, 95% CI 1.44–3.52), and with the attendance of post-graduate courses about EBM (OR 1.75, 95% CI 1.09–2.82). Finally, Italian physicians demonstrate a high level of interest in further training. Conclusions: The association between the EBM educational background of doctors and the appropriate professional use of RCTs and meta-analyses suggest that EBM training may promote a more evidence-based practice among physicians.

  • attitudes
  • knowledge
  • meta-analysis
  • physicians
  • professional use
  • RCT

Introduction

The Evidence Based Medicine (EBM) ‘movement’ was born officially in the early 90s with the aim of making health care professionals able to apply the best evidence to their practice. The skills needed to provide evidence-based solutions to clinical problems include defining the problem; constructing and conducting an efficient search to locate the best evidence; critically appraising the evidence; and considering that evidence, and its implications, in the context of patients’ circumstances and values. However, attaining these skills requires intensive study and frequent, time-consuming applications.

Practicing clinicians are not necessarily interested to learn and improve their own ability in using original literature and evidence-based methods and, however, those who want to do it are often short of time to apply these skills.1,2 Instead, many of them are more inclined to use secondary sources of evidence which provide a comprehensive point of view and immediately applicable conclusions.3,4 The use of these sources assumes a familiarity with Randomized Control Trials (RCTs) and, particularly, meta-analysis, which currently receives more citations than any other type of study design.5 Therefore, the assessment of physicians knowledge, attitudes and professional use of these evaluation tools is an important public health matter.

Although several studies surveyed knowledge, attitudes and professional use of EBM methods,1,2,6–10 clinical practice guidelines4,11–15 and biostatistics,16,17 to our knowledge no surveys have investigated the use of meta-analyses and RCTs among physicians. Therefore, the aims of this study were to describe knowledge, attitudes and professional behavior of Italian physicians towards methods to evaluate the efficacy of the health interventions, particularly meta-analysis, and to find out their possible associated factors.

Methods

Participants

The source population includes all physicians of the Province of Rome and Calabria Region. The sampling frames were the registers of the Board of Physicians of the Province of Rome (33 533 subjects) and the five provinces of Calabria (13 604). To obtain a random sample of 1000 physicians (700 from Rome and 300 from the Calabria Region, ∼2% of the source population), two lists of 700 and 300 random numbers were computer-generated and the corresponding numbers of these registers were selected.

The analysis of the association between knowledge, attitudes and physicians use of RCTs and meta-analyses was deemed to be of most importance relating to the objectives of the study and was used to calculate the sample size. The study was designed to have 80% power to detect an odds ratio (OR) of 2 at α = 0.05 level. Assuming a conservative scenario of 10% prevalence of low professional use of RCTs and meta-analyses in physicians with poor knowledge or negative attitudes, this scenario results in a sample size of 614. Our sample size was inflated to take into account a response rate of ∼70%. Sample size calculation was performed using Epi-Info 2004 version 3.3 (CDC, Atlanta).

A self-administered anonymous questionnaire was mailed to the selected participants, with a cover letter providing details about the study and reassuring physicians of their anonymity. Second and third questionnaires were mailed to all physicians in the sample 3 and 6 months later, respectively. In an attempt to maximize the responding rate, telephone calls were made to all physicians before each of the follow-up mailings. A total of 98 physicians could not be contacted by telephone because their numbers were not available.

Survey instrument

The questionnaire was divided into five sections: demographic and professional characteristics; knowledge about RCTs and meta-analyses; attitudes towards methods for the evaluation of efficacy of health interventions; use of RCTs and meta-analyses in clinical practice; self-reported level of knowledge and training needs concerning RCTs and meta-analyses.

For assessing knowledge, physicians were asked four questions formulated using a three-point Likert scale with options for ‘agree’, ‘uncertain’ and ‘disagree’ and the same scale was used for the six questions exploring physicians’ attitudes (see Results section for details). In the behavior section, physicians’ use of RCTs and meta-analyses in clinical practice was investigated by asking how often they read RCTs and meta-analyses and if they use RCTs and meta-analyses results to make their clinical decisions. Questions on physicians’ behavior were in a five-answer format of ‘never’, ‘rarely’, ‘sometimes’, ‘often’, and ‘very often’ or in a ‘yes/no’ format. Finally, a self-judged level of knowledge in a four-answer format (‘inadequate’, ‘sufficient’, ‘good’ and ‘excellent’) and the needs of training (‘yes/no’ answer) were explored.

Extensive pre-administration piloting was conducted with a convenience sample of physicians similar to the study population to ensure practicability, validity and interpretation of answers. On the basis of the comments and suggestions obtained from the pilot study, the questionnaire was revised before distribution to the study sample. Instrument revision included changes to questionnaire item wording and format only if there was near universal consensus on their meaning.

Statistical analysis

Stepwise logistic regression with backward elimination was performed to identify predictors of each of: knowledge about RCTs and meta-analysis (Model 1), positive attitudes towards the methods to establish the efficacy of the health interventions (Model 2), and use of RCTs and meta-analysis in clinical practice (Model 3). For purposes of analysis, the outcome variables, originally consisting of multiple categories, were collapsed into two levels: physicians who agreed with all correct responses versus all others (Model 1); those who showed positive attitudes versus all others (Model 2); physicians that used RCTs and meta-analyses in their clinical practice versus all others (Model 3).

The following predictor variables were initially tested in all models: location; gender; age; postgraduate training; academic appointment; exposure to meta-analysis during graduate/postgraduate courses; postgraduate courses about EBM attended; level of English language knowledge; internet at the workplace; hours per week dedicated to continuing medical education. In the model concerning attitudes (Model 2), the variable adequate knowledge of RCTs and meta-analyses was also included, while in the model about the use of RCTs and meta-analyses in clinical practice (Model 3) both variables adequate knowledge and positive attitudes towards RCTs and meta-analyses were considered.

Multiple logistic regression models were built using the strategy suggested by Hosmer and Lemeshow.18 Each variable was examined by univariate analysis using the appropriate statistic test (Student's t-test and chi-squared test) and included in the model when the P-value was lower than 0.25. Subsequently, multivariate logistic regression with backward elimination of any variable that did not contribute to the model on the grounds of the Likelihood Ratio test (cut-off at P = 0.05) was performed. Variables whose exclusion altered the coefficient of the remaining variables were kept in the model. Interaction terms were tested using a cut-off of 0.15 level significance. Adjusted OR and 95% confidence intervals (CIs) were calculated.

All statistical calculations were performed using Stata version 8.0 (College Station, TX, Stata Corporation, 2003).

Results

Study population

Of the original sample of 1000 physicians, 67 were ineligible because they were retired or no longer in clinical practice. The final sample size included 933 physicians with 654 respondents for a response rate of 70.1%. Responders and non-responders were comparable, because the available demographic characteristics of non-responders (location, gender and age) were not statistically different from those of responders. Most of the respondent physicians were from Rome (69.3% versus 68.8% of non-respondents, P = 0.892) and male (68.7% versus 68.2%, P = 0.865), with a mean age of 49.4 years (49.8 years in non-respondents, P = 0.615). The demographic characteristics of our sample of physicians are also similar of those of Italian physicians, since 65.2% of the members of the National Board of Physicians are males, with a similar age distribution.19 Almost all respondents had a specialization (94.0%), but only 25.2% were exposed to meta-analysis during graduate or postgraduate training and only 19.2% attended specific postgraduate courses about EBM. The level of English language knowledge appears unsatisfactory, because almost half of the responders declared a very low or a low level of knowledge. An internet connection and a medical library at the workplace were available for 64.9% and 45.2% of physicians, respectively. The majority of the sample (60.2%) dedicates 1–5 h per week to continuing medical education (table 1).

View this table:
Table 1

Selected demographics and professional characteristics of the responding physicians.

VariablesN(%)
Location (654)a
    Calabria201(30.7)
    Roma453(69.3)
Gender (651)a
    Female204(31.3)
    Male447(68.7)
Age, years (650)a
    <40152(23.4)
    41–50239(36.8)
    51–60150(23.1)
    >61109(16.8)
Academic appointment (646)a
    No595(92.1)
    Yes51(7.9)
Postgraduate training (617)a
    No specialization37(6.0)
    Specialization580(94.0)
Exposure to meta-analysis during graduate/postgraduate training (646)a
    No483(74.8)
    Yes163(25.2)
Postgraduate training courses about EBM (641)a
    No518(80.8)
    Yes123(19.2)
English language knowledge (649)a
    Very low137(21.1)
    Low174(26.8)
    Intermediate198(30.5)
    Good97(15.0)
    Excellent43(6.6)
Internet at the workplace (636)a
    No223(35.1)
    Yes413(64.9)
Medical library on the workplace (637)a
    No349(54.8)
    Yes288(45.2)
Hours per week dedicated to continuing medical education (638)a
    <1139(21.8)
    1–5384(60.2)
    6–1085(13.3)
    >1030(4.7)
  • a: Number of physicians responding to the question

Knowledge

Italian physicians’ knowledge about RCTs and meta-analyses appears adequate (table 2), since 78.5 and 61.3% agreed that RCTs and meta-analysis are able to evaluate the efficacy of the health interventions, respectively. The majority of physicians knew that meta-analysis combines results of different individual studies (71.4%), but the level of knowledge decreased with more technical questions, because only 50.8% knew that relative risk and odds ratio are effect measures used in RCTs and meta-analysis. Although correct responses were frequent in single questions, a total agreement with all correct responses was encountered only in 27.8% of participants, and this knowledge was significantly higher if physicians attended a previous post-graduate EBM course (P = 0.026), when internet was not available at the workplace (P = 0.001), and if physicians dedicated a proper amount of time to continuing medical education (P = 0.007). Unexpectedly, multivariate analysis confirmed only the association with the absence of internet at the workplace, and identified the English language knowledge as an additional significant factor associated with adequate knowledge (Model 1 in table 3).

View this table:
Table 2

Knowledge of the responding physicians about RCTs and meta-analysis

Agree (%)Uncertain (%)Disagree (%)
RCTs are able to evaluate the efficacy of preventive and curative health interventions (628)a78.517.44.1
Meta-analysis is useful to draw conclusions about the efficacy of health interventions (622)a61.334.24.5
Meta-analysis combines the results of different individual studies with the purpose of integrating the findings (619)a71.425.72.9
Relative risk and Odds ratio are measures used in RCTs and meta-analyses to quantify the effect of health interventions (604)a50.840.48.8
  • a: Number of physicians responding to the question

View this table:
Table 3

Results of the logistic regression models

VariableOR95% CIP-value
Model 1: Knowledge about RCTs and meta-analyses
    English language knowledge (very low = 0; low = 1; intermediate = 2; good = 3; excellent = 4)1.271.07–1.520.007
    Internet at the workplace (no = 0; yes = 1)0.410.27–0.63<0.001
Model 2: Attitudes towards methods to evaluate the efficacy of health interventions
    Gender (female = 0; male = 1)1.931.13–3.310.016
    Academic appointment (no = 0; yes = 1)2.071.02–4.210.045
    Knowledge (not adequate = 0; adequate = 1)5.103.20–8.15<0.001
Model 3: appropriate use of RCTs/meta-analyses
    Postgraduate training (no specialization = 0; specialization = 1)0.380.16–0.870.023
    Exposure to meta-analysis during graduate/postgraduate training (No = 0; yes = 1)2.251.44–3.52<0.001
    Postgraduate training courses about EBM (No = 0; yes = 1)1.751.09–2.820.020
    English language knowledge (low or very low = 0; intermediate, good or excellent = 1)1.631.07–2.500.024
    Internet at the workplace (no = 0; yes = 1)1.681.05–2.670.029
    Hours per week dedicated to continuing medical education (<1 = 0; 1–5 = 1; 6–10 = 2; >10 = 3)1.351.02–1.770.036
    Knowledge (not adequate = 0; adequate = 1)1.721.09–2.710.020
    Attitude (negative = 0; positive = 1)3.382.06–5.52<0.001

Attitudes

Attitudes towards methods for the efficacy evaluation of health interventions are quite diversified (table 4). Although the large majority (75.5%) agreed that only health interventions with proven efficacy should be freely administered to the population, a lower percentage believed that meta-analysis contributed significantly to knowledge about prevention and cure of diseases (57.9%), that application of results of RCTs and meta-analyses should improve the health status of patients (58.9%), that meta-analysis is a useful tool to physicians to select effective health interventions (62.9%), and that clinical practice requires efficacy evaluation of health interventions carried out through meta-analysis (45.3%). Many physicians (61.1%) believed that decisions in clinical practice cannot be based on the results of meta-analyses but rather on the individual patient needs. However, this belief cannot be judged as a negative attitude because this statement is consistent with the current definition of EBM.20 A total of 20.3% of physicians showed a positive attitude in all first five questions, and this dichotomization was used to identify predictors of a positive attitude. Univariate analysis showed that male physicians (P = 0.019), physicians with an academic appointment (P = 0.024), physicians who attended previously a post-graduate EBM course (P = 0.036), physicians who can rely upon internet at the workplace (P = 0.003), and those with an adequate knowledge about RCTs and meta-analyses (P < 0.001) were more likely to have positive attitudes towards methods to evaluate efficacy of health interventions. Multiple logistic regression analysis, however, confirmed only the association between positive attitudes and gender, academic appointment and adequate knowledge (Model 2 in table 3).

View this table:
Table 4

Attitudes of the responding physicians towards clinical trials and meta-analyses

Agree (%)Uncertain (%)Disagree (%)
Only health interventions with proven efficacy should be free (629)a75.512.412.1
Meta-analysis contributed significantly to knowledge about prevention and cure of diseases (617)a57.938.73.4
Application of results of RCTs and meta-analyses improve the health status of patients (618)a58.935.45.7
Meta-analysis is an useful tool to help physicians to select effective health interventions (620)a62.934.52.6
Clinical practice requires efficacy evaluations of health interventions carried out through meta-analyses (618)a45.343.711.0
Many decisions in clinical practice cannot be based on the results of RCTs and meta-analysis but rather on the individual patient needs (620)a61.129.89.0
  • a: Number of physicians responding to the question

Professional behavior

The use of RCTs and meta-analyses in clinical practice is not frequent among Italian physicians. Only 40.4% read often or very often RCTs and a lower percentage meta-analyses (18.6%). A quite high number of physicians do not use the results of RCTs (38.7%) or meta-analysis (55.5%) to make decisions in their clinical practice. A total of 32.1% of physicians read sometimes, often or very often RCTs and meta-analyses and use the results of RCTs and meta-analyses in their clinical practice, and this was defined arbitrarily as appropriate use. There was a strong association between this appropriate use and adequate knowledge (P < 0.001), as well as positive attitudes (P < 0.001). Other significant predictors of the appropriate use of RCTs and meta-analyses in clinical practice, at the univariate analysis, were lack of specialization (P = 0.034), academic appointment (P = 0.011), exposure to meta-analysis during graduate/postgraduate training (P < 0.001), attendance of postgraduate training courses about EBM (P < 0.001), higher level of English knowledge (P < 0.001), internet at the workplace (P = 0.005), and higher number of hours per week dedicated to continuing medical education (P < 0.001). Multivariate analysis confirmed almost entirely the results of the univariate analysis, with the exception of the academic appointment, which was found not longer associated with appropriate professional use (Model 3 in table 3). No statistical significant interaction terms were found, suggesting that the effect of each associated factor did not change with the presence or absence of other possible associated factors.

Half of the responding physicians (51.9%) considered inadequate their knowledge about RCTs and meta-analysis and only 13.0% good or excellent. The vast majority (87.4%) believed that their methodological knowledge should be improved.

Discussion

The results of this study show that Italian physicians surveyed in this study know the definition and the objectives of RCTs and meta-analyses, but their knowledge decreases facing more technical questions, such as, for example, those concerning intervention effect measures. The level of UK general practitioners’ knowledge of technical terms used in EBM is similar to that detected in our study, because the knowledge is higher when the questions concern the awareness of terms such as ‘systematic review’ or ‘meta-analysis’ and lower for knowledge of terms such as ‘odds ratio’.1 Similarly, Australian general practitioners showed that respondents were more confident of their understanding of the term ‘randomized control trial’ than of terms such as ‘confidence interval’ or ‘positive predictive value’.2 In the US the perceived physicians’ knowledge about biostatistics is quite low, despite a clear recognition of its importance to implement the transfer of best evidence in clinical practice.16,17 As expected, the English language knowledge in our study increases the probability of an appropriate knowledge of RCTs and meta-analyses, because the largest part of the published evidence is English-dominant. On the other side, appropriate knowledge is significantly lower when internet is available at the workplace. This finding is difficult to explain, and may be due to the scarce motivation or interest to become aware of tools such as meta-analyses and RCTs by those physicians who, facing a specific clinical topic with uncertain diagnosis or therapy, need and seek fast and ready-to-use information on internet.21

Overall, Italian physicians in this study show substantial positive attitudes not only about the role of RCTs and meta-analysis in clinical practice, but also for its role in health policy, because the majority believe that meta-analysis is an useful tool to select effective health interventions and that only those interventions with proven efficacy should be free. These results are consistent with the level of positive attitudes towards EBM detected in UK,1 Germany,8 Canada,6 Australia2 and Israel.10 Similar positive views towards the role of biostatistics in clinical practice16 and clinical practice guidelines4,11–13,15 were reported in other cross-sectional surveys performed in the US as well as in other countries. However, several studies revealed that physicians often consider guidelines too rigid to apply to individual patients, that they reduce professional autonomy and oversimplified medicine as a ‘cookbook’.4,11,13 Patient demand for treatment despite lack of evidence for effectiveness was the most cited barrier to EBM by Australian general practitioners,2 and the uselessness of EBM in informing clinical decisions about individual patients was evoked in a survey of senior health professionals in England.7 Similarly, the majority of the respondents in our study believes that many decisions in clinical practice cannot be based on the results of the meta-analysis but rather on the individual patient needs. As expected, an appropriate knowledge is a strong and significant predictor of positive attitudes toward meta-analysis and RCTs, which are associated also to academic appointment and male gender. Other two recent surveys documented an higher knowledge of biostatistics in male doctors who feel more confident at evaluating statistical tests, while females considered patient choice above the evidence more often than males.9,16

Our findings suggest that Italian physicians have not integrated the use of RCTs and meta-analyses into their practice to a large extent, because they infrequently read RCTs and meta-analyses and many of them do no use their results to make decisions in clinical practice. This finding is not new. Surveys carried out among physicians in Canada revealed that, despite a general agreement towards EBM, only a minority of respondents used EBM-related information sources on a regular basis.6,12 Similar behaviors were registered among general practitioners in Australia, who rely more on colleagues’ advices than on evidence consulting.2 The percentage of clinical practice that was evidence base was estimated around 50% by general practitioners in England,3 and only a minority of US physicians reported a change in their professional practice as a result of clinical practice guidelines.11 Despite their wide promulgation, the use of clinical practice guidelines is also low since they had limited effects on changing physician behaviors.14

However, the results of our multivariate analysis, which show a strong association between knowledge, attitudes and professional behavior, identified a sub-group of physicians as an example of successful use of RCTs and meta-analyses in clinical practice. These physicians, who know the English language, have internet access and dedicate a proper amount of time to continuing medical education, were exposed to meta-analysis during their training and attended specific post-graduate courses about EBM. The inverse association between professional use and post-graduate specialization seems contradictory, and might be explained by the fact that the minority of respondents in our survey without specialization is actually young physicians who are attending a specialization course, and, therefore, they are very likely to read and use RCTs and meta-analyses. Unexpectedly, the job setting does not influence the professional use of RCTs and meta-analyses. However, an academic appointment, that affects positively the attitudes, is an independent predictor of professional use if we use more stringent criteria to identify physicians with an appropriate use (those who read often or very often and use RCTs and meta-analyses, data not shown). At the same time, it should be emphasized that almost all Italian physicians of our sample demonstrate a high level of interest in further education to improve methodological knowledge about RCTs and meta-analyses, consistently with the findings of other surveys carried out in several countries.2,6,9,22 There is a clear need of more effective training concerning methods to evaluate efficacy/effectiveness of health interventions.23–29

Limitations of this study should be acknowledged. First, this is a survey, not an audit of actual practice, and thus data are self-reported and therefore potentially inaccurate. As with any survey, there is the possibility of social-response bias in the respondents’ answers. Therefore, our data of physicians’ low professional use of RCTs and meta-analyses may be even an underestimate of the picture in actual practice. Second, even if the response rate is high, the external validity of the findings may be questioned. However, no differences in demographic characteristics between responders and non-responders were found. Third, our data come from of a representative sample of physicians of two regions of Italy, limiting generalizability to the entire country. Nevertheless, the demographic characteristics of our physicians are similar of those of Italian physicians, and we have no reasons to believe that beliefs and professional behaviors of physicians may differ among Italian regions.

Studies in different countries suggest that ∼30–40% of patients do not receive care according to present scientific evidence, and ∼20–25% of care provided is not needed or is potentially harmful.30,31 The nature of modern health care makes it improbable that individual clinicians could make significant changes without some educational, organizational and structural changes in the healthcare system at either local or regional level.32 Substantial evidence suggests that changing behavior is possible, but this change generally requires multiple and comprehensive approaches.33 The results of this study show that education and EBM training may play a very important role to promote a more evidence-based professional behavior of physicians to reach the ultimate goal of pursuing an higher health care quality and containing costs due to inappropriate health interventions. The time appears ripe for new and effective EBM educational programs. Educators should use the most effective approaches according to the most recent scientific evidence, and further and continuous research should examine the effectiveness of EBM educational interventions.

Funding

Italian Ministry of Health; Emilia Romagna Regional Health Agency.

Conflicts of interest: None declared.

Key points

  • Physicians are more inclined to use secondary sources of evidence rather than to appraise the original literature. The use of these sources requires a familiarity with RCTs and, particularly, meta-analysis.

  • Adequate knowledge and positive attitudes improve physicians’ use of RCTs and meta-analyses. Other variables that significantly increase the professional use include knowledge of English language, internet availability and time dedicated to continuing medical education, as well as the exposure to meta-analysis during graduate/postgraduate courses and attending postgraduate training courses about EBM.

  • Education and EBM training may play an important role in promoting EBM clinical practice among physicians. Further and continuous research should examine the effectiveness of EBM educational interventions.

References

View Abstract