The European Journal of Public Health Advance Access originally published online on November 7, 2008
The European Journal of Public Health 2009 19(1):100-105; doi:10.1093/eurpub/ckn102
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Miscellaneous |
Statistical modelling needed to find the effects from a community-based elderly safety promotion program
Pia M. Johansson1, Antonio Ponce de Leon1,2, Siv Sadigh1, Per E. Tillgren1,3 and Clas Rehnberg4
1 Karolinska Institute, Department of Public Health Sciences, Stockholm, Sweden.
2 Department of Epidemiology, Institute of Social Medicine, University of Rio de Janeiro, Brazil.
3 Mälardalen University, School of Health, Care and Social Welfare, Västerås, Sweden.
4 Karolinska Institute, Medical Management Centre, Stockholm, Sweden.
Correspondence: Pia M Johansson, Norrbacka floor 2, SE-171 76 Stockholm, Sweden. tel.: +46 8 737 35 23, e-mail: pia.johansson{at}ki.se
Received March 26, 2008, accepted September 29, 2008
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Background: Multiple control areas and time-series analyses have been recommended for effect evaluations of community-based health promotion. Large fluctuations, maybe due to chance, among the areas and over the years might obscure the intervention effect. Methods: A quasi-experimental time-series analysis with several control areas was performed as an effect evaluation of a community-based elderly safety promotion program. The program was implemented during 1995–99 in a community in the Stockholm Metropolitan area (population +65 years: 5500; number of first hip fractures in 1995: 60). Four control areas were selected based on similar hip fracture-related characteristics as the intervention community, complemented with two larger control areas. The time series covered 6 years pre-intervention (1990–95) and 6 years post-intervention (1996–2001). The study population was divided into two age groups and gender, resulting in 28 panels. The first hip fracture incidence was obtained from the Swedish national in-patient register. Results: The time series revealed no discernible pattern, and conventional analyses showed no conclusive results. A multivariate analysis, examining the time trends by employing the intra-annual and intra-panel variance, revealed the underlying trends in hip fracture rates. Comparisons between predicted numbers of hip fractures in the intervention and control areas was enabled, which resulted in 14 less hip fractures in the intervention community than expected from the control communities. If one extreme value was altered, the result changed considerably. Conclusion: Effect evaluations of community-based health promotion programs using time-series from small communities might give faulty results, if statistical modelling is not employed.
Keywords: community intervention, effect evaluation, elderly injuries, panel data, random effects
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Community-based interventions need effect evaluations performed at the community level. As individuals contained within a community might be expected to share some characteristics as well as the physical and social environment, the appropriate level of analysis is not the individual, but the entire community, sometimes called group or cluster.1–3 The recommended design for effect evaluations of community interventions is thus the group-randomized trial, which assumes that the differences in the measured consequences in intervention and control community can be attributable to the intervention.4
If randomization of intervention and control community is not possible, the design is labelled quasi-experimental.5,6 The strongest quasi-experimental design, i.e. in terms of possibilities to ascribe the measured differences to the intervention, is the ‘interrupted time series with comparison series’.3,6 The design includes a time series of data, for a period before and after the intervention, and a control area. It thus measures the variables of interest in two dimensions: time (pre-intervention and post-intervention) and place (intervention area and control area).
The choice of control area is crucial for the results of the evaluation. The idea is that the situation in the control area is the same that would have occurred in the intervention area, if the intervention had not been implemented. The control area should thus be as similar as possible as the intervention area, at least in those factors that are expected to correlate with the measured consequences,3 and unaffected by the intervention. As finding such an area might be difficult, multiple control areas have been recommended.1 If several control areas are employed, the situation is still measured in two dimensions, time and place, but there are now several places, which all are to be compared with the intervention area.
However, when these recommended designs are applied, one might encounter problems. Communities are often small geographical areas, where large fluctuations between the years in the measured consequences might be due only to chance. It might be difficult to discern any underlying trends, ‘to see the woods for the trees’. The problem to interpret the data arises, and thus the difficulty to find the effects.7 This difficulty becomes more obvious with several control areas.
The aim of this article is to describe a situation where an effect evaluation using a quasi-experimental interrupted time series design with several control areas encountered large fluctuations among the control areas and over the years. Statistical modelling, that took into account the variation over time in the control and intervention areas as well as among population groups, was necessary to discern the underlying time trends.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The intervention was an elderly safety promotion program called Safe Seniors, which was implemented in Sundbyberg, a local community in the Stockholm Metropolitan area, Sweden. The community had a population of around 5500 aged 65+ (65 years and older), of which around 1400 were aged 80+ years,8 with a first hip fracture incidence of 12.5 per 1000 person-years for females and 7.8 for males during the years 1990–95. The intervention was implemented during 5 years, and combined structural changes in the environment with individually based measures for the target group.9,10
The evaluation design is an interrupted time-series analysis with six control areas. Controls 3–6 were chosen after a cluster analysis ordered from Statistics Sweden on the basis of seven background factors believed relevant for elderly injuries, in order to find the communities that are most similar to Sundbyberg in those factors in the year 1995. These four control areas are all local communities, but unfortunately, all situated in the Stockholm Metropolitan area. To obtain a larger control area, the whole Stockholm County was selected, as Control 1. Finally, to balance the Stockholm control areas, a number of local communities in Göteborg, the second largest city in Sweden, were selected as Control 2.
The intervention was performed during the years 1995–99. The 6-year period 1990–95 was selected as the pre-intervention period, and the 6 years 1996–2001 as post-intervention period. The effects are assumed to accumulate during this 6-year period, as some intervention measures might affect hip fracture rates immediately, while other measures will affect injuries in the longer run.
The first hip fracture incidence was obtained from the Swedish national registry of hospital admissions, which registers all in-patient hospital stays according to diagnoses, together with patient data such as age, gender, residence, etc. Only the first admission for hip fracture, defined as diagnoses 820–820.9 (ICD-9) and S720–S722 (ICD-10), during the time period was included. The effect evaluation was performed to enable an economic evaluation, so the effects are reported as the number of hip fractures avoided, not as the customary rate of change.
The study population was stratified by gender and age groups, 65–79 years and 80+ years, as epidemiological data indicate different injury rates between these groups.11 The analysis thus includes 28 different panels; (1 intervention area + 6 control areas) x 2 genders x 2 age groups. The word panels reflects that different individuals are included in the groups over time.
The data was first analysed as percentage changes in average rates between the pre- and post-intervention periods for the panels. The rates decreased in Sundbyberg with about the same magnitude as in most control areas: –9% for men aged 65–79 years but –6 to –17% in the control areas (increased rates in Controls 3 and 5); –13% for women 65–79 years compared with –10 to –17% in the controls (increases in Controls 3 and 4); –9% for men aged 80+ but –5 to –46% in controls (increases in Control 2 and 6) and –16% for women aged 80+ years in comparison with decreases of 10–27% (increase in Control 5) (data obtained from the authors on request). The conventional analysis thus concludes that there is no difference in hip fracture trends in Sundbyberg and the control areas.
However, the large fluctuations in rates between the years for all panels, see figure 1a, make analyses based on averages less appropriate. Statistical modelling, however, can take into account variation within panels over time, between age and gender groups, as well as variation within areas. The statistical modelling thus consists of estimating time trends of incidence rates for each of the 28 panels, taking into account panel specificities such as increasing or decreasing time trends, distinct variability patterns over time and linear/non-linear time trends. The analysis is performed in MLwiN 2.02 (Centre for Multilevel Modelling IoE, University of London).
|
The trends are based on cubic splines with fixed or random coefficients, depending on statistical significance. In other words, each coefficient of the spline function was allowed to vary among panels (random coefficient). Statistical significance for the inclusion of the random coefficient was evaluated according to deviance tests, provided models were nested within one another. A spline function was chosen due to its capability to adjust several shapes, from linear to curvilinear.12 In order to facilitate model interpretation the spline functions were centred at the beginning of the post-intervention period. Thus, the fixed coefficients for the two third-degree terms of the spline functions reflect how much more curvature there was before and after 1996, on average.
Fixed effects for age, gender and area as well as interaction terms for age and gender as well as for area and gender, were included regardless of statistical significance. The panel of males, aged 65–79 years, from the intervention community Sundbyberg is the model baseline. Furthermore, the statistical model regarded a specific within-panel variance for each combination of area and gender as well as each combination of gender and age, depending on statistical significance. The combination of area and gender was included to allow examining differences between panel time trends, whereas the combination of gender and age regards the possibility of panels having distinct patterns of fluctuation (i.e. variance) about the continuous time trends due to varying panel (area) population sizes.
The statistical modelling outcome is predicted rates of first hip fractures. These were used to estimate the numbers of hip fractures expected in Sundbyberg, had the situation been the same as in the control areas. The predicted rates for each panel and year were applied to the number of individuals in the age and gender groups in Sundbyberg, to obtain the panel predicted numbers of hip fractures in Sundbyberg. These panel- and year-specific predicted numbers for Sundbyberg were then compared with the predicted Sundbyberg numbers. To adjust for structural differences between communities, unaffected over time and thus not captured in the model, the average difference in rates between Sundbyberg and each panel during the pre-intervention period was used. The adjustment in effect entails a shift in the curves. The remaining difference between the panel predicted numbers and the predicted Sundbyberg numbers is considered the effect of the intervention (see also figure 2).
|
For the year 2000 among Sundbyberg females aged 80+ years there is an extreme value in the observed numbers of hip fractures, 41 vs. 23–29 during the rest of the post-intervention period. To investigate the effects of that value, an alternative model was fitted using the average number, i.e. 26 fractures, instead of the observed number. No other data point was altered.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The statistical modelling reports negative time trend coefficients, which indicates that the hip fracture rates are declining during the time period, even though no coefficient reaches statistical significance, see table 1. The average differences between population groups are considerable, while most coefficients (except for Control 3) on the differences between Sundbyberg and the control areas are negative, but not statistically significant. The results thus indicate that there were only minor differences in levels of first hip fracture incidence in 1996 between Sundbyberg and the control areas. The intra-class correlation was 0.834, i.e. 83% of the variability in the data is due to variation between the panels.
|
Table 1 also reports the model coefficients from the alternative analysis, with one extreme value among females aged 80+ years replaced. All coefficients are changed, which highlights the interrelationships in the statistical modelling, even though the changes are minor. When the base case model is run with the extreme observation assigned a special dummy variable, the coefficient is statistically significant (value: 12.478, SE: 4.439), which confirms that the observation is an outlier that increases the average incidence for the panel.
When the model estimates are used to calculate the panel predicted rates, in figure 1b, a pattern over time and panels evolves (in contrast to figure 1a). There is a marked difference in incidence between the age and gender groups. After 1996, there were decreasing rates for all age groups. The decreases were most pronounced for old females and young men, while there were some divergent trends among young females, in particular a large increase in rates for Control 3, and among older men, with increased rates for Control 6. In Sundbyberg, there was a decreasing trend for all age groups. The widening differences at the end of the time period are driven by the random effects.
The panel predicted numbers are shown in figure 2, as well as the observed and the model predicted numbers for Sundbyberg. The considerable fluctuations in observed numbers over the years are smoothed by the predicted numbers. Except among males aged 80+ years, the control area-specific predicted numbers tend to decrease over time. For the younger age groups and for the males aged 80+, most of the control area-predicted numbers are above the predicted Sundbyberg numbers, indicating lower numbers of hip fractures in Sundbyberg than expected from the control areas. For females aged 80+ years, the predicted Sundbyberg numbers are higher than most of the control area-specific predicted numbers, indicating an increase in hip fractures in Sundbyberg in comparison with the control areas. This is due to the very high observed number in year 2000. During all other years, the observed numbers in Sundbyberg are below the predicted numbers.
The accumulated numbers of first hip fractures in Sundbyberg in the post-intervention period according to the base model are below those expected from the control areas among females aged 65–79 years and among males in both age groups, see table 2. The median number of avoided hip fractures over the control areas is eight for females 65–79 years, and three each for the males in the two age groups, but an increase in 10 among the females aged 80+. However, when the differences are summed for each control area, over the age and gender groups, the total difference ranges between 21 avoided and one increased, with a median total difference of 14 avoided hip fractures. The result is thus interpreted as a total of 14 avoided hip fractures, with eight among females aged 65–79 and three each for the males, and no effects among females aged 80+.
|
The uncertainty surrounding these estimates is however considerable, as no trends were statistically significant. However, the comparison with the larger control areas, i.e. Controls 1 and 2, which exhibit smaller variation, as well as with two of the smaller areas results in a large number of avoided hip fractures in Sundbyberg. On the other hand, comparison with Controls 4 or 5 imply an increase in hip fractures in Sundbyberg with 1 fracture. The alternative analysis, which replaced an outlier among women aged 80+ years, however results in fewer hip fractures in Sundbyberg in comparison with all areas, of the magnitude 7–34 fractures.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
We have estimated a median decrease of 14 hip fractures in the intervention community in comparison with the control areas, by using statistical modelling. A more conventional analysis had concluded that the program had had no effect, because of the difficulties to interpret the highly variable data from six control areas over 12 years. The multiple control areas however enabled the use of statistical modelling, which smoothed the time series by considering the variability among the age, gender and control area groups. A pattern was revealed, which enabled estimation of the intervention effects.
Multilevel techniques are frequently used to evaluate community interventions,3,13 and for other applications within medicine,14,15 to take account of variability at both the individual and supra-individual level. The technique is particularly suitable for the current study,16 as hip fractures are simultaneously dependent on individual-level factors, such as age and gender but also safety behaviour and fitness, and community-level factors, such as routines in elderly care or a risk-reducing street environment. Furthermore, both individual-level and community-level factors were addressed in the intervention studied.
But even modelling cannot amend data that might be unreliable. In the data studied there was an extreme value during one year that managed to change the result altogether for one panel, i.e. females aged 80+ years. We do not know the reason for the extreme observation; it might be due to register misrecordings, an event or a circumstance in the community or just pure chance. It is however implausible that the observation is due to the intervention that is being evaluated. The risks, and effects, of unreliable data are enhanced the fewer the investigated event, which is connected to the number of study persons. In the case of community-based interventions, which are often implemented in small geographical areas, the risk that extreme observations due to pure chance is affecting the effect evaluation results is considerable.
Given the participatory ideal, and effectiveness considerations, of community-based initiatives, many interventions are implemented in communities that express an interest for the issue at hand. This makes a random selection of intervention communities unfeasible. Effect evaluations of interventions implemented in a real-world setting can thus seldom be based on randomized designs, forcing the evaluator to use the best alternative designs available. The quasi-experimental ‘interrupted time series with comparison series’ is such an alternative, which however leads to less confidence in the causal interpretation of the result, and to threats to the internal validity.4
The use of several control areas increase the confidence in the results, as a pattern in differences between Sundbyberg and control areas can be distinguished. If only one control area, e.g. Controls 4 or 5, had been chosen the result of the evaluation had been very different. The choice of control areas was based on an analysis of base-line community characteristics believed important for the community incidence of hip fractures. It included aspects such as housing standards, e.g. if elevators and high kitchen cupboards might be expected, and average community income level and educational attainment, which was assumed to affect the street environment and maintenance. Unfortunately, only small communities in the same metropolitan area shared the Sundbyberg characteristics, why two larger control areas had to be selectively chosen. The 6-year pre-intervention period enable a fairly stable estimation of the time trends, while the 6-year post-intervention period, which includes 4 years of intervention, might be too short to fully capture the effects of the intervention.
The modelling results, i.e. higher hip fracture rates among females vs. males and older elderly vs. younger elderly as well as the decreasing time trends for all groups, are in accordance with epidemiological data.11 But plausible rival hypotheses need to be discussed before claiming that the intervention led to 14 avoided hip fractures. Of the threats to ‘internal validity’ listed by Cook and Campbell,17 testing, instrumentation, maturation, statistical regression and differential mortality, can be ruled out as register data was used on all individuals in certain communities in certain ages. Some control areas were selected to resemble the intervention area, which seeks to minimize the effects of selection bias. The selection-maturation, i.e. that control area individuals change at different rates than the intervention persons, is considered in the statistical modelling. Remains history, as elderly injuries attracted considerable public health interest during the intervention time period. Many other communities, in particular in Stockholm, replicated parts of the intervention, as the methods were considered successful. This affects the plausibility of the evaluation result, i.e. the intervention was considered well implemented with a potential to affect elderly injuries. The spill-over effects to other communities (in particular to the Stockholm controls, i.e. all controls but number 2) might however also imply that the intervention effects are underestimated. Nationally, the public health interest led to an increased use of osteoporosis pharmaceuticals, which might explain the decreasing time trends in hip fracture rates. Given the elderly injury focus in Sundbyberg, medical personnel might be more attentive to the problem and thus more prone to prescribe the pharmaceuticals. The possibility that the intervention also had this indirect effect on injury rates in Sundbyberg cannot be out-ruled.
In conclusion, conventional analyses would have concluded that the intervention had had no effect on hip fractures in Sundbyberg. The small size of the intervention community, as well as some control communities, led to considerable fluctuations in numbers of hip fractures over the years. The statistical modelling revealed the underlying trends, which enabled the estimation that 14 less hip fractures occurred in Sundbyberg than expected from the control areas, which at least in part is due to the intervention. The 14 hip fractures are equivalent to one quarter of the 60 hip fractures that occurred in Sundbyberg the year before the intervention, and a decrease of 0.44 per 1000 person-years. This might appear modest, but as hip fractures lead to considerable societal costs and quality-of-life losses, the cost-effectiveness analysis shows a favourable result.18
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Stockholm County Council.
Conflicts of interest: Preliminary results of the study has previously been presented as an oral presentation at Nordic Health Promotion Research Conference, Esbjerg, Denmark, June 2006.
Key points
|
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The contribution to the study from Anita Hökby, former project leader, Unit of Safety Promotion, Stockholm County Council is gratefully acknowledged.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
1 Cummings P, Koepsell TD. Statistical and design issues in studies of groups. Inj Prev (2002) 8:6–7.
2 Murray DM, Varnell SP, Blitstein JL. Design and analysis of group-randomized trials: A review of recent methodological developments. Am J Public Health (2004) 94:423–32.
3 Brownson RC, Pettiti DB. Epidemologic issues in community intervention. In: Applied epidemiology – theory to practice (2006) 2nd edn. Oxford: Oxford University Press. 164–206.
4 Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care (2003) 12:47–52.
5 Cook TD, Campbell DT. Quasi-experimentation. In: Design and analysis issues for field settings (1979) Boston: Houghton Mifflin Company.
6 Campbell DT, Riecken HW. Quasi-experimental designs. In: de Vaus D, editor. Research design. London: Sage Publications, 2006: 269–303.
7 Luepker RV, Murray DM, Jacobs DR, et al. Community education for cardiovascular disease prevention: Risk factor changes in the Minnesota Heart Health Program. Am J Public Health (1994) 84:1383–93.
8 Statistics Sweden. Statistics Database. Available from: http://www.scb.se. (20 January 2007, date accessed).
9 Sadigh Andersson S, Hökby A. Säkra Seniorer i Sundbyberg – Slutrapport med utvärdering. (Safe Seniors in Sundbyberg – Final report with evaluations) (2000) Stockholm: Stockholm County Council, Socialmedicin.
10 Hökby A, Sadigh S. Säkra Seniorer – En metodbok för att förebygga fallskador hos äldre. (Safe Seniors – Methods to prevent fall injuries among elderly) (2002) Stockholm: Stockholm County Council, Centrum för skadeprevention.
11 Swedish Rescue Services Agency. Äldres skador i Sverige (Elderly injuries in Sweden) (2003) Karlstad: Swedish Rescue Services Agency.
12 Snijders T, Bosker R. Multilevel analysis: an introduction to basic and advanced multilevel modeling (1999) London: Sage Publishing.
13 de Leon AP, Svanström L, Welander G, et al. Differences in child injury hospitalizations in Sweden: The use of time-trend analysis to compare various community injury-prevention approaches. Scand J Public Health (2007) 35:623–30.
14 Riva M, Gauvin L, Barnett TA. Toward the next generation of research into small area effects on health: a synthesis of multilevel investigations published since July 1998. J Epidemiol Community Health (2007) 61:853–61.
15 Ohlsson H, Merlo J. Understanding the effects of a decentralized budget on physicians’ compliance with guidelines for statin prescription – a multilevel methodological approach. BMC Health Serv Res (2007) 7:68.[CrossRef][Medline]
16 Diez-Roux AV. Multilevel analysis in public health research. Annu Rev Public Health (2000) 21:171–92.[CrossRef][Web of Science][Medline]
17 Cook TD, Campbell DT. The casual assumptions of quasi-experimental practice: The origins of quasi-experimental practice. In: de Vaus D, editor. Research design. London: Sage Publications, 2006: 305–43.
18 Johansson P, Sadigh S, Tillgren P, Rehnberg C. Non-pharmaceutical prevention of hip fractures – a cost-effectiveness analysis of a community based elderly safety promotion program in Sweden. Cost Eff Resour Alloc (2008) 6:11.[CrossRef][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||