Example Evaluations of Theory-Based Interventions

The following two examples illustrate the concepts presented in this chapter. Each example will use one aspect of this chapter's key concepts.


Example #1:
The Mediterranean Lifestyle Program/Trial (MLP) was an intervention study designed to assess the impact of a lifestyle management program on women with type 2 diabetes. The program was designed with Social Cognitive Theory, Social Ecologic Theory and Goal-Systems Theory in mind and evaluated using mediation analysis, and the RE-AIM framework.

Evaluation Methodology

The investigators used a randomized controlled design to evaluate the effectiveness of the program. Inclusion criteria for the study were: women, current diagnosis of Type 2 diabetes for at least 6 months, post-menopause, under 75 years old, currently living independently, with a telephone and persons that can read English.

The evaluation team responsible for MLP also decided to utilize the RE-AIM framework for their evaluation. Research team members wanted to explore the ways in which MLP could be replicated in the future, and RE-AIM offers a look at the potential strengths and weaknesses of the program with a particular eye for replication and future dissemination.

MLP and the RE-AIM Framework

Reach:
The evaluation team assessed the extent to which the program reached the intended audiences (participants, clinics).

Evaluation Questions & Source of Data:

  • Percentage of eligible patients who participated in the program: collected by reports of recruitment activities
  • Examination of representativeness of participants: collected by a demographic survey of participants
  • 51% of eligible patients participated in the program
  • The sample was diverse and the only significant difference between the two conditions was receipt of hormone replacement therapy, so this covariate was controlled for in analysis

Effectiveness/Efficacy:
The evaluation team conducted a comprehensive process and outcomes evaluation involving numerous data sources at different points of time during the intervention and follow-up periods.

Evaluation Questions & Sources of Data:

  • Did the program positively impact desired behaviors?

Results:

  • There were significant treatment effects on all of the desired outcomes (physical activity, percentage of calories from fat, and HbA1c levels).

Adoption:
The research team assessed the reach of the program through exploration of participating clinics and patients.

Evaluation Questions & Sources of Data:

  • Participation rate among eligible providers (e.g. clinics): collected as the percentage of clinics approached that agreed to participate in the program
  • To what extent are the participants representative of the participating clinics: collected through demographic surveys of participants as compared to the population of patients at each clinic

Results:

  • 70% of physicians approached agreed to participate in the program.
  • Participants were representative of clinics' patients; however, MLP participants tended to have fewer years taking medications and fewer years diagnosed with diabetes than clinic patients not participating.

Maintenance:
Both individual- and setting-level maintenance issues were examined during the course of this evaluation.

Evaluation Questions & Sources of Data:

  • Maintenance at 12- and 24-month follow-up periods

Results:

  • MLP participants showed significant 12- and 24-month improvements in all targeted lifestyle behaviors with one exception (there were too few smokers to analyze tobacco use effects), and in psychosocial measures of use of supportive resources, problem solving, self-efficacy, and quality of life.

MLP and Mediation Analysis

This logic model depicts the theoretical underpinnings of MLP including theorized mediators. The authors examined perceived support, social embeddedness, and social-ecological resources as potential mediators in the relationship between MLP program participation and desired outcomes.


examples

Measures:

  • Perceived support was measured using a 19-item social support survey, previously developed for use with patients with chronic disease. The survey had a high internal consistency in the MLP study (0.96).
  • Social embeddedness was measured using the Social Network Index, a 12 item measure of social relationships and the subject's frequency of contact with them. The Index has moderate 6-month test-retest reliability (0.42).
  • Social Ecological Resources were measured using the brief Chronic Illness Resource Survey (CIRS), a 22-item scale assessing use of resources from different sources including family, friends and the community. The Scale was previously validated and showed high internal consistency reliability (0.90) and high 1-month test-retest stability (0.83).

Results:

  • Receipt of MLP was associated with significant changes in CIRS and Social Network Index but not Social Support.

Citations

Efficacy/Effectiveness Trial Results:
Toobert, DJ, Glasgow, RE at al (2003). Biologic and quality-of-life outcomes from the Mediterranean Lifestyle Program. Diabetes Care, 26(8): 2288-2293
Toobert, DJ & Strycker, LA (2005). Effects of the Mediterranean Lifestyle Program on multiple risk behaviors and psychosocial outcomes among women at risk for heart disease. Ann Behav Med, 29(2): 128-137
Toobert, DJ, Glasgow, RE et al (2007). Long-term effects of the Mediterranean lifestyle program: a randomized clinical trial for postmenopausal women with type 2 diabetes. Intl J of Beh Nutrition and Physical Activity (online)

Mediation Analysis:
Barrera, M, Toobert DJ et al (2006). Social support and social-ecological resources as mediators of lifestyle intervention effects for Type 2 Diabetes. J Health Psych, 11(3):483-95

RE-AIM Evaluation Methods and Results:
Toobert, DJ, Strycker, LA et al. (2002). Enhancing support for health behavior change among women at risk for heart disease: the Mediterranean Lifestyle Trial. Health Education Research, 17(5): 574-585
Toobert, DJ, Strycker, LA et al (2002). If you build it, will they come? Reach and Adoption associated with a comprehensive lifestyle management program for women with type 2 diabetes. Patient Ed and Counseling, 48: 99-105

Economic Evaluation:
Ritzwoller, D, Toobert, DJ et al (2006). Economic analysis of the Mediterranean Lifestyle Program for postmenopausal women with diabetes. The Diabetes Educator, 32(5): 761-9



Example #2: Using RE-AIM to Evaluate Policy Change
RE-AIM can be used to evaluate a range of health behavior interventions. While the text explains numerous community-based interventions, other researchers have suggested that the RE-AIM framework could be successfully applied to evaluations of policy-level interventions. Because RE-AIM focuses on replication and dissemination of the intervention, the goal of the policy evaluation would be to expand the policy to broader, wider audiences either nationally or internationally.

A good reference for how to apply the RE-AIM framework to the evaluation of health policies is:
Jilcott, S, Ammerman, A, et. al. (2007). Applying the RE-AIM Framework to assess the public health impact of policy change. Ann Beh Med, 34(2): 105-114

Now let's see how you could apply the RE-AIM framework to the evaluation of a smoke-free indoor air legislation recently passed in the state of Florida. It's a good idea to begin any evaluation with a logic model depicting the program's activities (in this case, the aspects of the policy) and the connection to behavior change.


RE-AIM and Policy Change
RE-AIM Dimension Key Issues:
Reach Are teens in Florida impacted by the law?

Are teens most at risk for smoking initiation?
Efficacy/Effectiveness Did the policy have an effect on teen smoking initiation as envisioned?
Adoption What is the process for phasing-in the policy?
Implementation What is the fiscal impact of the policy for both establishments and the government (enforcement)?

How have establishments supported the law?
Maintenance What are the long-term effects of the policy for both establishments and teens?