Critical Appraisal 1

Critical Appraisal of an Experimental Study

Jill Radtke

University of Pittsburgh
Worksheet for Critical Appraisal of Experimental Design Study

Citation:

Littrell, K.H., Hilligoss, N.M., Kirshner, C.D., Petty, R.G., & Johnson, C.G. (2003). The effects of

an educational intervention on antipsychotic-induced weight gain. Journal of Nursing

Scholarship, 35(3), 237-241.

What type of article is this (e.g., research/data-based, clinical paper, review, editorial)?

Research/data-based

If this is a research/data-based article, what makes it this type of article? Identify 2-3 key characteristics of the article.

  1. Intervention: There is an intervention (weekly psychoeducation classes for nutrition/exercise/health lifestyle) for an experimental group, and results are compared to a control group.
  2. Methodology: The article/study seeks to obtain data in a systematic fashion (e.g., the “Background” section’s literature search, the attempt to randomly assign subjects and reliably and validly measure variables in the “Procedure” section, the summation of findings in the “Findings” section, etc.).
  3. Style: The article’s findings and design are presented in an objective and frank manner (also discussing the limitations) in order that the reader may judge, implement, question, and/or disregard the evidence.

State the research question posed by the authors:

What is the effect of an educational intervention, incorporating content on nutrition, exercise, and living a healthy lifestyle specifically tailored for schizophrenic individuals (“Solutions of Wellness”), on antipsychotic-induced weight gain among patients with schizophrenia?

What is my clinical question?

To what extent does social support in the group setting affect weight maintenance for individuals with schizophrenia taking antipsychotics, or does it affect it at all? For example, to test this, we might compare weight change between 3 groups of schizophrenic individuals taking antipsychotics: those who receive an educational intervention in a group setting, those who participate in a peer weight maintenance support group (without education), and those that do not receive any intervention.

Using PICO, identify the following, if applicable:

P (=population): Male and female individuals, 18 years or older, with a DSM-IV diagnosis of schizophrenia or schizoaffective disorder, taking conventional antipsychotics for at least 3 months immediately prior to study commencement, and defined as compliant with antipsychotic drug treatment as prescribed

I (=intervention): Weekly (for 16 weeks) 1-hour psychoeducation classes that were based on the “Solutions of Wellness” modules and conducted by the same master’s level clinician (trained in teaching the modules). The program is designed specifically for individuals with schizophrenia, and consisted of two modules: one based on nutrition and healthy living and one based on exercise. Material was delivered in a variety of formats (reading aloud, written exercises, etc.) with different group configurations (small groups, individual work, etc.). Additionally, weekly reminder letters and encouragement from staff to attend meetings were given to the intervention group.

C (=comparison group): Standard care group, consisting of 21 males and 14 females meeting the inclusion criteria for the study, as defined in “population.” This group, unlike the intervention group, did not receive the intervention of the psychoeducation classes. However, like the intervention group, the standard care group did receive stepped-initiation conversion of olanzapine at entry into the study (then adjusted to meet patient individual needs and side effects) and had BMI measured at baseline, four months, and 6 months (end of study); weight measured monthly; and eating and exercise routines asked about during the weight measurements.

O (=outcome): Weight change differences between the intervention and standard care groups

GUIDE / COMMENTS
I. Are the results valid/trustworthy?
1. Was the research question clear?
2. a) Was the need for the study adequately substantiated?
b) Explain / The research question was stated clearly, in the form of a statement, in the abstract under “purpose,”…”To assess the effect of an educational intervention on antipsychotic-induced weight gain among patients with schizophrenia.” It was stated again in the last paragraph of “background” (p.238).
The need for the study was adequately substantiated in the “background” section, as well as in the introduction. In the introduction, we are told of the obesity epidemic in this country and the associated morbidity. Further, individuals with schizophrenia are at an added risk for overweight and obesity because atypical and conventional antipsychotics are commonly known to associate with weight gain. Additionally, we are told that weight gain while on antipsychotics occurs mostly during the beginning of treatment initiation. However, the authors of the article only cite one study supporting this statement, and they use it as the basis of targeting the educational intervention at the initiation of olanzapine. The authors go on to describe the non-pharmacological studies that have been done on weight loss for individuals with schizophrenia. They cite the first study occurring fairly recently in 1999, and although there has been a recent increase in studies on this subject, there still exists a void. Thus, the need for more studies about safe non-pharmacological weight loss for individuals with schizophrenia is justified. Additionally, the authors describe a small pilot study that they conducted using an educational intervention on 12 schizophrenic patients. The participants gained less weight than those who did not participate, a promising result for the prevention of overweight in schizophrenic individuals taking antipsychotics. Thus, the need for the study is adequately justified for the larger sample here.
3. a) What was the experimental design?
b. Was it an appropriate fit for the research question?
c) Explain / The experimental design was quasi-experimental. I believe this could be an appropriate fit for the research question, as the authors describe several studies in which educational interventions in schizophrenic individuals taking antipsychotics have been shown effective in preventing weight gain. Additionally, the authors have already conducted a small pilot study using the same variables (weight gain, psychoeducation class). Thus, it appears that we know enough about the actual phenomenon to warrant an experimental design (quasi-experimental because the participants were not selected randomly from the population). If we did not know much about these variables and their relationship, we might start with a descriptive or correlational design.
However, the authors do not go into detail or cite much literature about several topics. For example, why does weight gain occur at the beginning of a trial of an antipsychotic. Nor do they cite the effectiveness of the particular educational program used in the study for individuals with schizophrenia. Perhaps there is adequate literature in these areas. If not, then a descriptive, correlational, or qualitative study might be more appropriate.
4. a) Describe the sample.
b) How was the sample selected?
c) How is the sample representative of the population?
d) Is the sample similar to your clinical population? / The sample consisted of a total of 70 outpatients with a diagnosis of schizophrenia or schizoaffective disorder, 35 in the intervention group and 35 in the standard care group. All individuals in the sample met the population eligibility criteria, including having taken conventional antipsychotics for at least 3 months prior to study enrollment and defined as being “compliant” with antipsychotics as prescribed. In terms of education, all had a high school diploma or GED equivalent or higher. There were 22 males and 13 females in the intervention group and 21 males and 14 females in the standard care group. There were 52 Caucasians (26 in each group) and 18 African Americans (9 in each group), and 54 individuals diagnosed with schizophrenia (28 in the intervention group, 26 in the standard care group) and 16 diagnosed with schizoaffective disorder (7 in the intervention group and 9 in the standard care group). Age in the intervention group was in the range of 33.66 ± 9.23 years, and age in the standard care group was in the range of 34.51 ± 9.99 years. Age at onset of either schizophrenia or schizoaffective disorder was 19.31 ± 3.06 years in the intervention group and 20.91 ± 3.86 years. Weight and BMI, respectively, was 178.84 ± 33.49 lbs and 26.26 ± 3.68 kg/m2 for the intervention group and 180.63 ± 41.26 lbs and 27.17 ± 5.79 kg/m2 for the standard care group. There were 15 individuals that had a “normal” BMI and 20 that had an overweight/obese BMI in the intervention group. In the standard care group, there were 12 individuals considered to have a “normal” BMI and 23 with an overweight/obese BMI. Individuals in the intervention group were taking a olanzapine dose of 16.64 ± 4.2 mg, and individuals in the standard care group were taking 16.29 ± 4.08 mg. It was not stated when in the study this dosage was measured. Additionally, 27 of the participants were taking haloperidol at the start of the study and 9 were taking decanoate formulations. It was not mentioned if these medications were continued throughout the study in these participants or which groups these participants were in (standard care or intervention). Also, the participants were allowed to take concomitant medications for breakthrough symptoms, and these included 6 individuals taking lithium, 3 taking valproate, and 13 taking SSRI’s. Again, it was not mentioned in which groups these individuals fell.
The sample was purposive, the participants referred from local community mental health centers and private practice psychiatrists. This purposive sampling likely contributed to the study being quasi-experimental rather then truly experimental. The entire sample of 70 individuals were then randomly assigned to either the intervention or standard care groups.
The sample seems to be representative of the population chosen for the study to the degree that all participants met eligibility criteria as defined under “Population” previously in this critique. It seems marginally representative of the clinical population of those with schizophrenia and schizoaffective disorder. For example, there were many more men than women in each group. Some literature suggests that men and women develop schizophrenia equally, but men younger and possibly more severely than women (Gurege, 1991). Schizoaffective disorder is thought to be more common in women (Mental Health America, 2007). The eligibility criteria for the study specifies men or women. So, ideally, we might like to have more equal numbers of men and women in the study. Additionally, the ages in each group tended to cluster around middle 30’s, when the population was defined as individuals older than 18. Schizophrenia is thought to develop usually in the late teens or early 20’s (Gurege, 1991). So, the ages of the sample are not very respresentative of the population defined or the clinical population. Additionally, the population criteria did not define a certain race. However, our sample consists of only African Americans and Caucasians. There are also many more Caucasians than African Americans in our sample. In the clinical population, African Americans are three times more likely to be diagnosed with schizophrenia than whites (Bresnahan, Begg, Brown, Schaefer, Sohler, Insel, Vella, &
Susser, 2007). Thus, the race of our sample is not very representative of the population or clinical population.
5.a) Was a power analysis performed?
b) Was the sample size large enough to statistically detect a clinically important difference? (With small N, greater chance of showing no statistical difference even when one exists; large N can show small statistical differences which may not have clinical relevance.) / It was not mentioned that a power analysis was performed.
Although the sample size was small (n=70), and each group had only 35 subjects, significant differences were detected between the groups’ weight change at 4 and 6 months. Because it is more difficult to find statistical significance with smaller samples, finding significance here indicates that results may be very telling of the actual patterns of weight maintenance for schizophrenic individuals taking antipsychotics.
6.a) How were study subjects assigned to treatment groups?
b) Were groups comparable at baseline? / It was stated that the 70 participants who entered the study were randomly assigned to either the intervention or standard care group. However, it was not stated how randomization was done (e.g., with a table of random numbers, etc.)
The groups appeared to be comparable at baseline in the variables measured. The research article states that the two groups did not differ significantly in “sex, diagnosis, race, age, age at onset, weight, BMI, BMI category, or olanzapine dose,” (p. 239). However, in the table provided on p.239, it appears that “age at onset” was marginally significant for a difference between the standard care and intervention groups (p= .06). Additionally, it is mentioned in the limitations that some participants were taking different medications (not clear if these medications were in treatment of schizophrenia or not), and these were not controlled for between the two groups. Additionally, the limitations state the groups were not matched on other extraneous variables, such as medication history and familial trends. Thus, we cannot be sure that the two groups are comparable at baseline; each might have differed significantly in these aspects from one other, which may have altered our results.
7. Describe the intervention or experimental manipulation / The intervention was weekly (for 16 weeks) 1-hour psychoeducation classes that were based on the “Solutions of Wellness” modules and conducted by the same master’s level clinician (trained in teaching the modules). The program is designed specifically for individuals with schizophrenia, and consisted of two modules: one based on nutrition and healthy living and one based on exercise. Material was delivered in a variety of formats (reading aloud, written exercises, etc.) with different group configurations (small groups, individual work, etc.). Additionally, weekly reminder letters and encouragement from staff to attend meetings were given to the intervention group.
All participants were started on olanzapine at the start of the study, and dosage adjusted according to individual needs.
8. To what extent was the experimental condition delivered and followed (adhered to)? / The article states that there was a 92% compliance rate for the intervention group attending the psychoeducation classes. This figure indicates excellent adherence to the intervention. However, we are not given information as to the subjects’ actual participation in these classes, which may be difficult to measure, but likely a more significant measure when assessing the efficacy of the intervention with weight change. Additionally, we are not given information regarding if or how compliance with the olanzapine dosing was determined. If, for example, subjects skipped doses more often in the intervention group, we may see less weight gain in this group (not because of the intervention, but because of decreased dosing of the antipsychotic).
9. a) Describe the reliability and validity of the measures.
b Are the measures appropriate for the population or the variable being studied?
c) If not, why?
d) What were the outcome measures? / N/A, manual-defined program.
10. a) Were all subjects accounted for in the analysis? (Drop-outs, discounting outcomes, other sources of bias that affect results of test and hence interpretation.)
b) Was Intention-to-Treat approach used? / The attrition rate in the study was not discussed. It actually appears (from Table 3, p.239) that there were no drop-outs in the study (that the study began and ended with the same 70 participants).
No particular subjects were noted in the article for biasing results (i.e., no outliers noted). However, racial and sex differences were described. It was found that men gained more weight than women, only in the standard care group, from baseline to both 4 and 6 months. This could have contributed to the significant weight gain found in these areas. Additionally, it was found that African Americans gained more weight than the Caucasians, but this was not statistically significant. Perhaps African Americans really do have a statistically significant difference in weight gain, but this was not detected due to the small number of African Americans actually in the study.
Intention to treat was not discussed, as attrition was not discussed in the study.
11. a) Were the correct statistical tests used according to the type of data analyzed and the questions asked? / I believe that the correct statistical tests were used in this case. Fisher’s exact test was used to detect statistically significant differences between categorical variables in the study, such as difference in the number of individuals in each group that were African American or had schizophrenia. This was appropriate, because Fisher’s test is used for nominal and ordinal data and our categorical variables here are nominal. Additionally, Fisher’s test is useful for small sample sizes, because it doesn’t approximate. Here, we have a relatively small sample size. Thus, using this test we are able to see whether the groups varied very much from each other in terms of certain characteristics.
T-tests were appropriate to use to detect statistically significant differences for the continuous variables (e.g., olanzapine dose, weight change, etc.) between each group and between two time periods, because t-tests ascertain whether means are significantly different between two variables. However, if the authors of the study had wanted to compare weight at baseline, 4 months, and 6 months, a more appropriate test would have been something like ANOVA.