Critical Appraisal 1

Critical Appraisal of a Descriptive Study

Jill Radtke

University of Pittsburgh
Worksheet for Critical Appraisal of Descriptive (Correlation, Comparative) Design Study

Citation:

Palmeira, A.L., Teixeira, P.J., Branco, T.L., Martins, S.S., Minderico, C.S, Barata, J.T., et al.

(2007, April 20). Predicting short-term weight loss using four leading health behavior change theories. International Journal of Behavioral Nutrition and Physical Activity,4,

Article 14. Retrieved June 15, 2007, from

What type of article is this (e.g., research /data-based, clinical paper, review, editorial?)

Research/data-based

If this is a research article/data-base article, what makes it this type of article? Identify 2-3 characteristics of the article.

  1. End Product: The article presents original findings based on the conception of a study design and its implementation.
  2. Methodology: The article/study seeks to obtain data in a systematic fashion (e.g., the introduction’s literature search, the attempt to measure variables consistently and accurately in the methods section, the summation of findings in the results sections, etc.).
  3. Style: The article’s findings and design are presented in an objective and frank manner (also discussing the limitations) in order that the reader may judge, implement, question, and/or disregard the evidence.

State the research question posed by the authors:

How do key exercise and weight management psychosocial variables, derived from four health behavior change theories, predict weight change during a short-term behavioral obesity intervention?

What is my clinical question?

Can the same exercise and weight management psychosocial variables found in this study to predict short-term weight loss in women predict weight loss in women six weeks postpartum?

Using PICO, identify the following if applicable:

P (= population): Premenopausal women from a community who are greater than 24 years of age, not pregnant, free from major disease, and have a BMI greater than 24.9 kg/m2

I (=intervention): 15 weekly weight management meetings of 120 minutes each where the groups of 32-35 participants met with a mix of PhD and Master’s level exercise physiologists, as well as dieticians and psychologists who administered to the participants exercise, behavioral, and nutrition content. The content included didactic material (e.g., information on caloric content of food), motivational tools (e.g., giving pedometers), self-awareness instruments (e.g., food log, exercise log), and goal-setting (e.g., dietary and physical activity). The intervention was based on Social Cognitive Theory (SCT), but designed to include constructs from three other behavior change theories: Self-Determination Theory (SDT), Transtheoretical Model (TTM), and the Theory of Planned Behavior (TPB).

C (= comparison group): N/A

O (=outcome): Weight change (and the specific behavioral change theories and psychosocial constructs yielding the most predictive power for weight change)

APPRAISAL GUIDE / COMMENTS
I. Are the methods valid/trustworthy?
1. Was the research question clear? Was the need for the study adequately substantiated? Explain / The research question was stated clearly both in the abstract and the “background” section of the research report (not under a separate “purpose” section), “…the purpose of this study was to investigate the predictive value of changes in exercise and weight management related variables on weight change, in a sample of overweight and moderately obese women participating in a University-based weight management program,” (“background” section). However, it was less clearly delineated how the content of the intervention parlayed into improvement of the psychosocial variables and weight loss. Although some examples were given (e.g., “…the intervention had the underlying goals of improving autonomy…These are highly motivational factors that should have an effect on SDT constructs…”), it seems that the study did not include, or at least did not mention, how constructs from each behavioral change theory would be incorporated into the intervention. Thus, the reader remains unsure as to what type of weight loss intervention program (i.e., which variables/constructs should be incorporated and how) would yield the same predictive power of certain behavioral change theories, as well as improvements in psychosocial variables and weight loss as found by the study team.
The need for the study was adequately substantiated in several instances in the “background” section. The authors comment that obesity has become an epidemic in industrialized countries, yet there has been a great void in the integration of biological, psychosocial, and environmental solutions in weight management programs. The authors hold that several psychosocial variables that they incorporate into the present study (and the basis of the four behavioral change theories in the study) are widely believed to explain weight management in this integration context, yet are underserved in weight management literature. For example, in the background section, “Questions remain about which model or set of variables could better explain the outcomes of choice, which constructs may overlap, or if a set of variables from different theories could delineate the way to a new paradigm. Rothmam highlights this last aspect as a likely cause of some of the disappointing results for most studies of behavior change interventions conducted to date.”
2. What was the design of the study? How were the data collected (one time (cross-sectional) or repeated over time (longitudinal)? What were the limitations of the data collection methods? / The design of this study was descriptive correlational, and the data were collected in a prospective manner at two different time points, baseline and four months (the study is not longitudinal, per se, as it only collected at two time points during a short span of time).
There were several limitations to data collection. One such limitation may be the weighing procedure. The article states that a “standardized procedure” was utilized in the weighing process and cites a specific scale used. Further elaboration is not provided. However, we are unsure how much clothing participants wore during weighing, what time of day they were weighed (e.g., morning versus later in the day), after what activities they were weighed (e.g., after working out, after eating, etc.), how the scale was calibrated, if participants weighed themselves on the scale without the study team (i.e., self-report—this is not specified in the article), etc. Moreover, we are unsure whether the conditions for weighing were similar for all participants.
Another limitation in data collection was the self-report used in the psychosocial variable questionnaires (as stated in the article). Although the instruments were validated, there is always a subjective limitation in self-report. For example, a participant may mark feeling competent and autonomous on an instrument at the follow-up because they feel that this is what the researchers would like to see, whether the researchers are communicating this subconsciously (experimenter effect) or not (Hawthorne effect).
Another limitation in the data collection (but could be considered a design limitation), as stated in the article, was only measuring the participants twice: at baseline and at follow-up at four months. Perhaps there was more fluctuation between the baseline and follow-up. Perhaps, as the authors suggest is likely, the predictive power of the weight management and exercise variables in weight loss would be reversed if the data were collected more long-term (i.e., at 16 months).
Other limitations in this study are discussed under “sources of bias” in this paper, as they seemed to be more of design limitations than data collection limitations, per se.
3. Describe the sample. How was the sample selected (eligibility criteria)? How is the sample representative of the population? / At the beginning of the program, the sample consisted of 142 women with BMI’s 30.2 ± 3.7kg/m2 (overweight and obese) and ages 38.3 ± 5.8 years (the sample had 133 completers at the end of the program). The women were free of major disease, premenopausal, not pregnant, and recruited from a particular community. The sample was a purposive sample (due to the very specific eligibility criteria used for selection), recruited using advertisements in the community: newspaper ads, a website, email messages on listservs, and announcement flyers. These recruitment methods were presumed by the reader (myself) to list eligibility criteria, though this is not explicitly stated (perhaps directly stated on the poster or the interested party is directed to call a number for eligibility criteria). The eligibility criteria given by the authors is: premenopausal women greater than 24 years of age, not pregnant, free from major disease, and have a BMI greater than 24.9 kg/m2. It is unclear whether the participants self-selected (i.e., if they called and met criteria they were in the study) or were specifically chosen among all applicants who met eligibility criteria, although the article seems to assume self-selection.
This sample is somewhat representative of the population, in that it satisfies all the eligibility criteria. However, the age range is relatively tight between about 10 years of young to middle adulthood. There are no individuals greater than 45, nor any younger than 32, despite the population requirement only specifying greater than 24 years old. Thus, the age of the sample is not very representative of the population. Additionally, the BMI’s of the sample constituted overweight individuals to obese individuals. There were no participants who were severely or morbidly obese. Thus, BMI is not completely representative of the population (population requirement: BMI great than 24.9 kg/m2). Also, we are not given demographics of the sample. Therefore, we cannot be sure that the sample can be generalized or applied to different communities (populations) that differed from the sample significantly on these variables.
4. Describe the variables of interest. If a comparison study, on what variable(s) are the groups being compared? How were the groups similar? How were the groups different? If it is a correlation study, on what variables are associations being examined? Were there any confounding variables? / There were multiple variables of interest in this study. One variable was weight (at baseline and at 4 months; the average taken of two readings each time and rounded to the nearest 0.1 kg). There were also weight management psychosocial variables from each behavioral change theory (except SDT) measured as scores on instruments administered to the participants, including self-efficacy and outcome expectancy from SCT; self-efficacy, stages of change (SOC), and processes of change (POC), including both behavioral processes and cognitive processes, from TTM; and intentions, attitudes, subjective norms, and perceived behavioral control (PBC) from TPB. There were exercise psychosocial variables also from each behavioral change theory measured as scores on instruments administered to the participants, including self-efficacy, perceived barriers, and social support from SCT; self-efficacy, SOC, and POC, including both behavioral and cognitive processes from TTM; intentions, attitudes, subjective norms, and PBC from TPB; and interest/enjoyment, perceived competence, importance/effort, pressure/tension, and intrinsic motivation from SDT. Typically, the higher the score on the instruments for the exercise and weight management psychosocial variables indicated greater embodiment of that variable by the participant. Additionally, the four behavioral change theories (SCT, SDT, TTM, and TPB) served as variables of interest in the study. Time was also a variable of interest (from baseline measures to four months). Generally, weight, psychosocial variables, and the behavior change theories acted as dependent variables, while time served as the independent variable.
This was a correlation study, and several associations among these variables were examined. First, weight was examined for its association with time (i.e., weight change from baseline to four months). The exercise and weight management psychosocial variables were also each individually studied for their association with time (change from baseline to four months). Then weight change was correlated with baseline exercise and weight management psychosocial variables in order to determine any possible moderator variables. Weight change was also correlated with four-month change in exercise and weight management psychosocial variables. Finally, the correlation between weight change and the four different behavioral change theories (SCT, SDT, TTM, and TPB) was examined by entering the psychosocial variable scores present in each theory into separate regression models for each theory.
The study did not note any confounding variables.
5. Was the sample size large enough to detect a statistically significant association or difference? Was a power analysis performed? / Yes, the sample size was large enough to detect statistically significant associations with 142 subjects to start and 133 completers. It was not mentioned that a power analysis had been performed.
6. Were there any potential sources of bias? (Differences between groups not accounted for in the analysis, drop-outs, discounting outcomes, funding agency, etc.) / There were many potential sources of bias in this study. One such bias involves the method of recruitment: through advertisements in the newspaper, on a website, announcement flyers, and email messages on listservs in one community. This is a sampling bias, in that study participants appear to self-select for a purposive sample. These study participants, due to their presence in one particular community and willingness to volunteer for the study (i.e., they likely desire to lose weight), may differ from the population in several fundamental aspects. This limits the generalizability of the study findings.
Another source of bias may be that the SDT was not accounted for in the weight management psychosocial variables. The authors state that this is due to the fact that a valid Portuguese instrument had not been validated for the constructs in this theory with weight management. However, it is plausible that psychosocial variables in this theory still affect weight change (even though they are not tested).
A source of bias also possibly existed in questionable construct validity. In fact, the article states that some variables were measured with less than ideal instruments, such as outcomes expectancies. The article does not tell us the reliability and validity of the instruments used to measure the psychosocial variables, and we are left to look up the instruments on our own or just accept the authors’ judgment.
Also it is mentioned that there was a 6.3% attrition rate from baseline to four months, with 142 women starting the study and 133 completing it. This is not an especially high attrition rate, but if the subjects dropping out differed in some fundamental way from those staying in the study, then we would have attrition bias (i.e., our results would not reflect the population of interest, but those individuals that had had certain characteristics that allowed or motivated them to complete the study). Because the characteristics of those dropping out (or those staying in) were not elucidated, and the point in the study when the drop-out occurred was not discussed, the reader is unable to make an informed decision as to whether attrition bias existed. Bias could also exist in the relatively small sample size in the study, which affects external validity.
Another potential source of bias is testing effects. The same instruments (questionnaires) were apparently given at baseline and at four months. It is entirely feasible that the subjects became sensitized to the material on the instruments at baseline, and then answered the same questions differently at four months due to the pre-test rather than an actual intervention effect.
Bias could also result from maturation effects. The subjects could have changed from baseline to four months, regardless of the intervention. For example, as women move into middle age, their metabolism slows and weight gain occurs more easily. This weight gain (or lack of weight loss) would have little to do with the intervention.
Validity may have been affected in the study by the Hawthorne effect (i.e., the subject answered the instruments in a certain way or lost more weight because they knew they were in a weight loss study). Experimenter effects could have also been present if the subjects perceived, for example, that the researchers wanted them to lose weight or answer the instruments indicating that their self-efficacy was improving.
Also, the study (as mentioned in “limitations”) did not include a control group. This is a source of bias—if a control group had been present and exposed to the possible Hawthorne effect, experimenter effects, and had differed as much as the intervention group on fundamental aspects (such as race, income, etc.), we could say that the intervention was likely the cause of the changes in weight and psychosocial variables. However, one has to also keep in mind that this is a correlation study and it did not claim causation.
Finally, a source of bias could exist in the outcome that the weight management psychosocial variables better explained weight change from baseline to four months as opposed to the exercise psychosocial variables. In fact, the authors note that in a similar study that was carried to 16 months, exercise psychosocial variables were better correlates of weight loss. If this study had been extended, perhaps they would have also found exercise psychosocial variables as more powerful predictors of weight change.