APPENDIXS1: MEASUREMENT DEVELOPMENT PROCEDURES1
Supplementary Material for Online Publication Only
Appendix S1: Elluam Tungiinun/Yupiucimta Asvairtuumallerkaa Measurement Development Procedures
James Allen
University of Minnesota Medical School,Duluth Campusand University of Alaska Fairbanks
David Henry
University of Illinois at Chicago
People Awakening Team
University of Alaska Fairbanks
Author Note
James Allen, Department of Biobehavioral Health &Population Sciences, University of Minnesota Medical School,Duluth Campusand Center for Alaska Native Health Research, University of Alaska Fairbanks. This research was funded by the National Institute of Alcohol Abuse and Alcoholism and the National Center for Minority Health Disparities [R21 AA016098-01, RO1AA11446; R21AA016098; R24 MD001626]. We also want to thank all of the People Awakening Team including participants, community co-researchers, our Coordinating Council and our project staff for their assistance in completing this research. The People Awakening (PA) Team includes the Yupiucimta Asvairtuumallerkaa Councils, the Ellangneq Councils, the Yup’ik Regional Coordinating Council, the Ellangneq Advisory Group, and the Ellangneq, Yupiucimta Asvairtuumallerkaa, and Cuqyun Project Staff. The Yupiucimta Asvairtuumallerkaa Councils included Sophie Agimuk, Harry Asuluk, Thomas Asuluk, T.J. Bentley, John Carl, Mary Carl, Emily Chagluk, James Charlie, Sr., Lizzie Chimiugak, Ruth Jimmie, Jolene John, Paul John, Simeon John, Aaron Moses, Phillip Moses, Harry Tulik, and Cecelia White. The Ellangneq Councils includes Catherine Agayar, Fred Augustine, Mary Augustine, Paula Ayunerak, Theresa Damian, Lawrence Edmund, Sr., Barbara Joe, Lucy Joseph, Joe Joseph, Placide Joseph, Zacheus Paul, Charlotte Phillp, Henry Phillip, Joe Phillip, Penny Alstrom, Fred Augustine, Mary Augustine, Paula Ayunerak, Theresa Damian, Shelby Edmund, Flora Patrick, Dennis Sheldon, Isidore Shelton, Catherine Agayar, Theresa Damian, Freddie Edmund, Shelby Edmund, Josie Edmund, and Flora Patrick. The Yup’ik Regional Coordinating Council includes Martha Simon, Moses Tulim, Ed Adams, Tammy Aguchak, Paula Ayunerak, Sebastian Cowboy, Lawrence Edmunds, Margaret Harpak, Charles Moses, Raymond Oney. The Ellangneq Advisory Group includes Walkie Charles, Richard Katz, Mary Sexton, Lisa Rey Thomas, Beti Thompson, and Edison Trickett. The Ellangneq Project Staff includes Debbie Alstrom, Carl Blackhurst, Rebekah Burkett, Diana Campbell, Arthur Chikigak, Gunnar Ebbesson, Aaron Fortner, John Gonzalez, Scarlett Hopkins, Nick Hubalik, Joseph Klejka, Charles Moses, Dora Nicholai, Eliza Orr, Marvin Paul, Michelle Dondanville, Jonghan Kim, Rebecca Koskela, Johanna Herron, and Stacy Rasmus. Cuqyun also acknowledges the invaluable contributions of James A. Walsh.
Correspondence concerning this article should be addressed to James Allen, Department of Biobehavioral Health andPopulation Sciences, University of Minnesota Medical School,Duluth Campus, 231 SMed, 1035 University Drive, Duluth, MN 55812-3031. Email:.
Abstract
Measurement development procedures for the Elluam Tungiinun and the Yupiucimta Asvairtuumallerkaa projects are described. An engaged approach in collaborative measurement development was an integral part of thisProject, which was a prevention research program that sought to promote protective factors against alcohol abuse and suicide in Yup’ik Alaska Native youth and families. The goals of measurement development were to identify culturally and contextually relevant constructs, then develop, adapt, and refine culturally and contextually appropriate measurement strategies and measures. Broader goals included construct elaboration and enhanced community engagement and ownership in the research process through the collaborative process. We found the process of culturally appropriate measurement development in a community based participatory research (CBPR) paradigm should include the communities in the selection of measurement constructs, methods of measurement, and development of measures.
Keywords: American Indian and Alaska Native; community based participatory research; measurement development; suicide, suicide prevention; alcohol; alcohol abuse prevention
Appendix S1: Elluam Tungiinun/Yupiucimta Asvairtuumallerkaa Measurement Development Procedures
We report here on the measurement development procedures for the Elluam Tungiinun (ET) and Yupiucimta Asvairtuumallerkaa(YA)projects. Our goal is to describe procedures within a community engaged perspective on measurement development for intervention research. We believe the approach is valuable as part of all community intervention work, and particularly important to work in culturally distinct contexts. The ET and YA projects are multilevel cultural interventions directed towards the prevention of suicide and alcohol abuse among rural Yup’ik Alaska Native youth (Allen, Mohatt, Beehler, and Rowe, this issue; Rasmus et al. this issue). The measures that are the focus of the development activity described here were developed to serve as outcome measures as part of the Cuqyun (measuring) project. Cuqyun wasa measurement development project that ultimately provided initial validity evidence for these outcome measures and tested the theoretical model for intervention (Allen, Mohatt, Fok, Henry, and Burkett, this issue). Gonzalez and Trickett (this issue) provide more detailed description on these process issues in collaborative measurement development with communities as partners, including some of the ethical, scientific, and process issues that emerge when doing this work, difficulties and dilemmas that can emerge in this process, and how partners can respond to them. The focus of the current paper is detailed description of more technical aspects associated with the process and content issues described in Gonzalez and Trickett that arose in the collaborative adaptation and development of culturally relevant measures.
The procedures we will describe respond to issues that arise in cross-cultural measurement development work (Allen & Walsh, 2000). These include such topics as construct equivalence, the functional equivalence of behavior expressed in items, the contextual relevance of item content, the indeterminacy of meaning arising with certain concepts, words, and even semantic constructions, such as within negative wording among reverse keyed items, and subtle issues in linguistic equivalence, including local variations in English language dialect and usage. They also include a number of additional culture specific issues arising in work with the Yup’ik cultural group. Some of these Yup’ik culture specific considerations include nuances in the features and styles of categorical thinking within the culture and important cultural rules that govern description and evaluation of experience, and in particular, the description and evaluation of the personal experience of others. Other culture specific issues include response style characteristics within the culture, where local interpretations of the semantic meaning of ratings interact with the proportional significances of anchor points in scaling created through response alternatives for items. A final overriding issue includes questions about the cultural inappropriateness of asking direct questions, numerous questions, repeated questioning.
We also seek to convey detailed information describing some aspects of the process of this work as these issues unfolded. In many ways, this process can be understood as involving a negotiation that occurred within a mode of discourse that was culturally patterned. We wish to convey the importance of our efforts in molding our work to these Yup’ik patterns of discourse, and their implications for work with other cultural groups. We also wish to convey that through this process of negotiation about measurement and measures, researcher and community partners navigated deeper waters to achieve some element of shared understanding, and underlying issues surfaced of important significance to the larger project. Through our experience, we came to appreciate how measurement development in an intervention project with indigenous, and by extension, with other culturally distinct groups, can serve as a flashpoint, where different values, worldviews, and meaning systems associated with two cultural knowledge systems converge. One knowledge system is embedded in the culture of the community. In the current case, this involved the cultures of two Yup’ik communities, which both possessed a deep and rich local indigenous knowledge system, along with additional more local knowledge and customs that were unique to each community. The other system is based in the culture of intervention science, and embraces the cannons of Western science. Much of the work we describe here involves a negotiation between these knowledge systems.
We believe cross-cultural measurement issues inevitably arise in community intervention research with nonwestern cultural groups. The resulting dilemmas provide a nexus wherein two at times discordant worldviews and epistemologies can converge through negotiated solutions. What becomes critical is the manner in which this negotiation occurs, and that these negotiations are open, transparent, and honest. For this reason, CBPR approaches applied to the measurement development process introduce a potential for new voices and perspectives that were formerly excluded to engage in this process. Inclusion of voice that is aligned with community knowledge and concerns can lessen the extent to which the research process can become colonizing.
Context and Measurement Development
Context largely determined our measurement development procedures, which can be understood through three phases of community engagement and measurement development. On reflection, both the quality and local responsiveness of the measures developed in parallel to the quality and responsiveness of the co-researcher relationship between the university and communities.
Typical to rural and much of ethnic minority assessment research (Okazaki & Sue, 1995), our research context is characterized by small populations. Further, in our review of the literature, no direct validity studies of measures of our variables of interest exist for the culturally distinct group with whom we work. Accordingly, our community intervention work involves small sample research requiring either development of new measures from the ground up or use of existing measures with unknown psychometric properties in the population of Yup’ik. Our efforts in measurement development within this context were guided by a single crucial rubric guiding one key principle. Our rubric was to listen to our community co-researchers and the local people doing the intervention work and to involve them in each key decision regarding measures. The key principal guiding us was to consistently implement procedures whenever they reduced error variance in the measures at whatever time we could empirically identify their source, and confirm our response reduced measurement error. Through doing this we sought to maximize the sensitivity of the measures by accounting for as much variation in our models as possible. This was particularly crucial in the use of statistical testing procedures for the assessment of our program outcomes, given their limited statistical power due to the small sample sizes we anticipated in our research in rural, remote, small communities where we faced significant logistical challenges to data collection.
One important implication of this rubric and principle led to the principled decision we made to continue to refine item content of the measures at baseline, as part of the methodology of using two baseline assessments in our two feasibility studies. The use of two baselines was initially selected to increase statistical power of our analyses with small samples. Though we made intensive efforts to devise measures with strong psychometric properties prior to baseline administration in these two feasibility studies, through years of cultural expert consultation, focus group work, and repeated pilot testing with follow-up interviewing, we continued to identify poorly functioning items at Time 1 of our feasibility studies. In response, we tested refinements of these poorly functioning items at Time 2.
Typically, changing item content in measures during any outcome study, even a study of feasibility, is methodologically questionable because of potential loss of measurement equivalence across the time points of the study. As a result, the meaning of any significant difference identified in the outcomes can become uncertain. However, we made any decision to revise an item empirically, rigorously basing it in the data, by making use of item response theory (IRT) analyses (Muraki, 1990; Rasch, 1966; Wilson, 2005) to guide decisions to revise any suboptimally functioning items. In most instances, this resulted in modifications to less than 5% of the baseline item pool. Only one measure, a scale of family functioning (the adapted Family Environment Scale, described below), required modification of approximately 15% of the baseline item pool at Time 1. Though the item response theory (IRT) literature has historically advised a the need for large samples to effectively use the approach (e.g., Embretson & Reise, 2000, p. 124), other authors (Wright & Tennant, 1996), and our experience with the technique here, found IRT approaches a useful diagnostic and decision tool even with small samples (n < 50) to determine if a revision or elimination of items in fact reduced measurement error. The approach allowed us to retain items with adequate discrimination indices and category thresholds suggesting reasonable coverage of the theoretical latent trait being measured. The consistent procedural use of this rigorous empirical approach to ongoing item refinement allowed us to revise select items after the first Time 1 baseline assessment, then examine these same characteristics at the two week re-testing at Time 2, with the knowledge that later time point assessments possessed less measurement error, and increased sensitivity in their assessment of the same underlying construct, and that any statistically significant difference that emerged would be meaningful. In a few cases, revising an item did not improve the fit of the item to the latent trait. In such instances we deleted the item.
Our approach also led to the development of two parallel sets of measures. One set were measures designed for theoretical model testing, the outcome of which is described in Allen, Mohatt, Fok, Henry, Burkett, and People Awakening Team (this issue), the second set was designed to maximize sensitivity to change, and was used in Mohatt, Allen, Fok, Henry, and People Awakening Team (this issue). The first set of measures designed to provide a test of the People Awakening (PA) protective factors model, which was the model guiding the intervention, aimed to comprehensively map the constructs defining the model using instruments with stable, internally consistent properties. This first set of measures was devised to provide a test of hypothesized causal relations between variables in the model; thereby providing empirical support for our emically derived cultural theory regarding protection from alcohol and suicide. In contrast, our measures of change were designed instead to be maximally sensitive to intervention effects as measures of outcome. This second parallel set of instruments tapped these same constructs from the protective factors model, and was a subset of the larger item pool. These were constructed as brief measures that were highly sensitive to change, and intended for repeated administration at baseline and at time points during and following intervention in a format minimizing the burden from lengthy assessments.
Phase I: Initial Proposed Measurement Model
The research group began the process of measurement development by identifying candidate measures and constructing a proposed measurement strategy for the intervention. The goal of this phase was to develop a measurement model to assess growth in protective factors as outcomes from the intervention we were simultaneously developing. Given the burden such a lengthy set of measures would impose on youth, we would be unable to individually assess all protective factors the intervention intended to address. In addition, the intervention was in development through a community directed process. Community members were directing the selection cultural activities and devising them as intervention activity modules and they believed would work. Measurement development needed to occur for pretest, while the development of the intervention was still occurring in this intervention development grant project. Through the year, after the activity was delivered to youth, university researchers would evaluate each activity for the specific protective factors from the People Awakening model (Allen, Mohatt, Howe, & Beehler, this issue) that were delivered in the module activity. Given the twin realities of intervention development work and participant burden of long questionnaires, we focused on development of key indicator measures as proximal variables at the level of family, individual and community of selected protective factors that were feasible to briefly measure, over exhaustive measurement of all the protective factors in the model. For ultimate variables, we attempted to measure consequences of alcohol use tailored to adolescents, who unlike adults, experience somewhat different consequences than adults (for example, limited alcohol related chronic disease health consequences emerge in adolescent alcohol abuse), and suicidal ideation, given suicide attempts are rare and have a low base rate, despite the divesting consequences when one single event occurs in a community. The aims of Phase I were to (1) adapt existing youth measures from the mainstream psychological literature for cultural appropriateness for use with Yup’ik youth, (2) adapt adult measures from the mainstream psychological literature for cultural and developmental appropriateness for use with Yup’ik youth (3) adapt adult measures developed for Yup’ik adults in previous research for use with Yup’ik youth. The approach began with identification of variables derived from our previous retrospective protective factors work with Alaska Native adults (Mohatt, et al., 2004; Allen et al., 2006). This measurement approach was guided by theory and a decade of discovery based and measurement development research described in Allen, Mohatt, Howe, & Beehler, this issue.