Author Affiliations
Jason R Williams,
Casey Family Programs
Lisa Merkel-Holguin, Heather Allan
The Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, University of Colorado, School of Medicine, Department of Pediatrics
Erin J. Maher
Casey Family Programs
John Fluke, Dana Hollinshead
The Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, University of Colorado, School of Medicine, Department of Pediatrics
Keyword Suggestions (to be checked against MeSH)
implementation
family preservation
family support
child protective services
Bayesian Model Averaging
Family group conferencing
See specific formatting instructions. Documents will use APA, as well.
http://www.press.uchicago.edu/journals/jsswr/instruct.html?journal=jsswr
Factors Associated with Staff Perceptions of the Effectiveness of Family Group Conferences
Abstract
Implementation research highlights the importance of agency climate, worker characteristics, and attitudes in predicting buy-in and ultimately the success of evidence-based programming. Family Group Conferencing (FGC) is a relatively new practice that challenges the entrenched practice of child welfare agency service providers making decisions for children and families. Instead, FGCs position families as capable planners and decision-makers given proper support and resources. Prior research has shown social work staff differ in their opinions of FGC, but little research has connected these perspectives to staff and agency characteristics and, ultimately, to referrals and outcomes. In the absence of literature describing agency and staff characteristics associated with support of FGC in practice, we explore factors related to staff perceptions of effectiveness of FGC using Bayesian Model Averaging. The results underscore that worker attitudes about the effectiveness of FGC are themselves a product of attitudes towards families, type of work responsibility, and the perception of resources or services in the external environment. Among those who carry a caseload, worker perceptions of FGC effectiveness depend on perceptions of the availability and competence of local services to help families in need. Among those who do not carry a caseload, endorsement of families’ abilities to make plans to address their issues predicts endorsement of the effectiveness of FGC.
Introduction
Implementation of a program or intervention within an organization hinges in large part on issues of ownership, efficacy, and discretion. Those charged with ground-level implementation are more likely to do so with fidelity if they feel the practices are theirs and will work. A program injected from outside will be treated like an immune system treats a foreign body. Where there is room for interpretation and authority to choose from among several possible options, these ground-level implementers become essentially makers of policy, the results of which may be different than what was intended by leadership (Lipsky 1980; Križ & Skivenes, 2014). In this paper, we address child welfare agency staff attitudes and perceptions of family group conferences and the implications for implementation. Specifically, we examine what predicts worker perceptions that FGCs are an effective practice.
Family Group Conferencing in Child Welfare
Family group conferencing (FGC) is firmly rooted in New Zealand’s Children, Young Persons and their Families Act of 1989, which required participatory decision making between child welfare systems and the family groups of children who are abused and/or neglected. This reform acknowledged the marginalization and exclusion of family groups when their children come to the attention of the State, honored indigenous and traditional ways of resolving concerns, and recognized institutional racism within the public systems that are charged with the protection of children and support of families. It emerged from a rights-based framework that all children are entitled to having their family systems come together on their behalf to make decisions about their well-being. The legislation deconstructed the prevailing, entrenched, and mainstream practice of child welfare agency service providers making decisions for children and families, and replaced child welfare decision making with the FGC, which positions family groups to lead decisions, with the support of service providers. It challenged the predominant construct of professionals as experts and requires systems and their professionals to view families more holistically and as solution builders and leaders able to address challenges that confront their family systems (Ministerial Advisory Committee, 1988).
The framework or rationale which underpins FGC—best practice, procedure or legal right (Doolan, 2007), government innovation (Brown 2007), citizen engagement (Eigen Kracht, 2014), or social policy (Ministerial Advisory Committee, 1988)—provides important context to understanding the implementation of FGC and other family meeting models in the United States.
FGC is the among the most recognized models that fit under the family group decision making (FGDM) framework. In 2008, five core components of FGDM were defined by experts with a component related to follow-up added in 2013 (Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, 2013, p.2). These are consistent with FGC and include:
1. An independent coordinator, responsible for convening the family group meeting with agency personnel.
2. Child protection agency personnel who recognize the family group as the key decision-making partners and offer time and resources to convene the family group.
3. After initial presentations, family groups meet on their own, without statuatory authorities, to work through information they have been given and formulate responses and plans.
4. When agency concerns are adequately addressed, preference is given to the family group’s plan over any other possible plan.
5. Follow up processes after the FGDM meeting occurs until the intended outcomes are achieved, to ensure that the plan continues to be relevant, current, and achievable. FGDM is not a one- time event but an ongoing, active process.
6. Referring agencies support family groups by providing the necessary services and resources to implement the plan.
Given the genesis of FGC and that it is enshrined in legislation and policy in New Zealand, the influence of service providers’ perceptions about FGCs have not been a primary line of inquiry in that country. That is not the case in other countries, like the United States, where FGC may be considered a relatively new practice, but its implementation is often at the discretion of frontline social workers to refer families to participate.. The organizational bureaucracy, the procedural nature of child protection work, and the values individual systems place on these types of decision making forums may furthermore hamper or aid successful implementation (Brown, 2003).
Independent of the extent to which policy supports or directs the occurrence of family group conferences, understanding the attitudes of the child welfare workforce (including administrators, supervisors, and social workers) is a critical component to evaluating their use and, ultimately, effectiveness in child welfare. Fixsen and colleagues (2005) pointed out in their influential review of implementation research that practitioners’ skills, attitudes, training, and coaching are keys to successful implementation of evidence-based practices, though they noted that workforce issues have been understudied. McCrae and colleagues (2014) found agency size, stress levels, worker tenure, and job position to be related to buy-in for implementation of a child welfare practice model. There remains a dearth of literature examining child welfare professionals’ attitudes and perspectives about FGCs and other family meeting models. We briefly review these studies here.
A few studies have suggested that the child welfare workforce possesses a generally positive attitude about FGCs, particularly the underlying empowerment and strengths-based philosophies of the model (Huntsman, 2006; Crow and Marsh, 1997; LeCroy and Milligan 2003). Even so, child welfare staff members have often raised a number of concerns about FGCs, including: skepticism of plans formulated at FGCs compared to other child welfare meetings (Trotter, 1999), meeting length (Velen and Devine 2005), and that FGCs resulted in too much family empowerment (Huntsman, 2006). These perspectives may reflect ambiguity around the workforce’s attitudes toward the core purpose of FGC which is for the child welfare agency to facilitate and allow for a process for family groups to make decisions. None of these studies connected these perceptions to worker characteristics or to outputs or outcomes.
One study is particularly germane to this line of inquiry. In Sweden and England, Sundell, Vinnerljung, and Ryburn (2001) studied the relationship between social worker attitudes toward FGCs and referrals to the decision making process. In their survey of social work professionals with limited experience with FGCs, a majority endorsed the need for private family time, the capacity of extended families to make decisions, and the usefulness of FGC as a mechanism to resolve child maltreatment concerns. Despite this, the rate of referrals to FGCs was generally low in both countries. They also found that rates of referrals were higher when social workers: 1) participated in developing FGC as a service in Sweden; and 2) had a more positive attitude about FGCs. The age and previous work experience of the social worker did not impact referral rates to FGCs.
As part of a broader examination of FGC implementation, this study focuses on understanding child welfare professionals’ perspectives about the effectiveness of FGC. These perceptions are important to study since professionals’ buy-in impacts referrals (Sundell, Vinnerljung and Ryburn, 2001) and other aspects of quality implementation. Further, quality implementation with a high degree of family participation is an important ingredient in FGC practice and one which is hypothesized to result in the achievement of desired outcomes. While this broader understanding of the role of workforce attributes on FGC practice and outcomes is a longer-term goal, our purpose with this exploratory analysis was to understand whether there are attributes and attitudes measured at the worker level that influence the perceptions about the efficacy of FGC. We view this as a necessary precursor for understanding the broader impact on implementation and, ultimately, outcomes.
Model Development Under Uncertainty
While child welfare staff members’ perceptions of the effectiveness of FGC are likely to affect fidelity and success of implementation, few factors or combination of factors have been identified in the literature to be associated with such perceptions. Thus, data collection and analysis for this project began with few a priori hypotheses about individual factors and no hypothesis about the proper combination of factors to examine. This uncertainty about model composition or functional form is common in research examining new practices and has implications for the selection of an analytic approach.
Traditional inferential statistics is predicated on testing a single model against a null model. In practice, however, even when authors present a single model, it is likely that multiple models were actually considered. Several combinations of control variables may have been considered through some formal or informal screening process (Raftery, 1995; Tobias & Li, 2004). Similarly, alternate ways of measuring constructs or capturing (e.g., nonlinear) effects may be considered. The more such uncertainty enters into model building, the greater the number of potential models there are to evaluate. Existing screening and selection methods that rely on p-values or R2 or other goodness of fit statistics face numerous problems. These procedures do not inherently account for the multiple comparisons necessary to generate the final model, and the resulting “p-values based on a model selected from among a large set of possibilities no longer have the same interpretation as they did when only two models were ever considered” (Raftery, 1995, p. 112). Standard errors or confidence intervals resulting from these procedures are downwardly biased because they do not account for the uncertainty in model selection itself.
Given a relatively undeveloped literature from which to develop hypotheses on this topic and the faults associated with traditional testing of multiple models, we employ Bayesian Model Averaging (BMA) as our tool to weigh the evidence for each predictor and potential model. Bayesian methods focus on which model fits the data best, and express all uncertainty, including about the unknown parameters of interest, in terms of probability. We employed the BMA methodology of Raftery and colleagues (Raftery, 1995; Hoeting, Madigan, Raftery, & Volinksy, 1999) as implemented in the package BMA (Raftery, Hoeting, Volinsky, Painter, & Yeung, 2013) for the statistical program R (R Core Team, 2014). This procedure allows for competition even among highly correlated variables and facilitates the examination of different operational definitions of the same construct.
BMA produces a model posterior probability, which is the likelihood that the candidate model is the “correct” model that produced the data. It also produces inclusion probabilities that capture the likelihood that each predictor is in the correct model—essentially, a measure of predictor strength or the evidence supporting the inclusion or exclusion of that predictor (Hoeting et al., 1999). The “Averaging” part of the name refers to the production of an expected value of the coefficient weighted by the model posterior probability across all models considered. A probability weighted standard deviation is calculated similarly, and is analogous to the usual coefficient standard error except that it captures all uncertainty about the coefficient. While users can report and act upon the expected values for all considered predictors, BMA is often used to select a subset of predictors because the inclusion probabilities for some variables will be quite low, providing moderate or strong evidence against there being an effect. Many analysts select either the modal model (variables with inclusion probabilities above 0.50) or the best model (highest posterior probability), which are often one and the same.
In this paper, we utilized BMA within an exploratory model building and testing effort, combining the tool with more traditional inferential approaches as recommended by Gelman (2011). We applied this exploratory approach to an on-going child welfare evaluation project, identifying available variables potentially related to our outcome of interest—staff perceptions of the effectiveness of FGCs. The data source and potential predictors considered are described below.
Methods
Data Source
Data for these analyses were collected from child welfare agency staff in three jurisdictions west of the Mississippi participating in a 3-year evaluation of the use of family meetings in child welfare (referred to as States 1, 2, and 3). These sites were chosen due to their established FGDM practice; collectively the three sites had over 20 years of family meeting implementation experience. At the time of survey administration, the staff respondents worked in agencies that have been implementing FGCs and other family meetings for a minimum of 6 years and a maximum of 14 years. At the start of the evaluation training for the project, staff completed an on-line survey composed of demographic, position, and experience questions and scales pertaining to case skills, family meeting knowledge and attitudes, organizational culture and climate, service availability, and child safety vs. family preservation proclivity. In addition, as some new staff joined the agency, they were trained on the evaluation and completed the electronic survey at that time. Completing the survey took approximately 15 minutes. IRB approval for this study was received from a western state’s Institutional Review Board.