Parenting Practices among Depressed Mothers in the Child Welfare System
Patricia L. Kohl, Jacqueline Njeri Kagotho, and David Dixon
The purpose of this study was to analyze a nationally representative sample of families referred to Child Protective Services (CPS) agencies, the National Survey of Child and Adolescent Weil-Being, to examine the association between maternal depression and parenting practices over a 36-month follow-up period.Three hypotheses were tested: (1) Depressed mothers are' more likely to demonstrate harsh parenting than are nondepressed mothers; (2) depressed mothers are more likely to demonstrate neglectful parenting than are nondepressed mothers; and (3) depressed mothers are more likely to demonstrate emotional maltreatment than are nondepressed mothers. The interaction between depression and time was also analyzed for each parenting practice to determine how changes in maternal depression affected changes in parenting. The sample for this study was 1,536 mother-child dyads in which the child was age three to 10 years and remained in the home after a CPS investigation. Depression remained high across time points and was associated with increased risk of emotional maltreatment and neglect over a 36-inonth period. In addition, self-reported emotional maltreatment remained high across time points. Implications of this work are the needs for better identification of mental health needs for mothers entering the child welfare system and parent training to specifically address positive parenting.
KEY WORDS: child welfare; maternal depression; National Survey of Child and Adolescent Well-Being; parenting
M aternal depression, a critical public health concern, is prevalent among mothers referred to Child Protective
Services (CPS) agencies. In fact, nearly a quarter of adults entering the child welfare system meet the diagnostic criteria for a major depressive episode in the preceding 12 months (U.S. Department of Health and Human Services, Administration on Children.Youth and Families [HHS, ACYF], 2005), compared with only 7% of adults in the general population (Kessler, Chiu, Demier, & Walters, 2005). Furthermore, w ômen have an increased likelihood of experiencing depression compared with men (Kessler et al., 2003), and women exposed to a high number of chronic Stressors—as many women referred to CPS agencies are—are three times more likely than women with less exposure to Stressors to experience maternal depression (Orr,James, Burns, & Thompson, 1989). Given that women comprise the vast majority of primary caregivers among the child welfare population (HHS, ACYF, 2005), it is important to understand how maternal depression affects outcomes after a CPS referral.
The high rate of maternal depression in the child welfare system is a concern given its influence on parenting practices. Symptoms of depression may impede a woman's capacity to provide care for her children, placing her at risk to engage in neglectful parenting practices. For instance, depressed mothers may lack sensitivity to their children's physical and emotional needs (Campbell et al., 2004; Trapolini, Ungerer,&McMahon,2008) ormay be unavailable or otherwise unresponsive to their children (Cum- mings & Cicchetti, 1993).
The literature also demonstrates that maternal depression is related to a higher risk of other harmful parenting behaviors, including emotional maltreatment and harsh parenting. Depressed moth- ers are more likely than are nondepressed mothers to have conflict-related interactions with their children, including feeling aggravated with the child, yelling at the child, and spanking the child (Lyons-Ruth,Wolfe, Lyubchik, & Steingard, 2o[)2). Maternal depression increases the likelihood of corporal punishment toward children (Chung, Mc- CoUum, Elo, Lee, & Culhane, 2004; Shin & Stein,
CCCCode: 1070-5309/11 $3,00 62011 National Association of Social Workers 215
2008). Using meta-analysis techniques to examine reported findings about maternal depression and parenting behavior across 46 studies, Lovejoy, Crac- zyk, O'Hare, and Neunian (2000) found a moderate effect size {d = .40) for negative parenting behav- iors (for example, coercive, hostile, or threatening gestures), indicating a fairly strong relation between depression and harmful parenting.
Additional studies have shown that maternal depression places children at risk of abuse. Longi- tudinal analysis of the National Institute of Mental Health's Epidemiologie Catchment Area Survey {N = 7,103) revealed that, among cases with no reported abuse at baseline, depressed respondents (parents) were more than three times as likely to report physical abuse toward their child at wave 2 than were nondepressed parents (ChafEn, Kelleher, & Hollenberg, 1996). Finally, symptoms of mental illness, including depression, were associated with higher scores on the Child Abuse Potential In- ventory in the Women, Co-occurring Disorders and Violence Study, indicating an elevated risk of future abuse (N = 371) (Rinehart et al., 2005). In summary, these studies have clearly demonstrated that maternal depression adversely affects parent- ing among community-based samples. The extent to which maternal depression influences parent- ing practices among one of the country's most vulnerable populations—mother and child dyads referred to CPS agencies for allegations of abuse or neglect—is not yet known.
The aim of child welfare intervention is to improve the safety and well-being of children, a goal that is adversely affected by maternal depres- sion. There is currently a dearth of information on the association of depression and changes in parenting behaviors after referral to CPS agencies. Unanswered questions remain. Do the parenting behaviors of depressed mothers improve at similar or different rates than do those of nondepressed mothers? Does a change in depression status affect parenting behaviors? Underst:anding which, if any, parenting behaviors remain a risk will help child welfare professionals better target limited resources to more accurately address specific parenting be- haviors. Furthermore, this understanding could be used to inform policy and practice decisions about the mental health service needs of mothers referred to CPS agencies.
The objective of this study was to analyze a land- mark nationally representative sample of children
and families referred to CPS agencies, the National Survey of Child and Adolescent Well-Being (NS- CAW),to examine the association between maternal depression and changes in self-reported parenting practices over a 36-month period after referral to CPS agencies. Specifically, these three hypotheses were tested:
1. On average, depressed mothers would be more likely to demonstrate harsh parenting over a 36-month period than would nondepressed mothers.
2. On average, depressed mothers would be more likely to demonstrate neglectful parenting over a 36-month period than would nondepressed mothers.
3. On average, depressed mothers would be more likely to demonstrate emotional maltreatment over a 36-inonth period than would nonde- pressed mothers.
In addition, we analyzed the interaction between depression and time for each parenting practice to determine how changes in maternal depression between baseline and 36-month follow-up affected changes in parenting behaviors. Finally, other child, family, and case characteristics associated with par- enting practices were determined.
RESEARCH METHOD The NSCAW, a fixed-panel design with four waves of data collection, had a stratified two-stage sample. The primary sampling units (PSUs) were county child welfare agencies; the secondary sampling units were children (and their families) chosen from a list of completed investigations at the sampled agencies. The sample was selected from 92 PSUs located in 36 states (NSCAW Research Group, 2002). The random sample of families within each agency was drawn from those who underwent a complete investigation for child maltreatment.The targeted population was all children and families investi- gated for child maltreatment in the United States; however, four states that required child welfare agency personnel to make first contact with the family instead of the NSCAW field representative were excluded from the study. For statistical rea- sons, infants, sexual abuse cases, and cases receiving ongoing services after the investigation were over- sampled (Dowd et al., 2003). Weighting was then performed to adjust for the unequal probability
Social Work Research VOLUME 35, NUMBER 4 DECEMBER 2011216
of selection from oversampling and nonresponse. Cases with both substantiated and unsubstantiated maltreatment were included in NSCAW. The ra- tionale for inclusion of both types of cases in the proposed project was the significant evidence that the ultimate substantiation of a particular report is not a good indicator of the seriousness of the report or the likelihood of continued and serious problems in parenting (Drake, Jonson-Reid, Way, & Chung, 2003; Hussey et al., 2005;Jonson-Reid, Drake, Kim, Porterfield, & Han, 2004; Kohl & Barth, 2007; Kohl, Jonson-Reid, & Drake, 2009). Furthermore, many states now use a differential response system and offer voluntary services to at-risk families whose cases were not substantiated. Hence, substantiation status cannot be used as a proxy for service receipt.
The NSCAW data were collected from caregiv- ers and child welfare workers at four time points: baseline (between October 1999 and December 2000), approximately 12 months after baseline (wave 2), approximately 18 months after baseline (wave 3), and approximately 36 months after baseline (wave 4). At baseline, wave 3, and wave 4, an NSCAW field representative conducted face-to-face interviews with the permanent caregiver of children remaining in the home; for wave 2, the field representative con- ducted a telephone interview with the permanent caregiver. Child welfare workers also participated in face-to-face interviews at baseline. If a case remained open to child welfare services, additional worker face-to-face interviews were completed at wave 2, wave 3, and wave 4. Wave 1, wave 3, and wave 4 included comparable measures of maternal and child functioning and mental health that were not included in wave 2. Data regarding service receipt was collected from caregivers and child welfare workers at wave 2.
Sample The entire NSCAW sample included 5,501 children (ages 0 to 16 years) and their families investigated for child maltreatment. The following cases, rep- resenting a subset of NSCAW, were included in this study:
• The child remained in home after the index investigation and spent no more than 5% of the study duration in out-of-home placements.
• The child was between the ages of 3 and 10 years at baseline.
The child's primary caregiver was identified as his or her mother (biological, adoptive, or step).
The child age inclusion criterion was selected because of the potent influence of parenting duriiig the preschool and elementary school years. NSCAW did not capture parenting behaviors that are par- ticularly influential during infancy and toddlerhood; therefore, the youngest children were excluded. In addition, parenting influences may be less powerful during adolescence due to adaptational and matu- rational processes (Sim &Vuchinich, 1996). ¡
With these inclusion criteria, the final sample size was 1,536 cases. Only one child per family was included in the NSCAW; therefore, children were not nested within mothers. The sample was composed of 1,536 mother—child dyads. i
Measures Following is an overview of the manner in which variables were measured. ¡
The dependent variables were three parenting practices: harsh parenting, neglect, and emotional maltreatment. These were measured with three subscales of the Conflict Tactics Scale-Parent to Child version (CTS-PC) (Straus, Hamby, Moore; & Runyan, 1998) at baseline, wave 3, and wave 4.The Physical Assault subscale assessed harsh parenting with the following nine items: (1) spanked child on bottom with bare hand; (2) .slapped on the hand, arhi, or leg; (3) hit on bottom with a belt, hairbrush, stick, or another hard object; (4) hit some other part ¡of the body besides the bottom with a belt, hairbrush, or stick; (5) pinched the child; (6) slapped on the face, head, or ears; (7) hit with a fist or kicked hatd; (8) threw or knocked down; and (9) beat up (that is, kicked or hit the child over and over as hard as possible) .The Neglect subscale assessed neglect with the following five items: (1) had to leave your child home alone, even when you thought some adult should be with him or her; (2) were not able ¡to make sure your child got the food he or she needed; (3) were so drunk or high that you had a problem taking care of your child; (4) were not able to make sure your child got to a doctor or hospital when he or she needed it; and (5) were so caught up with your problems that you were not able to show br tell your child that you loved him or her. Finally, the Psychological Abuse subscale assessed for emotiorial maltreatment with the following five items: (1)
K O H L , K A G O T H O , A N D D I X O N / Parenting Practices among Depressed Mothers in the ChildWelfare System 217
shouted, yelled, or screamed at child; (2) threatened to spank or hit the child but did not actually do it; (3) swore or cursed at child; (4) called child dumb or lazy (or similar statement);and (5) said you would send child away or kick child out of the house. As recommended by the scale developers (Straus, 1991), median scoring was used to assess the frequency of each parenting behavior, with one incident scaled as I, two incidents scaled as 2, three to five incidents scaled as 4, six to 10 incidents scaled as 8,11 to 20 incidents scaled as 15, and more than 20 incidents scaled as 25. The three parenting variables exhibited a high degree of skewness, in large part due to the high occurrence of 0 values (neglect: about 70%; harsh parenting: about 90%; emotional maltreat- ment: about 40%). Data transformations failed to normalize these data.Thus, a natural dichotomiza- tion at 0 versus not 0 was appropriate. Responses on the parenting outcome measures were analyzed as a series of individual time points (for example, baseline, wave 3,and wave 4) in the bivariate analyses and were analyzed as time-varying variables in the multivariate analyses.
The primary independent variable in our analytic models was maternal depression, which was mea- sured as a binomial variable with the Composite International Diagnostic Interview—Short Form (CIDl-SF) at baseline, wave 3, and wave 4. The CIDI-SF is a structured interview designed to screen for common psychiatric disorders with diagnostic criteria established in the DSM—IV (American Psychiatric Association, 1994; Kessler, Andrews, Mroczek, Ustun, & Wittchen, 1998). Mothers who met the diagnostic criteria for clinical depression were coded as 1 ; mothers who did not meet these criteria were coded as 2. As with the parenting out- come measures, responses on the depression measure were analyzed as a series of individual time points in the bivariate analyses and as a time-varying variable in the multivariate analyses.
Control variables included in the analysis were child gender, child age at baseline, mother race/ ethnicity, mother age at baseline, mother educa- tional attainment, family income, urban or nonurban status, and most serious maltreatment type of the baseline maltreatment report. Family income was categorized as "poor" versus "nonpoor" on the basis of the federally defined poverty level. This measure was calculated on the basis of procedures followed by the U.S. Census Bureau and includes both the family's income level and the number of adults and
children in the household (Dalaker & U.S. Census Bureau, 2001).The poverty measure was used as a dichotomous variable in the analyses (at or below poverty threshold or above poverty threshold). Urban/nonurban status of the county was defined consistent with U.S. Census definitions. Urban was defined as greater than 50% of the population liv- ing in the urban area, and twnurban was defined as all other areas that did not meet this description (NSCAW Research Croup, 2002) .The maltreatment type of the official report at basehne investigation was obtained from the child welfare worker. From a list of 10 categories, the worker first indicated all maltreatment types included in the report. When multiple maltreatment types were reported, the most serious maltreatment type was determined by using a slight modification of the Maltreatment Classification System (Manly, Cicchetti, & Barnett, 1994), resulting in five categories of maltreatment: (1) physical abuse; (2) sexual abuse; (3) neglect: failure to provide; (4) neglect: failure to supervise; and (5) other. For purposes of our analyses, we col- lapsed the categories into physical abuse, neglect, and other. Physical abuse was the referent category in our analytic models.
Data Analysis Strategy Data were analyzed using Stata 10 data analysis software. All analyses used the NSCAW sampling weights; therefore, findings are nationally repre- sentative and generalizable to child welfare cases in which a child (between the ages of 3 and 10 years) remained in the home with his or her mother for at least 95% of the time in the 36 months after a maltreatment investigation.
The data analysis strategy included univariate, bivariate, and multivariate analysis techniques. Frequencies were calculated to provide a general description of the data. Chi-square tests, ( tests, and unadjusted odds ratios were used to analyze the bivariate relation between major depression and the outcome and control variables. Finally, cross- sectional and longitudinal logistic regression models were built to analyze associations and interactions between dependent and independent variables. Generalized estimating equations (GEEs) were used (Diggle, Heagerty, Liang, & Zeger, 2002).The GEE methodology provides a method of analyzing cor- related data that arise from longitudinal studies in which subjects are measured at different points in time. GEEs are most effective when the focus is on
218 Social Work Research VOLUME 35, NUMBER 4 DECEMBER 2011
estimating the average response over the population (population-averaged effects),also referred to as the "marginal mean model." The resulting model re- gression coefficients have interpretations that apply to the population of individuals defined by fixing the values of the other covariates in the model.The correlated binary nature of our longitudinal inde- pendent variable (maternal depression—yes/no) lent itself to the GEE methodology as hkelihood-based inference was less applicable.
The xtgee command in Stata was used for GEEs, with the binomial specification for family to indicate the binary dependent variables represented by the three dichotomized parenting practice outcomes. In addition, compound symmetry was obtained by using exchatigeable for the correlation specification among the binary outcomes.
To conduct the longitudinal multivariate analy- ses, we transformed the data from a wide to a long file. The time-varying dependent variables were coded as follows: If wave = baseline, then the base- line score was used; if wave = 3, then the wave 3 score was used; and if wave = 4, then the wave 4 score was used. Wave was then controlled for in all our analytic models.Three parenting measures were analyzed as dependent variables in separate models. In each model, the other two parenting measures were included as independent variables (for ex- ample, when neglect was the dependent variable, harsh parenting and emotional maltreatment were included). Neglect and emotional maltreatment were moderately correlated (a = .29, p < .001). Although this correlation is low enough to indicate that they are distinct constructs, the correlation is high enough that the relationship should be ac- counted for in the models.
Both main effect and interaction models were analyzed with this approach.The interaction model included a dummy-coded interaction term of de- pression by wave.The resulting interaction term was a three-level categorical variable (no depression at baseline, no depression at wave 3, no depression at wave 4), with no depression at baseline held as the reference group across all models.
To correct for missing values in the dependent variables, independent variables, and other control variables, we performed multiple imputation by chained equations. The missing values were im- puted in 10 iterations to create a simulated data set. All analyses were conducted on the simulated data set.
Table 1: Description of Sample (Unweighted N = 1,536)
Child gender
Male
Female
Child age at baseline (years)
3-5
6-10
Mother's race/ethnicity
Black, non-Hispanic
White, non-Hispanic
Hispanic
Other
Mother's educational attainment
Less than high school
High school graduate
Some post-high school education
Family's income
At or below poverty threshold
Above poverty threshold
Primary maltreatment type
Physical abuse
Sexual abuse
Neglect: Failure to provide
Neglect: Failure to supervise
Other
Prior maltreatment reports
Yes
N o
Urbanicity of community
Nonurban
Urban
Child age
Mother's age
Number of people living in home
53.6
46.4
35.4
64.6
22,9
50,8
19.2
7.1
29.0
45.5
25.5
49.8
50.2
28,8
13,3
20,1
26,2
1 1 7
48,3
517
24,1
75.9
6.5
32.0
4,3 Note: We conducted chi-square and í tests to test for differences between cases w i t h depressed mothers at baseline and cases with nondepressed mothers at baseline for each variable reported in this table. No significant differences were found.
RESULTS
A description of the cases included in the sample is presented in Table 1, Slightly more than half of the children were male (53.6%).The racial and ethhic composite of the sample of mothers was 22.9% black, 50.8% white, and 19.2% Hispanic. Faniily income was evenly distributed between at or belpw the poverty threshold (49.8%>) and above the pQv-
KoHL, KAGOTHO, AND DIXON / Parenting Practices among Depressed Mothers in the Child Welfare System 219
erty threshold (50.2%).The majority of the sample (75.9%) lived in urban areas.The mean age for the children was 6.5 years, with 64.6% between the ages of 6 and 10 years.The mean age of the mothers was 32.0 years. Overall, the mothers had low levels of educational attainment; 29.0%) of the mothers re- ported less than a high school education. Regarding child welfare case characteristics, a slight majority (51.7%) of the sample had no previous referrals to CPS agencies. Neglect was most frequently identi- fied as the most serious child maltreatment type by child welfare workers (failure to provide: 20.2%; failure to supervise; 26.2%).
Mothers' self-reports of maternal depression and parenting practices at each of the three waves are reported in Table 2. Approximately one in five mothers (21.1%) met the diagnostic criteria for major depressive episode at baseline, and this percentage was fairly stable across waves (15.5% at wave 2, 21.5% at wave 4). More than half (59.4%) of mothers did not report depression at any wave, and 5.7% of mothers reported depression at all waves; for 34.9%, the results were mixed across waves (not shown in table). As shown in Table 2, harsh parenting practices were highly skewed in the direction of the absence of these behaviors across all three waves. Nearly one out of every 10 mothers (9.6%) reported harsh parenting practices at baseline, whereas approximately 14% of mothers reported harsh parenting at waves 3 and 4. Approximately one-third (35.0%) of mothers reported neglectful parenting behavior at baseline, whereas 30.8% and 35.2% ofmothers, respectively,reported the same at waves 3 and 4. Finally, a higher percentage ofmothers reported emotional maltreatment at all three time points; 61.5%, 55.4%, and 56.1% at baseline, wave 3, and wave 4, respectively.
The association between maternal depression and parenting behaviors reported at baseline are reported in Table 3. The unadjusted odds of self-reporting neglect for depressed mothers were approximately three times those of nondepressed mothers at base- line (odds ratio [OR] = 2.7, p < .001) and wave 3 (OR = 3.5, p < .001). In addition, the unadjusted odds of emotional maltreatment for depressed moth- ers were approximately twice those of nondepressed mothers at baseline (OR = 2.0, p < .001), wave 3 (OR = 2.3,p < .001), and wave 4 (OR = 2.6,;; < .001).The odds ofself-reported harsh parenting were not statistically significantly different for depressed and nondepressed mothers.
Results of the main effects multivariate models assessing the relation between parenting and depres- sion are reported in Table 4. Consistent with the bivariate analysis, depression status and self-reported harsh parenting were unrelated.The overall model fit, however, was significant fWald)(^(17) = 145.6,/) < .001]. For this and the other models, the average Wald chi-squares for the 10 produced completed data sets are reported because Stata output did not include Wald chi-squares for analyses of the simu- lated data set. As demonstrated by the statistically significant wave variables, harsh parenting signifi- candy changed over time.The odds ofself-reported harsh parenting were significantly higher at wave 3 than at baseline (OR = 1.8, p < .05) and at wave 4 than at baseline (OR = 1.7, p < .05), with the other variables in the model controlled for. Racial and ethnic differences were found. Black and His- panic mothers were about two times more likely to self-report the use of harsh parenting practices over the 36-month study window than were white mothers (OR = 2.3,p < .001, and O R = 2 . 0 , ; J <
.05, respectively). FinaUy, self-reported emotional
Table 2: Frequencies of Maternal Depression and Parenting Practices Measured at Multiple Time Points (Unweighted N = 1,536)
msMïus Dependent variable
Maternal depression
Parenting practices
Harsh parenting
Neglect
Emotional maltreatment
S5t3S
21.1
9,6
35.0
61,5
cs®
78,9
90,4
65,0
38,5
S33S
15.5
13.9
30.8
55.4
Ws
84.5
86.1
69,2
44.6
21.5
13,9
35,2
56,1
C3®
78.5
86.1
64.8
43.9 Note: All values represent weighted percentages.
220 Social Work Research VOLUME 35, NUMBER 4 DECEMBER 2011
Table 3: Odds Ratios for Maternal Depression and Parenting Practices (Unweighted N = 1,536)
Harsh parenting
Neglect
Emotional maltreatment
1.4
2 T * * *
2 . 0 * * *
1.3
3.5***
2.3***
1.1
1.5
2.6'
maltreatment and self-reported neglect frequently co-occurred with harsh parenting. Mothers report- ing emotional maltreatment (OR = 3.8, p < .001) and neglect (OR = 2.2,p < .001) had much higher odds of also self-reporting harsh parenting than did mothers not reporting emotional maltreatment and neglect, respectively.
Depression was statistically significant in the ne- glect model [overall model fit: Wald x^(17) = 104.8, p < .001]. Depressed mothers were 1.8 times more likely to self-report neglectful parenting behaviors than were nondepressed mothers. Mothers engaging in self-reported emotional maltreatment had a higher odds (OR = 2.4, p < .001) of also self-reporting
neglect than did mothers without self-reported emotional maltreatment. In addition, mothers with self-reported harsh parenting were two times more likely to self-report neglect than were mothers without self-reported emotional maltreatment (QR = 2.0,p<.01). '
The odds of emotional maltreatment were greater among depressed mothers than nondepressed mothers (OR = 1.8, p < .001).The overall motfel fit was good [Wald x\\7) = 142.2,;; < .001], and additional variables were associated with emotional maltreatment across the study window. Emotional maltreatment was associated with self-reported harsh parenting and neglect. For mothers reporting harsh
Table 4: Multivariate Models Assessing the Relationship between Self-reported Parenting Practices and Depression (Main Effect Models)
Major depression (No depression)
Wave 2 ( 18-month follow-up) (Baseline)
Wave 3 (36-month follow-up) (Baseline)
Child gender (Male)
Parent age
Mother race: Non-Hispanic black (Non-Hispanic white)
Mother ethnicity: Hispanic (Non-Hispanic white)
Mother race: Other (Non-Hispanic white)
No high school education (More than high school)
High school education (More than high school)
Urban/rural status (Urban)
Prior reports (No prior reports)
Poverty (At or below poverty threshold)
Official report: Neglect (Physical abuse)
Official report: Other (Physical abuse)
Self-reporr: Emotional maltreatment
Self-report: Harsh parenting
Self-report: Neglect
1.0
1.8*
1.7*
1.1
1.0
2.3***
2.0*
1.4
0.9
1.3
0.9
0.7
1.4
0.8
0.7
3.8***
0.6, 1.7 1.1,2.9
1.1,2.6
0.7, 1.3
1.0, 1.0
1.3,3.3
1.2,3.3
0.7, 2.9
0.3, 1.7
0.8,2.1
0.6, 1.4
0.3, 1.1
0.9, 2.2
0.3 1.3
0.4, 1.2
2.2,6.3
1.8**
0.9
1.0
1.1
1.0
1.1
1.2
1.0
1.1
1.0
1.3
1.2
1.0
1.4
1.2
2.4***
2.0**
1.3,2.3
0.7, 1.1
0.8, 1.3
0.8, 1.4
1.0, 1.0
0.7, 1.6
0.7, 1.6
0.6, 1.8
0.7, 1.6
0.7, 1.4
0.9, 1.7
0.9, 1.6
0.7, 1.3
1.0,2.1
0.8, 1.9
1.8,3.2
1.3,2.9
1.8***
0.8
0.7
0.9
1.0
1.5*
0.7
0.9
0.7
0.7
0.9
1.1
1.0
0.8
0.8
3.0***
1.3,2.
0.6, 1.
4
0
0.6, 1.0
0.8, 1.4
1.0, 1.0
1.1,2.2
0.3, 1.1
0.3, 1.7
0 . 4 , l . i l
0.3, 1.1
0.7, l.j
0.7, 1.3
0.7, 1.3
0.6, 1.2
0.5, 1.2
1.8.5.0 2.2* 1.4,3.3
Note: Reference groups are given in parentheses, OR = odds ratio; CI = 95% confidence interval. ' p s ,05, •*p s ,01, ***p Ä ,001,
KOHL, KAGOTHO, AND DIXON / Parenting Practices among Depressed Mothers in the ChiU We/fare System 221
parenting, the odds of emotional maltreatment were three times greater than for riiothers not reporting harsh parenting (OR = 3.0,p< .001).Likewise,for mothers reporting neglect, the O R of self-reported emotional maltreatment was 2.3 (p < .001). Black mothers were more likely to self-report emotional maltreatment than were white mothers (OR = 1.5,p< .05).
To assess whether changes in maternal depres- sion between baseline and wave 4 affected changes in parenting behaviors, we analyzed additional multivariate models that included a Depression x Wave term. Results from the harsh parenting and neglect model are not reported because inclusion of the interaction term did not contribute to the models. The overall fit of the emotional maltreat- ment model was good [Wald x^(19) = 151.7, p < .001], and the Depression x Wave interaction was significant (not shown in table). Depressed mothers at wave 3 were two times more likely to self-report emotional maltreatment than were nondepressed mothers at baseline (OR = 2.2; confidence interval = 1.1,4.3;/) < .05)—an indication that risk of emo- tional maltreatment varied over time by mother's depression status. The significance and strength of association of the other variables in the model were similar to the main effects model and are therefore not reported again.
DISCUSSION By using a national probability sample, we were able to demonstrate that maternal depression impedes the achievement of the primary objective of child welfare services: child safety. Maternal depression, which is prevalent among this population, was found to place children at risk for both self-reported ne- glect and emotional maltreatment. On entry into the child welfare system, 21% of mothers met the diagnostic criteria for major depression—well above the national average of 7% in the general population (Kessler et al., 2005). Furthermore, the percentage of mothers reporting depression remained fairly stable across the study window. That only 5.7% of mothers reported depression at all three time points means that different women experienced depression at dif- ferent time points after entry into the child welfare system. Although, at first glance, the percentage of mothers reporting depression at all three time points appears low, this represents a substantial number of mothers. This rate is a concern given the harmful effects of persistence of maternal depression on chil-
dren.The investigators of the Sequenced Treatment Alternatives to Relieve Depression trial found that, although children improved when their mothers' depression subsided after a medication intervention, symptoms worsened when their mothers' depression continued (Weissman et al., 2006).
Harsh Parenting We hypothesized that, on average, depressed mothers would be more likely to demonstrate harsh parenting over a 36-month period than would nondepressed mothers.This hypothesis was not supported; among the child welfare population, depressed and nonde- pressed mothers had similar rates of self-reported harsh parenting.This unanticipated finding is con- trary to the published literature. Among community populations, maternal depression has been found to increase the risk of harsh parenting (Chung et al., 2004; Lovejoy et al., 2000; Lyons-Ruth et al., 2002). However, this relation was not upheld among this national probability sample of mothers in the child welfare population whose children remained in the home after the index maltreatment investiga- tion. This finding may be a result of differences in measurement of harsh parenting across studies or of differences between the community population and the child welfare population. Families enter- ing the child welfare system are often faced with a complex web of problems, and it may be the cumu- lative nature of those problems that places children at risk for harsh parenting practices, not maternal depression alone.
Harsh parenting was a fairly rare event, but the percentage of mothers self-reporting harsh parent- ing increased from baseline (9%) to wave 4 (14%); the increase remained statistically significant in the multivariate models. It is alarming that harsh parent- ing increased over the course of the study window. This finding highlights the need to effectively as- sess discipline strategies used by mothers receiving voluntary or mandatory services following entry into the child welfare system and, when warranted, provide effective interventions aimed at reducing the use of harsh parenting behaviors.
Consistent with the findings of others, we found an increased risk of harsh parenting toward black children (for example, Deater-Deckard, Dodge, Bates, & Pettit, 1996; Pinderhughes, Dodge, Bates, Pettit, & Zelh, 2000). These results need to be discussed within their cultural context. Culture influences parental beliefs about child development
222 Social Work Research VOLUME 35, NUMBER 4 DECEMBER 2on
promotion and appropriate socialization strategies (Caughy & Franzini, 2005; Murry, Smith, & Hill 2001). Hence, harsh parenting may serve different functions and have different meanings for black and white families. Among black famihes, harsh (physical) parenting appears to have a socialization role, the purpose being to prepare youths for adult competence (Deater-Deckard & Dodge, 1997). Furthermore, although harsh parenting increases the risk for externalizing behavior problems among white children, this same relation does not hold true for black children (Deater-Deckard & Dodge, 1997). Parenting interventions aimed at changing parenting behaviors must therefore be culturally relevant.
Neglectful Parenting Hypothesis 2 was supported. On average, depressed mothers were more likely to engage in neglect- ful parenting over a 36-month period than were nondepressed mothers. Bivariate findings revealed some variation in self-reported neglect from baseline (34%) to wave 3 (24%) to wave 4 (33%); however, these differences were not statistically significant when other variables in the multivariate model were controlled for. Surprisingly, we did not find a significant Depression x Wave interaction term in the neglectful parenting model. This indicates that the risk of neglect did not vary by changes in depression over time, which is likely related to the relatively stable percentage of depressed mothers (although different mothers) across time points.
Emotional Maltreatment Hypothesis 3 was also supported. Depressed moth- ers were more likely to demonstrate emotional maltreatment over a 36-month period than were nondepressed mothers, and emotional maltreatment improved more for nondepressed mothers than for depressed mothers.
Overall, rates of self-reported emotional mal- treatment were high across waves. In fact, over half of mothers reported emotional maltreatment at each of the time points.These high rates of ongoing emotional maltreatment after a CPS referral are a concern, given the long-term adverse consequences of experiencing this type of maltreatment in child- hood. Emotional maltreatment has been found to be an important contributor of psychological adjustment in young adulthood, with higher levels of emotional maltreatment being associated with
poorer outcomes (Miller-Perrin, Perrin, & Kociir, 2009).
Emotional maltreatment, which can cause these adverse outcomes, frequently co-occurs with both harsh parenting and neglect. This suggests that physical abuse or neglect should not be the scjle focus of interventions after entry into the child welfare system. Assessments of mothers determined to need voluntary or mandated services following child maltreatment investigations should evaluate a range of parenting behaviors, including emotional maltreatment, physical abuse, and neglect. '
Assessments for maternal depression also appebr to be essential, given the high percentage of mothers in the child welfare system meeting the diagnostic criteria for major depressive episode. Although all mothers determined to need services on entry into this system of care should be provided with inteN ventions to improve positive parenting and parent- child interactions, for some mothers, adaptatioris to'interventions may be necessary to concurrently address the mothers mental health needs. '
! Limitation and Strengths J An important limitation is that our sample was liinited to mother-child dyads in which the child remained in the home after the index maltreatment investigation.The experiences of children who were removed from the home are likely different froiii those of children who remained in the home after a CPS referral; however, we were unable to analyse these important differences. Self-report measures of major depression and parenting were only obtained at baseline, wave 3, and wave 4 from the permanent caregiver if the child remained in the home. '
Another limitation is the sole reliance of self- report of parenting practices. Mothers may be re- luctant to divulge information about their parenting practices (Knight et al.,2000),so parental self-report of their behaviors could result in lower bound estimates of the actual behavior (Straus, Gelles, & Steinmetz, 1980). Although NSCAW took steps tp increase the disclosure of sensitive topics through the use of an audio computer-assisted self-interview, harsh, neglectful, and emotionally abusive parenting may be underreported by mothers. ¡
Although it is important to acknowledge these limitations, the strengths outweigh the limitations. NSCAW provides rich epidemiological data about children and families investigated for child maltreat- ment. Because it is a national probability study, it
K O H L , K A G O T H O , A N D D I X O N / Parenting Practices among Depressed Mothers in the Child Welfare System 223
allows for the generalization of findings to all in- vestigated cases of maltreatment in which children between the ages of 3 and 10 years remained in the home.
IMPLICATIONS Rates of maternal depression were high across the 36-month follow-up period. Yet rates of mental health service receipt among the child welfare population are lower than rates among the general population. For example, 41% of those in the gen- eral population with a mental health need receive treatment (Wang et al., 2005), compared with 14% in the child welfare population (Libby et al, 2006). To improve long-term outcomes, efforts are needed to improve assessment and identification of mental health needs and access to treatment when deemed necessary.
Although there are a growing number of evi- dence-based parent training programs that aim to promote positive parenting, improve parent-child relationships, and reduce harmful parenting behav- iors, they are rarely provided within the child wel- fare system. Although the majority of child welfare families have parenting services included as part of their service plan, the services that are typically pro- vided have been harshly criticized for their lack of empirical support and applicability to a child welfare population (Barth et al.,2005;Hurlburt,Barth,Leslie, Landsverk, & McCrae, 2007).Translation efforts of evidence-based programs into this system of care should include research to determine whether they reduce the risk of emotional maltreatment, neglect, harsh parenting, and physical abuse. Furthermore, efforts should be undertaken to examine the sub- populations for which these programs are effective. For instance, do they work with both depressed and nondepressed mothers? In addition, our findings highlight the importance of providing culturally relevant services to the diverse population referred to CPS agencies, and efforts are needed to determine what, if any, cultural adaptations should be made to address their unique needs. KlVlil
REFERENCES American Pfsychiatric Association, (1994), Diagnostic
and statistical manual of mental disorders (4th ed,), Washington, DC: Author,
Barth, R, P., Landsverk,J,, Chamberlain, R, Reid,J, B,, Rolls,}, A,, Hurlburt, M, S., ec al, (2005), Parent- training programs in child welfare services: Planning for a more evidence-based approach to serving
biological parents. Research on Social Work Practice, 15, 353-371,
Campbell, S, B.,Brownell, C. A,, Hungerford, A,, Spieker, S,J,, Mohan, R,, & Blessing, j , S, (2004), The course of maternal depressive symptoms and maternal sensitivity as predictors of attachment security at 36 months. Development and Psychopatholo^y, 16, 231-252,
Caughy, M, O,, & Franzini, L, (2005), Neighborhood cor- relates of cultural differences in perceived effective- ness of parental disciplinary tactics. Parenting: Science and Practice, 5, 119-151.
ChafFin, M., Kelleher, K,, & Hollenberg,J, (1996), Onset of physical abuse and neglect: Psychiatric, substance abuse, and social risk factors from prospective com- munity data. Child Abuse & Neglect, 20, 191-203.
Chung, E. K., McCoUum, K. R, E l o J . T , Lee, H,J,, & Culhane,J, E (2004), Maternal depressive symptoms and infant health practices among low-income women. Pediatrics, 113, e523—e529,
Cummings, E. M., & Cicchetti, D. (1993).Towards a transactional model of relations between attachment and depression. In M,T, Greenberg, D. Cicchetti, & E, M, Cummings (Eds,), Attachment in the preschool years:Tlieory, research and intervention (pp, 339—371). Chicago: Chicago University Press.
Dalaker,J,, & U.S. Census Bureau. (2001, September). Poverty in the United States: 2000. Washington, D C : U.S. Department of Commerce, Economics and Statistics Administration, U.S, Census Bureau,
Deater-Deckard, K,, & Dodge, K, A, (1997), Externalizing behavior problems and discipline revisited: Nonlinear effects and variation by culture, context, and gender. Psychological Inquiry, 8, 161—175,
Deater-Deckard, K., Dodge, K, A,, Bates,J, E,, & Pettit, G, S, (1996), Physical discipline among African American and European American mothers: Links to children's externalizing behaviors. Developmental Psychology, 32, 1065-1072,
Diggle, P.J., Heagert>', P., Liang, K.Y, & Zeger, S. L, (2002), Analysis of longitudinal data (2nd ed,). New York: Oxford University Press,
Drake, B,,Jonson-Reid, M,, Way, I,, & Chung, S, (2003), Substantiation and recidivism. Child Maltreatment, 8, 248-260,
Hurlburt, M, S,, Barth, R, P, Leslie, L, K,, Landsverk,J, A,, & McCrae, j , (2007). Building on strengths: Current status and opportunities for improvement of parent training for families in child welfare. In R. Haskins & F.Wulczyn (Eds.), Using research to improve policy and practice (pp, 81—106). Washington, DC: Brookings Institution Press.
Hussey,J. M,, Marshall,J. M., English, D, j , . Knight, E, Q , Lau, A. S,, Dubowitz, H,, & Kotch,J, B, (2005), Denning maltreatment according to substantiation: Distinction without a difference? Child Abuse & Neglect, 29, 479-492,
Jonson-Reid, M,, Drake, B,, Kim,J,, Porterfield, S,, & Han, L, (2004), A prospective analysis of the relationship between reported child maltreatment and special education eligibility among poor children, Cliild Maltreatment, 9, 382-394,
Kessler, R, C , Andrews, G,, Mroczek, D,, Ustun, B,, & Wittchen, H, U, (1998),The World Health Organization Composite International Diagnostic Interview short-form (CIDl-SF). International Journal of Methods in Psychiatric Research, 7, 171—185.
Ke.ssler, R, C , Berglund, P,, Demler, O,,Jin, R,, Korecz, D , Merikangas, K, R., et al, (2003),The epidemiology of major depressive disorder: Results from the National Comorbidity Survey Replication (NCS-R),_//!Ai/4, 2S9, 3095-3105.
224 Social Work Research VOLUME 35, NUMBER 4 DECEMBER zoii
Kessler, R. C , Chiu,W.T., Detnler, Q , «¿Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM—/K disorders in the National Comorbidity Survey Replication. Archives of Ceneral Psychiatry, 62, 617-627.
Knight, E, D., Runyan, D. K., Dubowitz, H,, Brandfor, C.,Kotch,J,,Utrownik,A.,& Hunter,W. (2000). Methodological and ethical challenges associated with child self-report of maltreatment: Solutions implemented by the LongSCAN Consortium.JoMrna/ of Interpersonal Violence, 15, idQ—llS.
Kohl, P. L., & Barth, R. P (2007). Child maltreatment recurrence among children remaining in-home: Predictors of re-reports. In R. Haskins & EWulczyn (Eds.), Practical knowledge for child welfare practitioners: findings from the National Survey of Children and Adolescent Well-being (pp. 207-225).Washington, DC: Brookings.
Kohl, P L.,Jonson-Reid, M., & Drake, B. (2009).Time to leave substantiation behind: Findings from a national probability study. Child Maltreatment, 14, 17—26.
Libby, A. M., Orton, H, D., Barth, R, P, Webb, M. B., Burns, B.J,,Wood, P, 6f Spicer, P (2006). Alcohol, drug, and mental health specialty treatment services and race/ethnicity: A national study of children and families involved with child weXhre. American Journal of Public Health, 96, 628-631,
Lovejoy, M, C , Craczyk, PA., O'Hare, E,, & Neuman, G. (2000). Maternal depression and parenting behavior: A meta-analytical review. Clinical Psychology Review, 20,561-592.
Lyons-Ruth, K.,Wolfe, R., Lyubchik, A,, & Steingard, R. (2002). Depressive symptoms in parents of children under age three: Sociodemographic predictors, cur- rent correlates, and associated parenting behaviors. In N. Halfon, K.T. McLearn, & M, A. Schuster (Eds.), Child rearing in America: Challenges facing parents with young children (pp, 217-259), New York: Cambridge University Press.
Manly,J,T., Cicchetti, Q , & Barnett, D, (1994).The impact of subtype, frequency, chronicity, and severity of child maltreatment on social competence and behav- ior problems. Development and Psychopathotogy, 6, 121-143.
Miller-Perrin, C, L,, Perrin, R, D , & Kocur,J, L. (2009). Parental physical and psychological aggression: Psychological symptoms in young adults. Child Abuse & Neglect, 3 3 , ] - ] \ .
Murry,V. M., Smith, E. P, & Hill, N, E, (2001), Race, ethnicity, and culture in studies of families in context, fournal of Marriage and the Family, 63, 911—914.
NSCAW Research Group. (2002), Methodological lessons from the National Survey of Child and Adolescent Well-Being: The first three years of the USA's first national probability study of children and families investigated for abuse and neglect. Children and Youth Services Review, 24, 513—541.
Orr, S.T.,James, S, A,, Burns, B. j , , &Thompson, B. (1989), Chronic Stressors and maternal depression: Implications for prevention. American Journal of Public Health, 19, 1295-1296.
Pinderhughes, E, E., Dodge, K. A., Bates,j. E,, Pettit, G, S., & Zelli,A. (2000). Discipline responses: Influences of parents' socioeconomic status, ethnicity, beliefs about parenting, stress, and cognitive—emotional processes. Journal of Family Psychology, 14, 380—400.
Rinehart, D. j . , Becker, M. A.,'Buckley, P R., Dailey, K,, Reichardt, C. S., Graeber, C , et al. (2005).The relationship between mothers' child abuse potential and current mental health symptoms: Implications for screening and referral. JoMrHa/ of Behavioral Health Services & Research, 32, 155-166.
Shin, D.W., & Stein, M. A. (2008). Maternal depression predicts maternal use of corporal punishment in children with attention-deficit/hyperactivity disorder. Yonsei Medical Journal, 49, 573-580.
Sim, H,, &Vuchinich, S. (1996).The dechning efFects of family Stressors on antisocial behavior from child- hood to adolescence and early adulthood._/o"r"a/ bf Family Issues, 7 7,408-427.
Straus, M. (1991). Handbook for the Conflict Tactics Scale. • Durham: University of New Hampshire Family Research Laboratory.
Straus, M., Gelles, R.J., & Steinmetz, S. K. (1980). Behind closed doors: Violence in the American family. Garden City, NY: Anchor Books,
Straus, M., Haniby, S. L., Moore, D.W., & Runyan, D. (1998). Identification of child maltreatment with the Parent-Child Conflict Tactics Scales: Developmeiit and psychometric data for a national sample of American parents. Child Abuse & Neglect, 22, \ 249-270. ¡
Trapolini,T., Ungerer,J. A., & McMahon, C. A. (2008). ' Maternal depression: Relations with maternal I caregiving representations and emotional availability during the preschool years. Attachment & Human • Development, 10,73-90. i
U.S. Department of Health and Human Services, ' Administration on Children,Youth, and Families. I (2005). National Survey of Child and Adolescent Welü Being (NSCAW): CPS sample component wave 7 data analysis report. Washington, DC: U.S. Government I Printing Ofl̂ ice.
Wang, P S., Lane, M., Olfson, M., Pincus, H.A.,Wells, K, B., & Kessler, R. C. (2005),Twelve-month use (¡)f mental health services in the United States: Results from the National Comorbidity Survey Replication, Archives of Ceneral Psychiatry, 62, 629-640. I
Weissman, M. M., Pilowsky, D. j.,Wickramaratne, PJ., j Talati, A.,Wisniewski, S. R., Fava, M., et al, (2006); Remissions in maternal depression and child psy- ; chopathology:A STAR*D-child report,_//4A/i/l, 295, 1389-1398,
Patricia L. Kohl, PhD, is assistant professor, Centerfor Mental Health Services Research, George Warren Brown School of So- cial Work, Washington University in St. Louis, One Brookings Drive, Campus Box Í196, St. Louis, MO 63130; e-thail: [email protected] NJeri Kagotho, PhD, is assistant professor. School of Social Work, Adelphi Uttiversity, Garden City, NY. David Dixon, PhD, is a statistical data analyst, Centerfor Mental Health Services Research, George Warren Brown School of Social Work, Washington University in St. Louis. Support for this project was provided by National Institute of Mental Health Grant R03MH082203. Patricia Kohl is an investigator with the Centerfor Metttal Health Services Research, George Warren Brown School of Social Work, Washington University in St. Louis, through an award from the National Institute of Mental Health (5P30 MH068519).
Original manuscript received May 8, 2009 Final revision received April 1, 2010 Accepted April 27, 2010
KOHL, KAGOTHO, AND DIXON / Parenting Practices among Depressed Mothers in the Child Welfare System 225
Copyright of Social Work Research is the property of National Association of Social Workers and its content
may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express
written permission. However, users may print, download, or email articles for individual use.
Sheri Bauman, Ph.D., is an assistant professor in
the Department of Educational Psychology
at the University of Arizona, Tucson.
-E-mail: [email protected]
This article describes comparison group research designs and discusses how such designs can be used in school counseling research to demonstrate the effective- ness of school counselors and school counseling inter- ventions. The article includes a review of internal and external validity constructs as they relate to this approach to research. Examples of relevant research using this design are presented.
•he lack of a sound research base in the field of school counseling has been lamented for many
Syears (Allen, 1992; Bauman, 2004; Cramer, Herr, Morris, & Frantz, 1970; Lee & Workman, 1992; Loesch, 1988; Whiston & Sexton, 1998; Wilson, 1985). The recent emphasis on research in No Child Left Behind legislation (2002) and the ASCA National Model® (American School Coun- selor Association, 2005) has moved the need for rig- orous empirical research to the forefront. The ASCA National Model stresses that school counseling pro- grams include learning objectives that are based on measurable student outcomes and that are data-driv- en and accountable for student outcomes. The focus on data and measurement makes clear that school counselors can no longer avoid conducting research and using empirical research to make decisions.
The nature and goals of such research are the sub- ject of a recent debate. Brown and Trusty (2005) have contended that research should focus on demonstrating that well-designed and appropriate interventions used by school counselors are effec- tive, and they further argued that research investi- gating whether comprehensive school counseling programs increase student academic achievement is not productive given the presence of numerous con- founding influences. Sink (2005) disagreed, noting that school counselors are expected to contribute to the total educational effort to raise academic achievement. He advised that research to examine how school counselors influence achievement can be conducted using carefully selected methodologies, and while not definitively establishing causality, such research can provide strong evidence of the impact
of comprehensive school counseling programs on student achievement.
In their review of school counseling outcome research from 1988 to 1995, Whiston and Sexton (1998) found that of the 50 published studies they located, most provided only descriptive data, used convenience samples, lacked control or comparison groups, used outcome measures of questionable reli- ability and validity, and did not monitor adherence to intervention protocol, Such studies do little to add to the knowledge base of the profession, and they do not meet established standards for scientific rigor.
In an era of limited resources for education and "accountability" becoming a watchword, counselors must demonstrate how they contribute to the aca- demic success of students. Heartfelt letters of appre- ciation and positive comments by constituents, while sincere, will not convince stakeholders and holders of purse strings of the value of the profes- sion. School counselors, occupied by providing serv- ices in schools, often neglect to demonstrate their
importance until their positions are considered for reduction. This reactive approach is less likely to sway opinion than ongoing proactive efforts to use research effectively. Collecting, analyzing, and dis- seminating data that provide evidence of counselors' effectiveness are consistent with the professional goals and models that define the profession.
Under No Child Left Behind, school counselors (along with other education professionals) are called upon to demonstrate their effectiveness using quan- titative data such as evidence of academic achieve- ment, attendance and graduation rates, and meas- ures of school safety (McGannon, Carey, & Dimmitt, 2005). No Child Left Behind and the ASCA National Model both emphasize the impor- tance of scientific, rigorous, well-designed research as an essential component of modern school coun- seling programs. These guidelines indicate that con- ducting research is no longer a peripheral activity that a few counselors might attempt but is a central part of the role of all school counselors. The ASCA
9:5 JUNE 2006 1 ASCA 357
~~sin Com~p-rs1G~ush Sc~ o Co~s~i a=3
0%P P
National Model says the folloxving about data:
Data analysis: Counselors analyze student achievement and counseling-program-related data to evaluate the counseling program, con- duct research on activity outcomes and dis- cover gaps that exist between different groups of students that need to be addressed. Data analysis also aids in the continued develop- ment and updating of the school counseling program and resources. School counselors share data and their interpretation with staff and administration to ensure each student has the opportunity to receive an optimal educa- tion. (ASCA, 2005, p. 44)
Although most school counselors have had a graduate course in research methods (and perhaps statistics), these introductory courses typically are designed to prepare students to be critical readers of research. Even among those few students who con- duct research in their graduate training programs, it is the rare school counselor who continues to do research once he or she is a practicing school coun- selor. Responsibility for the absence of research from school counselors' job description lies not only with the counselors, but also with the administrators and district officials who do not require or value research. No Child Left Behind has raised the aware- ness of educators in all fields that accountability is expected; data are the foundation for educational decisions, including decisions about counselors. In this climate, there is greater support (some might say pressure) for research.
There are a number of different kinds of research, and a description of all relevant types is beyond the scope of this article. The purpose of this article is to provide a rationale for using control and comparison group designs in school counseling research. I begin by defining some basic terminology and reviewing the concept of validity, which is fundamental to all research. I then briefly discuss single-group pre-post research designs, which often are used in schools because they are relatively easy to conduct. The main focus of the article is comparison group designs, and these will be described in more detail. Finally, I provide a discussion of relevant research using comparison group designs as examples of this research strategy.
DEFINITIONS
Several technical terms are used in this discussion of research, and it is important that the reader be clear about their meaning. Researchers study variables that can assume different values. The independent variable is the intervention variable, or the variable
manipulated by the researcher. The dependent var- able is the outcome variable, the effect. In a study of the effect of participation in extracurricular activities on graduation rates, the independent variable is the participation (which could be defined as the number of activities, or the number of hours per week of involvement, or a yes/no category) and the gradua- tion rate is the dependent variable. Researchers also may refer to moderator variables, which are variables that influence the relationship between the inde- pendent and dependent variables. Parental educa- tion might moderate the relationship between extracurricular participation and graduation rates, and it then would be a moderator variable. Statistical significance means that the obtained results are unlikely to have occurred by chance. If results are statistically significant at p < .05, the results would be obtained by chance in less than 5 out of every 100 cases. The counselor/researcher should keep in mind that with large samples, results might be statistically significant but not practically significant.
Let us imagine that a new program for elementary math skills were implemented in several schools in a large district. At the end of a school year, the differ- ence between the achievement scores of those who used the program and those who continued the usual math program was statistically significant. One might conclude that the new approach is better. But what if the difference in scores were only .10 (grade equivalent)? Depending on the cost of the program, one might conclude that although the difference is statistically significant (p < .05), in practice the dif- ference or gain is not substantial enough to justify a large expenditure on the new program. There are ways to describe the practical significance of the findings through the use of effect sizes; these are dis- cussed in Sink and Stroh's article in this issue ("Practical Significance: The IUse of Effect Sizes in School Counseling Research").
VALIDITY
Regardless of the design or method of research, school counselors must be concerned with the valid- ity of the research they conduct or read. In general, validity refers to the degree of confidence we can have in the findings of a research study. If a study does not demonstrate adequate validity, the results are of questionable application and should not be the basis for decisions. Internal validity refers to whether the observed change in the dependent (outcome) variable is due to the independent vari- able and only the independent variable. For exam- ple, if we are interested in whether student atten- dance (our dependent variable) improved for high school freshmen when a new orientation program
358 ASCA I PROFESSIONAL SCHOOL COUNSELING
The focus on data
and measurement
makes clear that
school counselors
can no longer avoid
conducting research
and using empirical
research to make
decisions.
was conducted by the school counselors (our inde- pendent variable; the orientation program), we want to be sure that no other variables could explain the obtained results. If, in addition to the new orienta- tion program for freshmen, the school employed additional truant officers, we could not be sure that the change in attendance was due only to the new orientation program and not to the truant officers' activities. The internal validity of the study would be compromised.
External validity refers to the degree that the results of one study can generalize to (apply to) other people in other places or times. School coun- selors reading the results of research in a journal want to know whether they can reasonably expect that the reported results would apply in their own setting with their own students. Researchers hope that their results will be useful to others in other locations and times. If the new orientation program improved attendance for students in one school or district, the issue of external validity asks the ques- tion of whether other schools are likely to achieve the same results with the same program. The threats to external validity are related to the population from which the sample was selected (was it repre- sentative, did it include members of all groups of interest?) and the context in which the study was conducted (was it in a laboratory or a school, did participants know they were involved in an experi- ment, did the researcher convey the hoped-for out- comes?). These two types of external validity often are referred to as population validity and ecological validity. A study conducted at a private school with European American upper-class students is of ques- tionable validity for an inner-city school with a large percentage of minority students.
Another factor in external validity is the nature of the research itself. If the students in the experimen- tal group were aware they were receiving a special program different from that of the control group, their efforts may have been changed by that knowl- edge. In addition, the researcher must incorporate a way to ensure that interventions delivered in a natu- ralistic school setting are faithfud to the protocol of the experiment. If the intervention is a series of les- sons, for example, the researcher must be sure that the lessons are delivered as described in the manual. If each teacher or counselor makes changes in the program, external validity is compromised by the absence of treatment (intervention) fidelity. Researchers can increase the external validity of their work by attending carefully to sample selection and to the conduct of the experiment.
Campbell and Stanley (1963) described important threats to internal validity. These are conditions that provide possible alternative explanations for obtained results, or -ways that events or conditions
other than the independent variable may explain observed changes in the dependent variable. The following is a brief review of those threats.
History In this context, history refers to any event not planned or part of the research that occurs during the research. In the example above regarding the new orientation program, let's imagine that the principal decides to visit each freshman English class during the first week of school. Although not part of the research, this event (history) might be an alter- native explanation for the difference in attendance rates. History is the greatest threat to internal valid- ity when it affects only one group of research partic- ipants. If your research design used a comparison group (last year's freshmen) who had not experi- enced the historical event, your internal validity would be reduced. However, if you were studying whether the attendance of males vs. females increased when the new orientation program was implemented, and both males and females experi- enced the visits by the principal, internal validity would not be affected. When research is being con- ducted in schools, there are often events that occur outside the counselors' control, and the counselor must be alert to these competing explainers of results.
Maturation Human beings change and develop over time. This means that some changes will occur independently of any intervention. For example, a middle school counselor might provide a series of guidance lessons on conflict resolution to seventh graders. If the counselor were to measure student attitude toward fighting, or the number of fights before and after the lessons, results might show a decrease in conflict after the lessons. However, maturation might be an alternative explanation for the results; students may be exhibiting less physical conflict because they are developing cognitively and socially, not because of the lessons. In the discussion of comparison group designs later in this article, I suggest designs that minimize the influence of this threat to internal validity.
Testing Researchers may want to give participants a pretest to determine the base rate of whatever behavior or attitude is of interest. If a counselor were going to do a series of guidance activities to reduce racial/ethnic stereotyping in a school, he or she may want to get a measure of the degree of stereotyping that students do at the start of the project. However, the pretest may sensitize participants to the issue of stereotyping, and that may influence their scores on
9:5 JUNE 2006 1 ASCA 359
In an era of limited
resources for
education and
"accountability"
becoming a
watchword,
counselors must
demonstrate how
they contribute to
the academic
success of students.
the posttest. This is called the testing effect.
Instrumentation A counselor who is leading an anti-bullying program at her school wants to measure the effect of the pro- gram on student bullying behavior. She knows that much bullying occurs on the playground, and she uses a behavioral observation method to determine the frequency of playground bullying before the program begins, and after the program has been in place for a semester. The behavioral observation method requires several observers, and it may be that some observers are more alert than others. Or, the observers may become more adept with practice. If the observers are not the same at both measure- ment points, instrumentation is a threat to internal validity. The changes may not be a result of the chil- dren's behavior, but of the observers' skill.
Regression to the Mean When a counselor is interested in extreme groups (students high or low in a particular characteristic), a pretest-posttest design is vulnerable to this threat. We know that on subsequent testing, both high and low scores tend to become closer to the mean (aver- age score). So observed changes may be due to this tendency rather than any real change in the charac- teristic being measured.
Selection In some research, counselors are studying more than one group of students (e.g., classes, genders). If you are trying a new program with one class and using another class as a comparison group, the groups might be different on some other factors (e.g., read- ing level, intelligence) that can affect the results.
Mortality This threat to internal validity refers to loss of par- ticipants during the course of the study. In a com- parison group design, this becomes a problem when mortality is greater in one group than in another. For example, in a study where the comparison group is another school, there might be asbestos discov- ered in one of the schools and many students trans- fer out of that school. That group would have greater mortality than the other group.
Selection Interaction It is possible that one of the other threats to internal validity combines with selection. This means that one of the comparison groups is affected by those threats (e.g., history, maturation) differently than other groups in the study. For example, an elemen- tary school counselor implements a new program to teach empathy skills to fifth-grade classes. Lessons are given throughout the school year, and a nearby
school serves as a control group. On an outcome measure, the counselor finds that at the end of the year, girls show more improvement in empathy than boys do. It might be that those findings are because girls of this age tend to develop these skills naturally at this age, while boys develop them later. The find- ings may reflect a selection (grade level) maturation (girls faster than boys) effect.
School counselors doing research must be alert to potential threats to internal validity. While it is impossible to avoid all such threats, especially when doing the research with students in schools (rather than in a laboratory), if results are to be meaningful, the researcher must acknowledge them. In some cases, there are statistical methods to control for the influence of these threats.
One of the advantages to publishing the results of research is that school counselors do not have to reinvent the wheel. That is, we read about research in the hope that findings will generalize to other stu- dents, settings, and times. Generally, results should be replicated in other contexts so that it is not just a single study but a body of research that establishes the generalizibility of findings. Let us assume that the original study of the orientation program was conducted in a large, urban high school. Will the same program have the same outcome in a small, rural high school and with a different racial/ethnic composition?
SINGLE-GROUP PRETEST-POSTTEST DESIGN
School counselors have the advantage of conducting research in settings that reflect real students. Some programs of interest to counselors cannot be effec- tively studied in a laboratory; if they could be, we would question the external validity of the findings. Conducting research in a school also has disadvan- tages, not the least of which is the inability to con- trol many factors in a research process. For example, researchers may not be able to randomly assign stu- dents to classrooms, and they may have to contend with numerous historical events that occur during a research study. Nevertheless, the findings are clearly relevant and applicable to the school of interest.
One research strategy that is relatively uncompli- cated to do is a single-group pretest-posttest design. In this design, the school counselor implements a program (a series of guidance lessons or counseling groups to address a particular topic). Prior to start- ing the program, the students take a pretest so their baseline levels can be determined. The program is delivered, and then the students take a posttest. The improvement in scores from pretest to posttest is used to measure the impact of the program. At first glance, that seems to be a logical approach. One
360 ASCA I PROFESSIONAL SCHOOL COUNSELING
advantage of the pretest-posttest design is that one does not have to include a control group, and the pretest information allows the counselor to deter- mine differential effects (e.g., the lessons increased t6lerance in boys more than in girls).
However, this design is particularly vulnerable to the threats to internal validity described above. How can the counselor demonstrate that it was the pro- gram that caused the change in scores, and not mat- uration, history, testing, or regression to the mean? For example, let us imagine that the lessons were developed to increase tolerance toward physically handicapped students. During_ the time the weekly lessons were being presented, there was a television special on the topic that many students watched. Or perhaps there were classroom disruptions during the time the lessons were presented. Were the observed changes the result of the TV program, the disrup- tions, or the lessons?
COMPARISON OR CONTROL GROUP DESIGN
A more rigorous design that avoids many of the threats to internal validity inherent in pretest- posttest designs is the control group or comparison group design. A control group is a group of partici- pants who get no intervention; if a group gets a dif- ferent intervention, then we call it a comparison group. History and maturation will affect both the experimental and the comparison groups, so any dif- ferences in the outcome variable cannot be biased by those threats. Testing and regression to the mean also are going to influence both groups, so observed differences can be attributed to the intervention rather than these alternative explanations. Of the 50 school counseling outcome studies published between 1988 and 1995, only 26% used this design (Whiston & Sexton, 1998). The authors of the review concluded that more research of this kind is needed, and they recommended the wait-list control group strategy used often in other counseling research. In the school setting, this means that class- es, schools, or students who do not receive the intervention (program, activity) during the research period will receive it at a later time (the following semester, year, etc.).
There are several ways in which comparison groups can be created. The first is random assign- ment. That means that all eligible participants are randomly assigned to one of the experimental con- ditions (intervention, comparison group, control group). When this is not possible, the researcher can use preexisting groups (e.g., already formed or intact classes) that are matched on key variables, such as reading level or socioeconomic status. An investigation of the impact of a new "transition to
kindergarten program" might use current students in kindergarten as the experimental group and stu- dents from a previous year (now first graders) at the same school as the comparison group. The assump- tion in this case is that previous students resemble current students on the relevant characteristics.
A final method would be to use pretest scores to ensure that the groups are matched on key variables prior to the introduction of the intervention. After creating matched groups, the researcher then can randomly assign the groups to the intervention con- ditions. If the intervention might have a differential effect based on levels of test anxiety, the researcher can administer a pretest of test anxiety; create groups of high-, average-, and low-anxiety students; and create two groups with equal representation from the different levels of anxiety. Once the groups are created, a random procedure can be used to assign one group to receive the intervention (e.g., instruc- tion in progressive relaxation) and the other group to serve as the control group.
Random Assignment The most rigorous comparison group design utilizes random assignment to condition (experimental group or control/comparison group). Random assignment means that every participant has an equal chance of being in the experimental condition. Most statistical software packages include features that allow the researcher to randomize assignment in a scientific manner. There are also sites on the Internet that a counselor might be able to locate and use if such software is not readily available. In most edu- cational settings, it is usually not possible to ran- domly assign students to one or another class or program. However, random assignment can be accomplished by using classes or schools as sampling units. For example, in a study evaluating the effects of a new drug prevention curriculum for middle school students, if there is more than one school interested in participating, the schools can be ran- domly divided into two groups (using a number of randomization procedures), with one group desig- nated as the experimental group (the schools receiv- ing the curriculum) and the other as the comparison or control group (which will not receive the cur- riculum at this time). If only one school is going to participate, the same procedure can be applied to classrooms.
In some cases, it may not be possible to random- ly assign classrooms to the intervention or non- intervention groups. If there are two schools or two classrooms that are potential participants, and only one is interested in testing the curriculum, the other can become the comparison group. The problem with this method is that the two groups (schools, classes) may be different prior to the curriculum
9:5 JUNE 2006 1 ASCA 361
No Child Left
Behind and the
ASCA National
Model both
emphasize the
importance of
scientific, rigorous,
well-designed
research as an
essential
component of
modern school
counseling
programs.
implementation in ways that affect the outcome (e.g., intelligence, reading level). If, however, the researcher is able to administer the pretests and posttests to both groups (or obtain data on both groups), these differences can be identified and con- trolled for statistically. That means that the analyses can determine whether the obtained differences would exist over and above the influence of these potentially influential variables. If reading level were a possible confounding variable, the researcher can use a statistical analysis called analysis of covariance, in which reading level is designated the covariate. The results of this analysis xvill reveal whether observed differences in the dependent variable (the outcome) are significant when differences in reading level have been taken into account. The researcher can include more than one covariate if several attrib- utes are potential competing explainers of results.
Measurement Concerns How are variables measured? This is a basic question that researchers must address when designing the study. For the results to be valid, the measures must be reliable and valid. Reliability refers to the consis- tency of the scores, and validity relates to whether the measure measures what it purports to measure and for whom it does so. Researchers need to use considerable care in the selection of the instruments to be used. While researcher-designed question- naires may be used, it is essential to establish the reli- ability and validity of such measures. Published measures generally will report such data so that the researcher can make informed decisions. If other methods of assessment are used (such as observa- tion), those also must be evaluated prior to use, and they must meet the same standards of psychometric adequacy as paper-and-pencil measures. Some stud- ies in the school counseling field have used self- reported student grades as outcome variables. A more precise measure would be to use actual record- ed grades from student records. To take this a step further, grades may be influenced by the grading practices and standards of different teachers; achievement test scores might be a more valid meas- ure to use as a dependent variable.
Data Analysis The word statistics invokes fear and anxiety in many for whom research is not a frequent activity. Counselors need to know that in the age of com- puters, the task of analyzing data is far less daunting. Even without the specialized statistical programs used by most researchers, school counselors can uti- lize statistical features of Microsoft Excel and EZAnalyze (Poynton, 2005), an add-in for Excel. Using these tools, the school counselor can easily obtain descriptive data about the sample (including
means and standard deviations) and can disaggre- gate data by group (e.g., by gender or ethnicity). In addition, the school counselor can perform several analyses, including correlations (to assess the strength of relationships between two variables such as math and reading test scores), t tests (to test the significance of differences between two groups or pretests and posttests for the same group), and analyses of variance (to test differences among more than two groups). The counselor also can obtain tables and graphs directly from the program, allow- ing for visual presentation of results.
For more complex analyses, many school districts have research departments that can help. And many counselor educators at universities are eager to assist, and can do so even when located at a distance, using e-mail to receive data. Consulting with university researchers is a good idea throughout a research project when school counselors are novice research- ers, but it can be especially important during the data analysis and interpretation step. A very useful review of the various analysis options for comparison group designs can be found in Gliner, Morgan, and Harmon (2003).
Reporting Results Whether the purpose of the research is for program improvement or to comply with mandatory report- ing regulations, it is important to present the results clearly and accurately. Much of the audience will be unfamiliar with the terms used, so definitions are essential. It makes sense to begin by stating the research question at the outset, and then describing how you went about answering that question. For example, if the question was, "What is the effect of the new study skills group on student achievement?" you would begin by describing the study skills group and defining student achievement (e.g., overall grade point average, scores on achievement tests). Then the research design is described. For example, researchers often will write something like, "A pretest-posttest comparison group design was used, with last year's students serving as the comparison group [or another school with comparable demo- graphic variables]." The next step is to present results dearly, using tables and graphs when they enhance the presentation. Finally, you want to pro- vide the answer to your question, and discuss the implications of your findings. Any limitations of your research should be acknowledged.
School counselors need to disseminate their research to advance the profession. Presentations to stakeholders and administration are one method of doing so. A brief summary report of findings to the administration or school board, or a more detailed presentation, can educate these important groups about the interventions that counselors are using
362 ASCA I PROFESSIONAL SCHOOL COUNSELING
Regardless of the
design or method
of research, school
counselors must he
concerned with the
validity of the
research they
conduct or read.
and will demonstrate to these stakeholders that pro- grams have been scientifically evaluated for effective- ness. Writing articles for state and national journals in which you present a report of your research is an important contribution to the field. Knowledge builds on previous knowledge, and if research is not published, others will not have the benefit of the findings.
SOME EXAMPLES
To illustrate the control/comparison group approach to research, several studies have been selected for review. These will demonstrate the advantages and pitfalls of this strategy. In the inter- est of space, the details of statistical analyses -ill not be given below. Interested readers may refer to the original journal articles for that information.
Random Assignment-RIPP A school-based violence prevention program, Responding in Peaceful and Positive Ways (RIPP), was studied in three public middle schools in a Southern city (see Farrell, Meyer, & White, 2001). All regular education sixth-grade classes were poten- tial participants. Thirteen classes, including 305 stu- dents, were randomly assigned to the experimental group (receiving the program), and 14 classes (321 students) to the control group. The schools also implemented a school-wide peer mediation pro- gram, which was available to all students in the schools. The vast majority of participants were African American.
The RIPP program consisted of 25 sessions (50 minutes each) taught by trained prevention special- ists who were African American men. The lessons were presented weekly during social studies or health education classes over one school year. A manual followed by the presenters was used to increase treatment fidelity; researchers observed the implementation and provided an additional check of treatment fidelity.
In addition to pretest data obtained in October and posttest data collected in May of that school year, follow-up data were gathered 6 and 12 months after the completion of the program. These meas- ures were administered by research assistants who were "blind" to (did not know) the condition (inter- vention or control group) of the classes. Of the 626 students who began the study, complete pretest and posttest data were obtained from 474. Four hun- dred ten students were available at the 6-month fol- low-up, and 359 at the 12-month follow-up.
A variety of measures was used to assess the vari- ables of interest: frequency scales for problem behav- iors, violent behaviors, and drug use; a RIP-P multi- ple-choice knowledge test, a problem situation
inventory, and two scales assessing relevant attitudes (beliefs supporting aggression, attitudes toward con- flict). Reliability and validity data are given for all measures, and all were acceptable. Demographic data also are available, include disciplinary code vio- lations.
Analyses included a comparison of experimental and control groups on demographic variables of gender, age, and ethnicity. No differences were detected between the groups. Because some stu- dents did not complete the entire program, the researchers examined the effect of attrition on the two groups. Their analyses determined that attrition affected both the experimental and control groups in a similar way. Analyses also investigated differ- ences on the pretests between experimental and con- trol groups. There were no differences between the two groups on disciplinary referrals, although differ- ences were detected by age and gender (the older the student, the higher the rate of disciplinary refer- rals; boys also were more likely than girls to have violations). No differences were found on specific violent behaviors. In fact, the only difference found at pretest was the higher incidence of positive atti- tudes toward nonviolence in the control group.
The researchers then analyzed the differences at posttest. Although many of the results were in the predicted direction, statistically significant differ- ences between the intervention group (which received the RIPP curriculum) and the control group were found on the number of disciplinary vio- lations for violent behavior (the control group had 2.2 times as many) and the rates of in-school sus- pensions (the control group had 5 times as many). The intervention participants used peer mediation more frequently at posttest than did the control par- ticipants and also reported fewer fight-related injuries at posttest. Differences at follow-up time periods (6 and 12 months post-intervention) were in the expected direction but were not significant, with the exception of the rate of in-school suspen- sions at 12-month follow-up, which was three times greater for boys only.
Further analysis revealed an important finding: The participants who reported high levels of vio- lence on the pretest had lower scores at 6- and 12- month follow-up than did comparable participants in the control group. Results also indicated that while students in the intervention group demon- strated increased knowledge of the material in the curriculum, they did not show any change in their attitudes or on the use of nonviolent responses in hypothetical situations. The authors speculate that because knowledge improved but attitudes and skills did not, there may not have been adequate support for the use of these skills in the school environment. This is important for counselors: If they teach skills
9:5 JUNE 2006 1 ASCA 363
to students, the school community must endorse the use of the skills, reinforce the skills, and encourage
students to apply them in the school setting. Although this study was one of few conducted in
schools using random assignment and a large sam-
ple, the research was somewhat compromised by the nature of the design. With some classes in each school receiving the curriculum, it is difficult to
ensure that the students in the control condition
were unaware of the program and did not learn
some of the material via modeling from peers in the
intervention condition. In those cases where the
results "approached significance," this limitation
may have had an effect, and it in fact may have influ- enced other results as well.
Random Assignment-SSS Researchers (Brigman & Campbell, 2003) tested the Student Success Skills (SSS) on a random sample
of 180 students in four grades at six schools in a
Southern state. The SSS program is an intervention in which school counselors teach skills found to pre-
dict school success using effective teaching strategies
for classroom guidance and group counseling com- ponents. Students in the participating schools who
scored between the 25th and 50th percentile on a
state assessment test were eligible to participate.
Thirty students from each school were randomily
selected for participation. The comparison group
was created by matching participating schools with
nonparticipating schools based on geography, race, and socioeconomic status and then randomily select-
ing students. In the study, the outcome (dependent)
variables were math and reading scores on a stan-
dardized test and teacher ratings of classroom
behavior. The authors reported the psychometric properties of all the measures used.
In order to ensure that the counselors were pro-
viding the program correctly, counselors received three days of training and three half-day follow-up
training sessions. Half-day peer coaching sessions
were scheduled during months when follow-up
training did not occur. The group counseling com-
ponent consisted of eight weekly 45-minute sessions
and four booster sessions using a structured format
for teaching cognitive, social, and self-management
skills. The number and frequency of classroom guid- ance lessons were not specified for each site, but at
least three classroom guidance lessons were to be delivered at each grade level. Five of the six schools
met the established criteria for implementation guidelines.
Results showed statistically significant differences
between the intervention and control groups on the
math and reading posttest scores. In this case, the
size of the difference between the two groups is
large. The behavior scale was not administered to
the comparison group, but authors noted that 70% of participants demonstrated an average of 22 per- centile points improvement in behavior between September and April administrations of the behavior rating scale.
The use of the comparison group was essential for these results to have an impact, as any improvement found for the intervention group could have been the result of history, maturation, or regression to the mean. Although different idiosyncratic historical influences may have affected results, the findings are strengthened by the use of the comparison group. The researchers noted that the absence of the behav- ior ratings for the comparison group was a limita- tion. In addition, it is not known how this interven- tion compares to other interventions used to increase student achievement. This study demon- strates how counselors can use research to demon- strate that their interventions positively affect stu- dent achievement.
To further document the impact of this interven- tion, the researchers have published two replications of this study (for details, see Webb, Brigman, & Campbell, 2005, and Campbell & Brigman, 2005). Replication is a way to demonstrate that a research study conducted using the same intervention, pro- cedures, and measures yields similar results in other settings or contexts. In this case, while results were similar in the replications, all three studies' partici- pants were predominantly European American, and it remains to be demonstrated that the intervention is successful with more diverse students.
Using Both Comparison and Control Groups Both a comparison group and a control group were used to investigate the effects of a social cognitive group on self-esteem and difficulties in peer rela- tionships among adolescents aged 13-16 years (Barrett, Webster, & Wallis, 1999). In order to be certain that the effects of the intervention could be attributed to the intervention and not just the atten- tion received as a result of being in a group, the researchers used a comparison group. This group received the special attention of being in a group, but instead of the program using skill development process and procedures, the comparison group was given similar information in a didactic fashion using lecture and films. A wait-list control group also was used.
To place the 51 participants (students recom- mended by teacher nomination) in one of the groups, the researchers matched the students on gender, age, and scores on the three pretests (meas- ures of self-esteem, self-related cognitions, and social competence with peers). Thus, the design ensured that the three groups were comparable prior to the interventions. Measures were described and the psy-
364 ASCA I PROFESSIONAL SCHOOL COUNSELING
This is an optimal
time for counselor
educators and
researchers to
collaborate with
school counselors in
order to enhance
the credibility of
the school
counseling
profession.
chometric properties were adequate. The researchers used change scores from pretest
to posttest to assess the impact of the intervention. The experimental group showed significantly greater improvement in self-esteem and self-related cogni- tions and perceptions than did both the comparison and control groups. Also detected was a significant increase in the self-reported interpersonal difficulties with peers in the intervention group. The researchers speculated that the increased difficulties may reflect a concern with ecological validity. That is, while the participants may be able to use the new skills within the program, they were not able to transfer those skills to their daily environment. This is a useful finding: Programs in schools need to include components to help apply the skills in the participants' natural environment.
Although this study utilized a relatively small con- venience sample from one school, and it did not incorporate a follow-up component, the employ- ment of both comparison and control groups is a useful feature given the nature of the intervention, and one that counselors might consider for other research questions.
CONCLUSION
This article has discussed how comparison and con- trol group designs strengthen research. Well- designed research studies that examine the effective- ness of school counseling programs and practices contribute to a growing body of evidence that sup- ports the positive impact of school counselors on academic achievement. Although there has always been a need for school counselors to be accountable, the current emphasis on data in education makes this need particularly salient. The urgency of the sit- uation is evident from Whiston's (2002) observa- ton:
In my opinion, this is a critical time for leaders in school counseling to invest in the future of the profession and support school counseling research. School counselors may believe they make a difference, but without "hard data" to support these claims, school counselors run the risks of losing their positions. (p. 153)
This article assists school counselors and researchers by describing an approach that produces the most robust findings.
To ensure both internal and external validity, school counselors must attend carefully to all the elements of their study: They must choose the best measures (e.g., achievement test scores vs. grades, objective testing vs. self-report when possible) that have adequate reliability and validity. They must
ensure that data are analyzed in the most effective manner (e.g., using statistical methods to control for possible pre-intervention differences in groups). They must clearly describe the population and con- text from which participants were selected, explain procedures for assignment to groups (with random assignment being the ideal), and articulate how the researchers ensured that the intervention was deliv- ered correctly in the same way to all groups.
School counselors have an ethical obligation to seek evidence of their effectiveness. Principle A.9.g of the Ethical Standards for School Counselors (ASCA, 2004) maintains that a professional school counselor assess "the effectiveness of his/her pro- gram in having an impact on students' academic, career and personal/social development through accountability measures especially examining efforts to close achievement, opportunity and attainment gaps." It may seem that the pressure to produce sci- entific evidence is unreasonable, given the many duties that most school counselors perform. It is important to remember that without such evidence, school counselors may have difficulty justifying their role in a system that is increasingly data-driven.
This is an optimal time for counselor educators and researchers to collaborate with school coun- selors in order to enhance the credibility of the school counseling profession. Together they can produce the high-quality research that is necessary for both accountability and advancement of the field. I
References Allen, J. (1992). Action-oriented research: Promoting school coun-
selor advocacy and accountability (ERIC Digest No. ED347477). Retrieved June 9,2002, from http://www.ed. gov/databases/ERIC_Digest/ed347477.html
American School Counselor Association. (2004). Ethicalstan- dards for school counselors. Alexandria, VA: Author. Retrieved August 21,2005, from http://www. schoolcounselor.org/files/ethical%20standards.pdf
American School Counselor Association. (2005). The ASCA national model:.A framework for school counseling pro- grams (2nd ed.). Alexandria,VA: Author.
Barrett, P. M.,Webster, H. M., & Wallis, J. R. (1999). Adolescent self-esteem and cognitive skills training: A school-based intervention.Journal of Child and Family Studies, 8, 217-227.
Bauman, S. (2004). School counselors and research revisited. Professional School Counseling, 7, 141-151.
Brigman, G., & Campbell, C. (2003). Helping students improve academic achievement and school success behavior. Professional School Counseling, 7, 91-98.
Brown, D., &Trusty,J. (2005). School counselors, comprehensive school counseling programs, and academic achieve- ment: Are school counselors promising more than they can deliver? Professional School Counseling, 9, 1-8.
Campbell, C., & Brigman, G. (2005). Closing the achievement gap: A structured approach to group counseling.Journal for Specialists in Group Work,30, 67-82.
9:5 JUNE 2006 1 AsCA 365
Campbell, D.T., & Stanley, J. C. (1963). Experimental and quasi- experimental designs for research. Chicago: Rand McNally.
Cramer, S. H., Herr, E. L, Morris, C. N., & Fra ntz7T.T. (1970). Research and the school counselor. Boston: Houghton Mifflin.
Farrell, A. D., Meyer, A. L, & White, 1 S. (2001). Evaluation of Responding in Peaceful and Positive Ways (RIPP): A
school-based prevention program for reducing violence among urban adolescents.Journal of Clinical Child Psychology,30, 451-463.
Gliner, J. A., Morgan, G. A., & Harmon, R.J. (2003). Pretest- posttest comparison group designs: Analysis and inter- pretation.Journal of the Academy of Child and Adolescent
Psychiatry, 42, 500-503. Lee, C. C., & Workman, D. J. (1992). School counselors and
research: Current status and future direction. School
Counselor, 40,15-19. Loesch, L. C. (1988). Is "school counseling research" an oxy-
moron? In G. R.Walz (Ed.), Research and counseling: Building strong school counseling programs (pp. 169-180).
Alexandria, VA: American School Counselor Association. McGannon,W., Carey, J., & Dimmitt, C. (2005). The currentstatus
of school counseling outcome research (Research Monograph No. 2).Amherst, MA: University of Massachusetts, School of Education, Center for School Counseling Outcome Research.
No Child Left Behind Act of 2001, Pub. L No. 107-110,115 Stat. (2002).
Poynton,T.A. (2005). EZAnalyze (Version 2.0) [Computer soft- ware]. Retrieved August 21,2005, from http://www. ezanalyze.com
Sink, C.A. (2005). Comprehensive school counseling programs
and academic achievement-A rejoinder to Brown and
Trusty. Professional School Counseling, 9, 9-12. Webb, L D., Brigman, G.A., & Campbell, C. (2005). Linking
school counselors and student success:A replication of
the Student Success Skills approach targeting the aca- demic and social competence of students. Professional School Counseling, 8,407-413.
Whiston,S. C. (2002). Response to the past, present, and future of school counseling: Raising some issues. Professional
School Counseling, 5, 148-155. Whiston, S. C., & Sexton,T. L. (1998). A review of school counsel-
ing outcome research: Implications for practice.Journal of Counseling & Development, 76,412-426.
Wilson, N. S. (1985). School counselors and research: Obstacles and opportunities. School Counselor, 33, 111-119.
Earn CEUs for reading this article. Visit www.schoolcounselor.org, and click on Professional School Counseling to learn how.
366 ASCA I PROFESSIONAL SCHOOL COUNSELING
COPYRIGHT INFORMATION
TITLE: Using Comparison Groups in School Counseling Research: A Primer
SOURCE: Professional School Counseling 9 no5 Je 2006 PAGE(S): 357-66
WN: 0615205985007
The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited.
Copyright 1982-2006 The H.W. Wilson Company. All rights reserved.
Get help from top-rated tutors in any subject.
Efficiently complete your homework and academic assignments by getting help from the experts at homeworkarchive.com