A set of criteria used in clinical judgments about program participants was first established in order to systematize judgments among a diversity of staff--so that staff would be making judgments on a common basis. Eight batterer counselors and seven battered women's advocates were convened in two separate focus groups. Using conventional focus group methodology ( Krueger, 1994 : Morgan, 1988 ), 13 criteria were identified as universally recognized in making judgments about the performance of program participants. These were behaviors exhibited in the sessions that program counselors looked to as indicators of a "successful" outcome. In other words, they were the implicit expectations for program performance.

The batterer counselors and battered women advocates then rated the criteria (13 items with operational definitions) using a Likert scale as to their importance and weight in their own judgments (see Appendix). The average rating for each item and a confirming factor analysis of these ratings were used to reduce the list to 10 items. An analysis of this 10-item list of discharge criteria was conducted later using the counselor ratings of 164 program participants when they left the program. A factor analysis produced 1 factor incorporating all variables with an Eigen value of 7.53. A reliability test produced an Alpha coefficient of .97, indicating a uni-dimensional and internally consistent instrument.

The set of criteria and its testing are derived from counselors and participants of a court-mandated batterers program in Pittsburgh which receives over 1,000 court referrals per year and maintains 18 groups of 12-17 participants in each group. The participants are required by the court to attend a minimum of 13 weekly group sessions (1 intake session, 2 orientation sessions, and 10 group discussions sessions). The program curriculum approximates what might be termed a gender-based cognitive-behavior approach reflecting the prevailing model programs in the field (e.g., Pence & Paymar, 1993 ; Stordeur & Steule, 1989 ). Men are dismissed from the program and returned to the court if they have two unexcused absences or they reportedly reoffend (i.e., are physically violent during the course of the program).

Participant Ratings: Five counselors (3 men and 2 women) rated each program participant two times: 1) when the participants entered their assigned weekly discussion group following the intake and orientation sessions (initial rating); and 2) when the participants finished the program by completing the required 13 weeks, or by either dropping out or being dismissed (final rating). The counselors rated the participants using the discharge criteria form (see Appendix) by indicating a Likert response (1-5 ordinal scale) for each of the 10 criteria. The counselors did not have direct access to any background or assessment information beyond their own observations of the participants' performance in the group (with an exception noted later). The men's participation in 3 sessions (1 intake and 2 orientations sessions) prior to the initial rating in the first discussion group, we believed, would reduce distorted first impressions resulting from the batterers' cautious, resistant, and hostile behavior associated with the initial program contact following the court-mandate. The sample of rated men was comprised of 164 program participants who were consecutively enrolled in the batterers program over a 5-month period (October 1993 to March 1994).

Follow-up Interviews: Research staff attempted to contact the rated participants between 9-11 months after their initial contact with the program or anequivalent to 6-8 months after their scheduled completion of the 3-month program. The researchers made a maximum of 6 calls to a participant over a 3-week period before considering a participant unreachable. A response rate of 40% was achieved for a follow-up sample of 65 men from the original 164. Thirty percent were not reached after the maximum number of calls; 24% had disconnected phones with no forwarding information; 6% were contacted but refused to be interviewed. This response rate is comparable to that achieved in similar program follow-up studies (see Tolman & Bennett, 1990 ). The follow-up sample, as might be expected, was biased in favor of program completers: 73% of the follow-up sample (n=64) completed the program, as opposed to only 62% of the initial sample of rated men (n=164).

A subsample of the men's partners was also contacted as a means to verify the self-reports of the program participants. A random sample with replacement was derived for one-third of the men in the follow-up sample (n=22). The women were called separately from the men at a time when the men were not present. The size of the subsample of women was set in order to be both economical and sufficient for verification (see below). The subsample of women did manage to reflect the characteristics of the follow-up sample of men; 70% were with partners who completed the program, for example, and 72% of the follow-up sample of men were program completers.

Variables: The follow-up interviews consisted of a structured interview schedule administered over the phone and addressing the status of the relationship; the use of abusive or violent behavior; additional treatment, intervention, or services; and alcohol intoxication or drunkenness during the follow-up period. These variables were assessed primarily using inventories like the Conflict Tactics Scale ( Straus, 1979 ) for abuse and violence, a list of help-seeking options ( Gondolf and Fisher, 1988 ) for additional treatment, and a frequency of the number of times "drunk or high." The respondents were also asked to estimate, using a Likert scale, the man's likelihood of hitting his female partner within the next 3 months ("How likely is it that you will hit your partner in the next 3 months?"). The variables were collapsed into dichotomous categories reflecting the presence or absence of a variable (e.g., abuse, treatment, intoxication) in order to accommodate the small sample size available for the data analysis (see Table 1).

The principal outcome measure was any report of abuse. Non-physical abuse was defined by an inventory of controlling behaviors and verbal abuse (swearing or screaming at the woman; stopping her from going some place, following her against her will; throwing, kicking or hitting something; and threatening the woman in any way). Physical abuse was defined by the categories of the Physical Aggression portion of the Conflict Tactics Scale ( Straus, 1979 ) (pushing, grabbing, shoving, slapping, hitting, punching, kicking, choking, using a weapon, or forcing sex). In order to expand the base rate of the outcome, we constructed a variable of "abuse in general" that combined the non-physical abuse and physical abuse, and a variable of "severe abuse" that consisted of a report of any physical abuse or threats, or any arrest or hitting of any other persons.

The "abuse" and "severe abuse" variables used in the analysis were based on the program participants' self-report and reinterpreted using the subsample of women's reports. The reports of the men for each type of abusive behavior were compared to the report of their female partner (n=22 men and n=22 women or 33% of the follow-up sample of 65). Only one-third (35%) of the men whose partners reported them being "severely abusive," reported severe abuse: 14% of the men reported "severe abuse," as opposed to 41% of their women partners. These men, however, reported other forms of less severe abuse. Nearly half of the men (51%) reported "abuse in general" but about a third (31%) of these men did not commit any acts considered "severe abuse" according to their partners. Therefore, we project for the follow-up sample (n=65) that the men's report of "abuse in general" over-represents the incidence of severe violence by approximately 15%. "Abuse in general" is, however, closer to the actual level of severe abuse than the men's report of "severe abuse," which underestimates the incidence of severe abuse by 27%. The men's report of "abuse in general" might be used as a proxy for actual "severe abuse" in this preliminary exploration of discharge criteria. Although "severe abuse" underreports the incidence of violence, it offers a verified indicator of the most severe cases of abuse.

We first examined variations in ratings of the discharge criteria among the 5 batterer counselors. On the initial ratings, the men counselors rated the men significantly lower than the women counselors (M=1.4 [d=.56] vs. M=2.1[sd=7.5]; diff.=.73 [t=6.5;df=133; p is less than .001; n=164]). In discussing this difference with the counselors, we found the men counselors also conducted the court intake and orientation sessions which exposed them to the program participants at their most resistant and hostile period and gave them additional information on the extent of the men's previous abuse. The average ratings of the men and women converged in the final ratings. However, a slight but insignificant difference does exist between the ratings of the full-time permanent staff and part-time contract counselors at the final ratings (M=3.12[sd=1.04] vs. M=3.41[sd=1.17]; n.s.). The permanent program staff received additional information through phone calls and complaints of which the contract counselors did not have access. In sum, the ratings appeared to be fairly consistent among the counselors except for predictable differences; therefore, the rating did not need to be converted to standardized scores or Z-scores. Both the initial and final ratings were normally distributed rather than bimodal in a "good guys" and "bad guys" split. The total final ratings were collapsed into low and high categories at the median rating (low is less than 3.49 and high is more than 3.5; 50% of the sample in each category) for use in the cross-tabulations with the outcome variables. The total ratings were computed by summing the item Likert ratings (1-5) and dividing that sum by the total number of completed items minus those items that received a zero for uncertain or not applicable.

The average ratings for each of the 10 items on the discharge criteria form significantly increased from the initial rating to the final rating (n=164). This would be the expected change in response to participation in a program that expects or addresses these criteria. The greatest increases were with the items "using techniques" and "acceptance of a problem," which are the most dependent on program instruction and at the heart of the curriculum. The averages for the total ratings significantly increased from the initial to the final rating as expected in response to program participation (M=1.82 [sd=.70] vs. M=3.24 [sd=1.09]; diff.=1.43 [t=12.35; df=135]; p is less than .001). The ratings for the men increased one full category (on average) from the criteria being "a little present" to "somewhat present," suggesting positive but limited change in the men overall.

No significant relationship is evident between the initial ratings and program completion, according to a cross-tabulation of low-high rating categories and completion/dropout (n=164). Slightly more than half (54%) of these cases were correctly classified. This finding suggests the limitation of first impressions in assessing program participation and making triage or lethality decisions. This finding may reflect the men's initial resistance to being court-ordered to a program and also the need for extended observation in order to make predictive clinical judgments.

The final ratings were highly associated with the program completion or dropout with a 78% correct classification rate (n=164). Forty-four percent of those men with low ratings completed the program, and 100% of the men with high ratings completed the program (X2=71.46; df=1; p is less than .001). This association may reflect the fact that the completers were exposed to more of the program and tend to be less anti-social than the dropouts ( Grusznski & Carrillo, 1988 ; Hamberger & Hastings, 1989 ). It also may reflect the counselors' tendency to confirm the participants' final status: the counselors give program completers higher ratings in response to their accomplishments while attending.

The final ratings were not significantly associated with the men's reporting of "severe abuse" after the program. This finding reflects the small base rate of reported severe abuse and the consequent large number of false positives. However, the correct classification of severely abusive men is fairly high for what is a relatively crude measure at 78%. Seven men who reported severe abuse were rated low on the discharge criteria, while only two who reported severe abuse were rated high. The correct classification rate overall was 61%. In sum, the so-called sensitivity of the discharge criteria is fairly high (e.g., the ability to predict or identify eventual abuse).

The final ratings were not significantly related to the "abuse in general" variable, perhaps because this variable is confounded by other factors as later findings suggest. A tendency in the expected direction does emerge when men who are separated from their partners are partialed out in a three-way cross-tabulation between the final ratings and abuse in general controlling for separation. Sixty percent (6) of the men with low ratings vs. 33% (4) with high ratings reported abuse during the follow-up period (n=22). Also, the expected tendency between rating and abuse appears when "being drunk during the follow-up period" is partialed out in a three-way cross-tabulation. Forty four percent (9) with low ratings vs. 10% (1) with high ratings were abusive during follow-up (n=19). The lack of statistical significance for the Chi square test in the three-way cross-tabulations reflects the small sample size, further compartmentalized by the additional level of tabulation. Nonetheless, it is apparent that the specificity of the discharge criteria is substantially increased when controlling for the otherwise confounding factors of either separation or drunkenness during follow-up.

One additional, related outcome variable was examined: the man's estimate of the likelihood of his hitting his partner in the next few months. The likelihood of hitting was significantly associated with a low-medium-high categorization of the final ratings of discharge criteria. Seventy-eight percent of the men rated low versus 42% with a medium rating and 89% who were rated high said it was "extremely unlikely" that they would hit their partners (X2=27.06; df=2; p is less than .05). The apparent portion of false negatives (the 78% rated low who were extremely unlikely to hit) is confounded by the fact that a disproportionately high percentage of these men are "unlikely" to hit their partners because they are not living with them. Otherwise, the highly rated men are extremely unlikely to hit their partners as predicted. This association suggests that the clinical judgments correspond to the men's estimates of their own potential for violence. The potential for violence may in fact be a better indicator of the men's dangerousness than the abuse reported in follow-up, since the abuse variable is so confounded by other factors such as access to the victim and level of drinking.

A preliminary examination of influential factors confirms the role of intervening variables in the program outcome. Twenty-seven percent of those men who were not drunk during the follow-up versus 65% of the men who were drunk reported being abusive, for an overall correct classification of 68% (X2=8.81; df=1; p is less than .01). A logistic model that included the final ratings along with separation and treatment increased the classification only to 73%, confirming the primary influence of the drunkenness. The percentage of men reporting abuse during the follow-up period was not significantly different for those not separated versus those who were separated (53% vs. 45%), but the low rated men were more likely to be separated. Participating in some additional treatment also did not significantly relate to the abuse outcome; however, only 6 men received other treatment and 5 of these men were not abusive. These preliminary findings suggest that intervening factors or "conditions" may confound clinical judgment and that the "conditional predictions" may increase the accuracy of clinical judgment, as the mental health research on dangerousness indicates ( Lidz et al., 1993 ).

Our examination of the discharge criteria ratings suggest respectable sensitivity but not specificity in predicting abusive behavior after the program. It appears likely that the correct classification of these ratings--particularly their specificity-- would be substantially increased if intervening variables such as drinking and separation were controlled. Drunkenness during follow-up emerged as a significant predictor of abuse. The clinical judgments implied in the criteria ratings may relate most to men's estimates of their own potential for violence--that is, the participants' judgments which may weigh the conditions influencing their actual abuse. Overall, our preliminary evidence supports the utility of clinical judgments based on performance criteria. The discharge criteria examined in this study are not meant to be predictions of dangerousness but rather criteria associated with clinical judgments about a participants' successful completion of a batterer program. However, predictive studies are one way to test the relevance and validity of such judgments. The sensitivity of the criteria offers grounds for exercising clinical judgment in discharge, and points to some criteria beyond attending some arbitrary number of sessions. Moreover, the convergence of ratings among counselors, the change in initial and final ratings over the course of program participation, and the relationship of final ratings to program completion or dropout, all support the validity of the discharge criteria and a role of clinical judgment in discharge decisions.

Several methodological shortcomings remain in this examination of clinical ratings and in research on clinical judgment in general ( Mulvey & Lidz, 1993 ). The base rate of violence or abuse is too small to develop significant or complex analyses. We have attempted to alleviate the small base rate by constructing a broad but confounded category of abuse in general. The validity of self-report remains an issue for the primary outcome variable of interest: abuse and violence. We did attempt to reinterpret the men's self-report by using a subsample of women's reports, but encountered an over-reporting of "severe abuse" in the proxy variable of "abuse in general." The small follow-up sample, resulting from the difficulty in contacting program participants, is problematic with batterer program outcome research in general. We attempt to compensate for the small sample by using collapsed dichotomous variables, but the categories remain crude indicators of what is complex behavior. We are consequently in the process of attempting to replicate and elaborate the current results as part of a multi-site program evaluation incorporating 800 program participants, more precise measures and indicators, extensive tracking procedures, and periodic follow-up interviews (every 3 months over a 15-month period) and substantiating medical and police records ( Gondolf, 1994 ). This research design should alleviate many of the methodological shortcomings mentioned above.

Several unresolved practical and conceptual issues remain. The translation of clinical judgment into a set of fixed criteria may distort a more complex decision-making process, as recent research on dangerousness among psychiatric patients suggests. The clinical significance of the discharge criteria ratings is an unanswered question. For instance, what level of rating would warrant retaining a man in a program as opposed to releasing him? One program director lamented what she saw as yey another checklist to be misused. Discharge criteria could be inadvertently used to "certify" some men as a "good bet" or condemn others as "bad bets." Finally, program directors have pointed to the problem of lethal false negatives that faces such ratings or classifications. There have been threatening cases of men who "look good" in programs but who have gone on to attempt or commit murder, similar to the "overcontrolled" murderers referred to in homicide typologies ( Kalichman, 1988 ).

Some of these concerns might be addressed by comparing predictions of program counselors to the predictions of female partners of the batterers. Advocates have argued that battered women must daily make judgments about their safety and in a sense rate their partner's behavior ( Gondolf & Hart, 1994 ). The discharge criteria developed for this study might be compared to criteria from other programs and counselors to assess further the reliability of the criteria. Do programs with differing philosophies or modalities have substantially different criteria? Finally, the predictiveness of the clinically-based discharge criteria might be compared to, and possibly combined with, that of actuarial models and to combined decision-making models to determine whether other means of "judgment" are superior (see Gottfredson & Gottfredson, 1988 ).

Our development and testing of discharge criteria appears to support the use of clinical judgment in batterer programming. Our original notion was that discharge criteria might, at the minimum, be used as a matter of record and recommendation to courts, advocates, female partners, and the men themselves. It could serve as the basis for post-program planning--which might include continued participation in a batterer program at some level. Other program researchers, upon reviewing this study, suggest that the discharge criteria be used periodically throughout the program to help counselors evaluate the progress of participants and to revise their programmatic response to them. The ratings might also be used as a form of feedback to or "grading" of participants in order to identify areas the men might improve. The criteria could be used as the basis of a program contract for participants or a kind of mid-term review. Additionally, the discharge criteria could be useful in program outcome studies. It offers a measure of valence to in addition to the use of attendance for assessing the central variable of program participation.

In sum, there are grounds for considering clinical judgment in participant discharge from batterer programs, despite the previous reluctance to do so in the field. The discharge criteria developed for this study appear to be associated to some degree with program outcome. The criteria have substantial sensitivity when it comes to identifying abusive behavior after the program, and would likely show adequate sensitivity in identifying men who are not likely to be abusive when conditions such as drinking and separation are taken into account. The harbored observations and concerns of batterer program counselors may, in fact, warrant some systematic outlet. They may merit a primary role in determining the discharge of men from batterers programs.