47
THEORY-DRIVEN EVALUATION
4 Keywords evaluability assessment approach program impact theory service utilization plan organizational plan
CHAPTER OUTLINE
Introduction 47
Evaluability Assessment Approach 49
Describing and Producing Program Theory 50
Program Impact Theory 52
Service Utilization Plan 53
Program Organizational Plan 53 Step 1: Defi ne Boundaries 54
Step 2: Explicate Program Theory 55
Step 3: Defi ne Program Goals and Objectives 55
Step 4: Describe the Program Functions, Components, and Activities 56
Step 5: Final Corroboration of the Description of the Program Theory 56
Analyzing Program Theory 57
Link between Program Theory and Social Needs 58
Evaluating Logic and Plausibility of Program Theory 59
Comparing Research with Practice 60
Summary 61
Discussion Questions 61
References 61
Introduction In this chapter, we focus on the concepts and procedures neces-
sary for a criminal justice program evaluator to examine the concep-tualization of a program, also known as program theory . Program theory may be expressed implicitly or explicitly. Either way, it explains why a program does what it does and provides the ratio-nale that doing so will achieve the expected or desired results. At the outset of evaluating program theory, criminal justice evaluators may
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

48 Chapter 4 THEORY-DRIVEN EVALUATION
be dismayed by the incomplete or nonexistence of program theory. Many criminal justice programs are poorly designed with errors that can be traced back to a program’s conceptualization. Th ese errors are the result of neglecting the objectives and theory in the planning stages, or implementation, discussed in Chapter 3 . Th is is not always the fault of program planners. Sometimes the political climate sur-rounding criminal justice programs does not allow for extensive plan-ning. Unfortunately, when this is not the case, the conceptualization of the program theory gets little attention.
In criminal justice, this may be the case. Many times, programs are borne from familiar or “off -the-shelf” services without a clear understanding of the match between the program and services ( Coryn, Noakes, Westine & Schroter, 2011 ). In other words, the match between the program and services and needs is not clear. For instance, crime programs generally have a mix of education and counseling. Th e unfortunate issue is that the underlying assump-tions—individuals will change their behavior due to education or counseling—are not discussed or made explicit. Evidence has shown that criminal behavior is often resistant to change from education or counseling. Th erefore, using only this as an underlying set of assump-tions may not produce the desired results.
Th e rationale of a program and conceptualization deserves a sub-stantial amount of scrutiny in an evaluation. Th e scrutiny should result in an understanding of the criminal justice program’s goals and objectives. Welsh (2006) argues that program objectives must be clearly specifi ed and, optimally, be measurable. However, if these goals and objectives do not match the needs that the program is designed to improve, at best, the program can only be expected to be marginally eff ective ( Welsh & Harris, 2004 ).
A necessary function in assessing the program theory is to artic-ulate it—that is, make sure that the descriptions of concepts (i.e., abstractions from reality), assumptions (i.e., hypotheses), and expec-tations (i.e., ideas for outcomes) are in congruence with the manner in which the program is structured and operated. It is a rare case that a criminal justice program will reveal its theory to an evaluator. While implicit, this occurs because the entire statement of the program the-ory is rarely written down. For instance, a program designed to pre-vent juveniles from joining gangs called Gang Resistance Education and Training (GREAT) has a nine-lesson curriculum that is delivered to middle-school children. Esbensen & Osgood (1999) argue that the curriculum was not developed with any specifi c theory in mind, but they state that curriculum does have its basic etiology from crimino-logical theory. Even when it is written, the explication of the program theory comes from an abstract grant or funding proposal that has not been consulted during implementation or program practice.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 49
Criminal justice program evaluators have an important task to understand and clearly delineate the program theory in a manner that can be used for an evaluation. Th is chapter will use two views as guides to help criminal justice program evaluators understand and clearly delineate program theory. First is presenting program theory in a way that represents stakeholders’ views and understanding of the program that makes it workable for a program evaluation through the evaluability assessment approach. Second is describing a program theory for diff erent parts of a program.
Evaluability Assessment Approach Th e fi rst approach is a presentation of an evaluability assess-
ment. An evaluability assessment refers to a method of identifying and describing the program theory by outlining its components and deciphering which of them is measurable ( Welsh, 2006 ). Van Voorhis, Cullen & Applegate (1995) argue that evaluability assessments are designed to ascertain if a program has a sound theoretical basis, has a well-designed treatment protocol, has been implemented as designed, and is suitable for further inquiry such as an outcome evaluation.
Th is type of assessment is important for several reasons. First, the program theory may or may not be clear. Second, the program staff may or may not have diff ering views about the program theory. Understanding the program theory and the views of the program’s staff are important for the evaluator. Knowing this information, the evaluator is able to sort out with the stakeholder the program theory that can be used to evaluate the program.
Performing an evaluability assessment can take many forms. Van Voorhis & Brown (1996) argue that an evaluability assessment requires four steps: 1. Identify the purpose and scope of the assessment. 2. Develop a program template that describes the goals and objec-
tives of the program, the theory underlying the program, and the intended treatment protocol.
3. Validate the program design through interviews and focus groups with program staff members and stakeholders and through obser-vations of program activities.
4. Prepare a report that details the assessment fi ndings and provides the proper recommendations for future evaluation or program improvements. Welsh (2006) takes a diff erent focus in outlining the process for an
evaluability assessment. He argues that the evaluator should do the following: 1. Review the documentation that describes the program. 2. Interview administrators and line staff members.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

50 Chapter 4 THEORY-DRIVEN EVALUATION
3. Read case fi les. 4. Interview program staff .
Th e evaluability assessment may provide information to improve the program. Smith (1989) argues that evaluability assessments gen-erally show the problems within a program, including: ● Th e need to clarify the target population. ● Th e need to reconceptualize the intervention. ● Th e realization that few objectives are agreed upon by stakehold-
ers and program staff . Th e evaluability assessment can make it clear that more work
is necessary to fi ne tune the program. Welsh (2006) argues that this type of assessment results in a program model that can be reviewed by program administrators and staff for accuracy. Regardless, if the intent is to arrive at information from Smith’s (1989) perspective or Welsh’s (2006) perspective, the evaluability assessment may help pro-gram staff deal with diff erences.
While this type of evaluation may be used for a single program, Matthews, Hubbard & Latessa (2001) see evaluability assessment as a means of improving correctional programming on a large scale. Using the Correctional Program Assessment Inventory (CAPI), Matthews et  al. (2001) examine whether 86 treatment programs are meeting the principles of eff ective intervention. Th e results of this assessment show that 34.1% of the programs studied were unsatisfac-torily meeting the principles of eff ective intervention. Th e main prob-lem with these programs is that they lack integrity. In other words, the evaluability assessment shows that the programs are not being implemented properly. While this perspective is good for improve-ments, it does not necessarily focus on a description of the program theory. Th is perspective, rather, uses the program theory as part of the evaluability assessment to determine the problems with the program. In other words, the program theory may be lost using this model. An evaluability assessment may be fruitful for eliciting pro-gram theory on a program-by-program basis.
Describing and Producing Program Theory Th e second approach is describing and producing program the-
ory because not all evaluations involve the entire program theory. Each part of the program may operate using diff erent theories. Th is approach focuses on describing program theory for diff erent parts of the program rather than the entire program.
Th e evaluation literature has long recognized the importance of program theory and its potential uses for formulating and prioritiz-ing questions, research designs, and interpreting evaluation fi ndings
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 51
( Chen, 2005; Chen & Rossi, 1980, 1983 ). Chen (1990) writes that pro-gram theory is important because not using this type of theory results in an evaluation that is mechanical or uniform without concern for the theoretical implications of the program’s content, setting, partici-pants, or implementing organizations. In other words, without using program theory, the evaluation is only able to determine if the pro-gram met narrowly defi ned goals rather than satisfi ed theoretical implications. Chen (1990) notes:
Th e question is of whether or not goals are appropriate for the eff ectiveness of the program, or whether these rational goals and procedures could lead to unintended consequences, might not be considered. Evaluators may sometimes be serving only bureaucratic interests and neglect the broader implications for human needs and purposes from the perspective of the stakeholders. To avoid such problems, a new conceptual framework for evaluation should be concerned with value rationality and should provide more insight into the real purposes of a program and its implication for wider social interests. (p. 34)
In criminal justice, evaluations do happen in this manner. For instance, the early evaluations of boot camps as shock incarceration fall into this category. Four- to six-month boot camp–like situations that emphasize military-style discipline and physical exercise served as a correctional alternative to incarceration in the 1990s. Th e early evaluations of this type of program only considered the goals (i.e., reduce recidivism) ( MacKenzie & Shaw, 1993 ) and not the broader theoretical implications (i.e., get tough on crime or self-regulation through self-esteem improvement).
With this in mind, we take the perspective that a program involves a series of interactions between the program staff and target popu-lation. Th ese interactions may involve a multitude of things (e.g., counseling sessions, education sessions, nutrition, medical services, etc.). Th ere are multiple interactions in a program. For instance, the program serves as the organization (i.e., infrastructure, personnel, resources, or activities), and the target population brings their lives (i.e., circumstances) to the organization. Th ese factors may infl uence the success of the program.
Th is perspective brings three components together: impact theory, service utilization plan, and the program’s organizational plan ( Rossi, Lipsey & Freeman, 2004 ). Th e impact theory provides instances about the change process and the change in the conditions. Central to the impact theory is the program’s staff –target population interactions because this is the only way that the change can occur. Th e impact theory may be simple, complex, formal, or informal, but the end result is that change occurred. If change did not occur, then something is faulty about the impact theory.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

52 Chapter 4 THEORY-DRIVEN EVALUATION
Th e impact of a program does not occur by itself. Programs need a stimulus to provide services to the target population. To provide the services as suffi ciently as possible, program staff turn to the service utilization plan. Th e service utilization plan is vital to a theory-driven evaluation because it contains the assumptions and expectations for reaching the target population. In other words, the service utilization plan provides the proper manner to initiate and terminate the service.
Th e program has to be developed and organized in a manner to provide the intended services. Th e organizational plan provides a sche-matic as to how the program resources, personnel, administration, and general organization are composed and utilized. Th e organizational plan is a series of hypotheses that tie together the infrastructure, per-sonnel, resources, etc. to provide the services as intended. Th e combi-nation of resources and organization come together so that the service may be provided and maintained. Th e organization and services are under the control of program staff . Th ese two things—organization and service—represent the program’s process.
Program Impact Theory Program impact theory is about causality. Th is type of theory
describes the cause-and-eff ect sequence of events that come from the program. Specifi cally, there is a stimulus and certain outcomes that come from the stimulus. Generally, evaluators discuss program impact theory using a causal diagram that outlines the cause-and-eff ect events ( Chen, 1990 ). However, it is relevant for an evaluator to keep in mind that programs rarely have direct control over the social conditions that they are expected to improve, and this means that they will have some indirect eff ect for a benefi t to occur.
Th e most basic form of program impact theory takes place in two stages. In the fi rst stage the services are administered that infl uence some intermediate condition, and in the second stage some improve-ment occurs ( Chen, 2011 ). For example, a program to improve alco-hol abuse will use motivation or attitudinal services to infl uence the intermediate conditions that lead to alcohol abuse. While this type of theoretical premise is worthwhile, many programs operate in a more complex manner. Th e complexity comes with many more stages between the program and the benefi ts, and this may or may not mean more than one path to the benefi t.
Th e key to representing any program impact theory is that each part of the program will have a cause-and-eff ect feature. Th at is, each service will cause a linkage between some other services within the program. In the boot camp example, the impact theory is that disci-pline, exercise, and education will result in less recidivism because the attendees have better levels of self-regulation.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 53
Service Utilization Plan Th e service utilization plan provides a clear understanding of how
and why program participants became or will become involved in the program. Th is plan continues to show how the participant will follow through to the point of completing the program. In other words, the service utilization plan provides an outline of program–targets inter-actions from the perspective of the participant, and his or her journey through the program. For instance, a drug treatment program will have a specifi c plan as to how counselors and attendees will interact. Th e plan may consist of days, times, locations, and types of materials.
Program Organizational Plan As the service utilization plan is written from the perspective of
the participant, the program organizational plan is written from the perspective of program management. Th is plan brings together the functions and activities of the program. Further, the program orga-nizational plan provides management with the expectations of the program and the resources (i.e., human, fi nancial, and infrastructure) that are required for the program to function. Key to this plan is the program services that the management or staff are to perform so that the participants are able to reap the intended benefi ts. Th e program organizational plan has to provide information about the sustainabil-ity of the program, including fundraising, personnel management, facilities acquisition and maintenance, and political climate.
A program’s organizational plan may be shown in several diff er-ent ways. Focusing on the interactions between the program and its participants allows the fi rst element—a description of the program’s objectives for the services provided—of the program organizational plan to permeate to the top. If this element does not permeate to the top, a number of questions may assist in this process: ● What are the services? ● How much is to be provided? ● To whom are the services to be provided? ● On what schedule are the services to be provided?
Th e second element of the program organizational plan is to describe the resources and functions that are necessary for the program. Th is includes a description of the personnel with proper credentials and skills, logistics, proper facilities, funding, supervision, or clerical support.
Having been exposed to the issues noted here does not guarantee that it is clear on how to extract program theory. Th e production of program theory is important to develop a high-quality program eval-uation. Program theory may be articulated or spelled out clearly so that everyone understands the theory behind the program. Th is tends
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

54 Chapter 4 THEORY-DRIVEN EVALUATION
to occur when a program comes from a criminological theory. For instance, social learning theory may be a theoretical premise used to reduce instances of gang involvement.
Unfortunately, all program theory is not well articulated. Th is is known as implicit program theory ( Chen, 1990 ). Th is type of theory means the underlying assumptions about program services and prac-tices are not well discussed and articulated, in the context of theory. Th is type of program theory provides evaluators the most trouble.
Implicit program theory requires the evaluator to extract and describe the theory before it can be analyzed or evaluated. Th e eval-uator has to determine the intentions of the program framers and stakeholders about what the program should be doing. Th e extraction of this type of theory works best using a set of steps.
Step 1: Define Boundaries Th e fi rst step of extracting an implicit program theory is defi ning
the boundaries of the program, which are contingent on the scope of the program and the scope of the program staff ’s concerns. An evaluator may identify the boundaries of the program if he or she works from the perspective of the decision makers. Th e decision mak-ers should have some idea of the activities of the program and orga-nizational structures . In other words, the decision makers, especially those who are charged with acting on the results of the evaluation, should have some idea where the other decision makers stand on the reasons why the program exists and how the program should operate.
Th e defi nition of program boundaries has to include all the important activities, events, or resources that have a link with one or more outcomes that are central to the program. Th e evaluator may be able to uncover these boundaries by beginning with the benefi t from the program and working backward to identify the relevant activi-ties and resources that may make a contribution to organizational or programmatic objectives. From this perspective, a drug treatment program at the local or state level could be distilled as a set of activi-ties that are organized by a rehabilitation group to alleviate drug abuse in a specifi ed participant group.
While these two approaches are good starting points, their ratio-nal presentation may oversimplify the process of extracting program theory. Evaluators have to keep in mind that programs may be com-plex. Further, each of the objectives of a program may be diffi cult to establish. Th is puts the evaluator into a position where he or she has to negotiate the defi nition of program boundaries with program staff , stakeholders, and decision makers. Further, evaluators have to recog-nize that the defi nition of boundaries is a fl uid process that requires them to be fl exible.
For example, Esbensen & Osgood (1999) describe the boundaries of the GREAT program. Th e program is designed to reduce instances
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 55
of juveniles joining gangs. To address this specifi c need, the GREAT program is bound to middle-school students, and the curriculum is delivered by law enforcement. Here, the boundaries are the age group (i.e., 11–13 year olds) of the students.
Step 2: Explicate Program Theory Not all program theory is original. Some program theory may
come from prior experience of program planners, research, or prac-tice. Th is is a welcomed situation for an evaluator because he or she is able to develop a well-articulated theory. Th e evaluator encoun-ters issues when dealing with an existing program. He or she has to work through the structure and operations of the program to derive the theory. Th is means that the evaluator has to work with stakehold-ers to bring out the theory that comes from their actions and assump-tions. Th e evaluator accomplishes this by producing multiple drafts describing the program theory. He or she presents these drafts to stakeholders and decision makers to ensure that their approxima-tions of the program theory are correct.
Esbensen & Osgood (1999) describe the development of the GREAT program in this manner. In 1991, when the Phoenix, AZ, Police Department introduced GREAT as a school-based program, they used D.A.R.E. (Drug Abuse Resistance Education) as a model to create a nine-week curriculum. D.A.R.E. is a program specifi -cally designed to provide resistance training and education for drug use. Th is program generally consists of a 17-lesson curriculum that is off ered once a week in schools. Both curriculums for GREAT and D.A.R.E. are off ered by trained law enforcement offi cers.
To begin the process of drafting a program theory like this one, the evaluator needs to turn to several sources for information. Th e evalu-ator has to analyze the program documents. Th is can be all reports, mission statements, or goal statements. Th en, the evaluator should interview the stakeholders and decision makers. Th is will provide information from both of these respective sides of the program. Next, the evaluator should perform site visits to observe the functions and circumstances of which the program operates in its natural environ-ment. Finally, the evaluator should consult the criminological litera-ture. Information from these places will assist the evaluator in drafting a reasonable program theory, but the fi nal verdict comes from the review of the program theory by the stakeholders and decision makers.
Step 3: Define Program Goals and Objectives Program goals and objectives are central to program theory. Th ey
provide an understanding of what the program should be accom-plishing. Th e issue that often arises with program goals and objec-tives is that they do not also mesh with mission statements, or they
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

56 Chapter 4 THEORY-DRIVEN EVALUATION
may not mesh with the responses or needs of stakeholders. Th is means that the evaluator has to be able to explicate the program goals and objectives that are realistic to the outcomes of the pro-gram. An important issue that the evaluator has to keep in mind is the consistency between the actual accomplishments of the program and the intended accomplishments of the program. Th is means that the evaluator should review the major program activities as well as the written goals of the program.
Th is defi nition of goals becomes part of the program theory; however, the goals and objectives have to be properly placed in the program theory. For instance, goals and objectives that bring about change are impact theory, but goals and objectives concerning pro-gram activities are service delivery. If the program aims to reduce juvenile delinquency, this is part of the impact theory, but if the aim is to off er after-school care for juveniles to reduce unstructured socialization, then a portion of the service delivery plan is present.
An example of this process is the GREAT program. Esbensen & Osgood (1999) argue that the objective of the program is to reduce gang activity and educate juveniles about the consequences of gang involvement. In a broader sense, GREAT provides life skills that empower juveniles with the ability to resist joining gangs through cognitive behavioral strategies.
Step 4: Describe the Program Functions, Components, and Activities
An evaluator has to be able to properly describe all of the program functions, components, and activities. Program functions include every-thing that a program does (e.g., intake, recruitment, etc.). Th ese types of activities are central to the understanding of the program theory. Without them, the description of the program theory will be incomplete.
Th e evaluator must also be able to link the program functions, components, and activities into a logical sequence of events that occurs within the program. Th is is consistent with the development of a logic model. In the GREAT program, the law enforcement offi cers who come to the school to deliver the curriculum engage the juve-niles in instruction, discussion, and role-playing. Th ese activities pro-vide the opportunity to introduce confl ict resolution skills, cultural sensitivity skills, and the negative aspects of gang life ( Esbensen & Osgood, 1999 ).
Step 5: Final Corroboration of the Description of the Program Theory
Th e evaluator has to keep in mind that the result of these steps will be a program theory that is consistent with what the program
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 57
was intended to do rather than the actual state of the program. Th is occurs because those involved think in ideal terms rather than real terms. In other words, those involved with the program tend to focus on alleviating a social problem, and do not see the faults of the program. Th e program staff who are further away from the program will see it in a manner as it should be, and be further away from the shortcomings of the program.
Th e diff erences between program theory and reality are not uncommon. Th is is another area that the evaluator needs to devote some attention. On one hand, without an understanding of the mag-nitude and nature of these diff erences, the evaluator may make some erroneous assumptions about the program theory. On the other hand, if the theory is so perfect that it does not depict reality, it needs to be revised. Suppose a drug treatment program calls for daily contacts between a drug counselor and participant. If the program resources do not allow for this, the program theory needs to be revised to account for meeting schedules that are realistic.
Because program theory is designed to capture reality, the cor-roboration of the program theory has to be confi rmed by stakehold-ers and decision makers. Without their input the program theory may be irrelevant. Worse yet, the program theory may lead to an evalua-tion that cannot be useable. Another situation may arise. No corrobo-ration of the defi nition of the program may lead to a poor defi nition of the program, or it could reveal competing philosophies between stakeholders and decision makers.
Th e evaluator needs a clear and concise description of the pro-gram theory. Th is is the guide to understanding the intentions of the program for proper analysis and evaluation. Th e corroboration of the program theory only serves as confi rmation between stake-holders and decision makers that the program operates as intended. Th is does not place a good- or bad-quality statement on the program theory. For instance, Esbensen & Osgood (1999) argue that no part of GREAT explicitly discusses any criminological theory, but that diff er-ent pieces of the GREAT curriculum capture parts of Gottfredson & Hirschi’s (1990) self-control theory and Akers’ (2009) version of social learning theory. Th rough corroboration and interviews with other academics and program staff this became clear. We will describe the steps for evaluating program theory next.
Analyzing Program Theory Analyzing some type of program theory is common, and it usually
takes place in a larger context of an evaluation of a program’s pro-cess or impact. In the criminological and criminal justice literature,
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

58 Chapter 4 THEORY-DRIVEN EVALUATION
little has been written about how to perform an analysis. Th is is because it typically is performed in an informal manner that usually comes from commonsense judgments. When the program theory is articulated well, the validity of the program theory is straightfor-ward and accepted on the basis of limited evidence or commonsense judgment.
Generally, programs are not based on simple expectations or goals. For instance, a parenting program that assigns case manag-ers to coordinate courses for parents of children with low levels of self-control involves several assumptions about what it is supposed to accomplish and how ( Piquero et al., 2010 ). In this situation, the program theory may be faulty, requiring a stringent evaluation or analysis.
Effi ciency suggests that an analysis or evaluation of each individ-ual assumption, goal, or objective of program theory may not be pos-sible. Th is does not imply that these assumptions, goals, or objectives cannot or should not be subjected to evaluation or analysis. Certain tests exist that can be conducted to provide assurance that they are sound. Here, we summarize the types of tests that may be used to provide evaluation or analysis information.
Link between Program Theory and Social Needs Th e program theory analysis and evaluation should begin with
the needs evaluation as described in Chapter  3 . Th is means that there needs to be a clear linkage between the program theory and the social need of the target participants. A program theory that does not have a link with social needs is an ineff ective program no matter how well it is implemented; thus, one of the most fundamental issues is to analyze or evaluate the link between the program theory and social needs.
No set form of evaluation exists to determine if the program the-ory properly links to or generates a suitable conceptualization of how the social needs may be met. Th e proper evaluation process comes from the evaluator’s judgment. To improve the validity of this type of evaluation multiple judgments from collaborators are necessary. Th e collaborators may be criminologists, policy makers, or advocacy groups associated with the target population.
Th e diversity of the group will make a contribution, but their chief contribution is specifi cation. When program theory and social needs are described in general terms, the evaluator may have a false sense of congruence. For instance, when juvenile delinquency appears to rise during the summer months, some areas may institute a curfew barring juveniles from being out past a certain time. Th e social issue of juvenile delinquency appears to be solved by the program of a
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 59
curfew. In reality, when the program theory or social need is vaguely written, the evaluator may have a false sense of service.
Greater detail is necessary to diagnose a social need and provide ample program theory for service. Th e diversity of the group may be able to bring experiences with other programs, knowledge from the criminological and criminal justice literatures, or knowledge of the political arenas for a better understanding. Th is will allow for a better program evaluation because the understanding of the social need is much better.
Moving forward to the program impact theory, it is instructive to remember that program impact theory is a sequence of causal links between program services and improved outcomes or benefi ts. Th e main issue is whether the program theory has had an impact or change on the social need that it is intended to alleviate from the needs evaluation. Consider for instance, a school-based educational program aimed at getting middle-school children to understand that gang involvement is a poor choice. Th e problem this program attempts to alleviate is gang membership. Th e program impact theory would show how linked educational modules raise the awareness of why joining a gang is a poor choice.
Evaluating Logic and Plausibility of Program Theory Extracting and espousing the program theory should reveal the
major assumptions and expectations of the program’s design. One form of evaluation or analysis is a simple review of the logic or plausi-bility of the parts of the program. As with other forms of evaluation or analysis, a panel of reviewers should be brought together to help per-form this type of evaluation or analysis of the program theory ( Chen, 1990, 2011; Wholey, 1979, 1987 ). Th e panel should include members of the program staff , stakeholders, decision makers, and the evalu-ator. Because of the intimate relationship between these individu-als and the program, it is advisable to involve informed individuals who do not have any connection to the program. Th is may include criminologists, other program administrators, or advocacy group members.
Th is type of evaluation is not a structured process—it should be open-ended. Th is does not mean that it should be without rigor. Th e rigor comes in some of the general issues that the review should address, including: 1. Are the program objectives well defi ned? Th e outcomes should
be stated clearly and in a manner where a determination can be made as to whether the objectives have been met. A well-written objective is that a school off ering an after-school program will reduce delinquency around the school during after-school
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

60 Chapter 4 THEORY-DRIVEN EVALUATION
hours by 10%. Th e panel has a clear objective and a measurable outcome.
2. Are the program goals or objectives feasible? Is it possible that the program goals or objectives can be met? Completely eliminating crime is grandiose, but decreasing crime is much more attainable. Th e panel has the ability to determine if the goals or objectives are feasible.
3. Is the change process plausible? Th e program operates using a cause-and-eff ect format. Th at is, the program will cause a desired change for a proscribed group of participants. Th is implies that the program operates in a logical format resulting in a plausible change. Th e validity of the program theory is the ability of the causal logic of the program to produce intended eff ects. Th e most desirable situation would be that the causal logic within the pro-gram is supported by evidence that the logical links actually occur. Th e panel should be able to determine if the change process is plausible.
4. Are the procedures for identifying the target population, reaching them to deliver service, and sustaining the service through comple-tion well defi ned or suffi cient? Program theory should specify the procedures and functions that are suffi cient for the purpose. Th is specifi cation should come from two perspectives: the program’s ability to perform and the target population’s likelihood of being engaged. Th e panel should be able to identify these two things.
5. Are the resources allocated to the program and its various activi-ties adequate? Th e resources for a program are vast. Th ese resources include personnel, material equipment, and other assets (e.g., buildings, reputation, relationships, and facilities). Th e panel should be able to comprehensively identify the link between the program theory and resources.
Comparing Research with Practice A method of assessing the program theory is to fi nd out whether
it is consistent with the research evidence or experience from else-where. Th e evaluator has numerous ways to be able to perform this type of comparison. Th e most straightforward manner is to evaluate a program using similar concepts. Th e results will provide information as to whether the program will be relatively successful. An evaluator should use evaluations of similar programs.
Other forms of research may be instructive as well. For instance, basic research in the criminological or criminal justice literatures may provide information about the program theory. Th e evaluator has to be very careful that a balance is struck between the basic research that does not have evaluation in mind and evaluation research.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

Chapter 4 THEORY-DRIVEN EVALUATION 61
Both will provide more information that will gain some insight into the program theory.
Summary Th is chapter covers the basic information that is necessary for a
criminal justice evaluator to complete a theory-driven evaluation. Th e assessment of evaluability is vital to the development of a theory-driven evaluation. To properly determine the evaluability of a pro-gram, the evaluator has to describe the program model, assess the opportunities for evaluating the model, and identify the stakeholders’ interest in the model. Th is chapter also provides information about the production of program theory. Th e steps that are involved in the production of program theory result in a concise description of the program theory.
Discussion Questions 1. Identify a program from the criminal justice fi eld. Work through
the production of program theory. Discuss the issues that arise from completing the production of a program theory.
2. Discuss the steps in performing an evaluability assessment approach.
3. Discuss the diff erences among a program impact theory, service utilization plan, and program organizational plan.
References Akers , R. ( 2009 ). Social learning and social structure: A general theory of crime and
deviance . Boston : Northeastern University Press . Chen , H. -T. ( 1990 ). Th eory-driven evaluations . Newbury Park, CA : Sage . Chen , H. -T. ( 2005 ). Practical program evaluation . Th ousand Oaks, CA : Sage . Chen , H. -T. ( 2011 ). Practical program evaluation: Assessing and improving planning,
implementation, and eff ectiveness . Th ousand Oaks, CA : Sage . Chen , H. -T. , & Rossi , P. H. ( 1980 ). Th e multi-goal, theory-driven approach to
evaluation: A model linking basic and applied social science . Social Forces , 59 , 106 – 122 .
Chen , H. -T. , & Rossi , P. H. ( 1983 ). Evaluating with sense: Th e theory-driven approach . Evaluation Review , 7 , 283 – 302 .
Coryn , C. L. S. , Noakes , L. A. , Westine , C. D. , & Schroter , D. C. ( 2011 ). A systematic review of theory-driven evaluation from practice from 1990 to 2009 . American Journal of Evaluation , 32 , 199 – 226 .
Esbensen , F. A. , & Osgood , D. W. ( 1999 ). Gang Resistance Education and Training (GREAT): Results from the national evaluation . Journal of Research in Crime and Delinquency , 36 , 194 – 225 .
Gottfredson , M. , & Hirschi , T. ( 1990 ). A general theory of crime . Palo Alto, CA : Stanford University Press .
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

62 Chapter 4 THEORY-DRIVEN EVALUATION
MacKenzie , D. L. , & Shaw , J. W. ( 1993 ). Th e impact of shock incarceration on technical violations and new criminal activities . Justice Quarterly , 10 , 463 – 487 .
Matthews , B. , Hubbard , D. J. , & Latessa , E. ( 2001 ). Making the next step: Using evaluability assessment to improve correctional programming . Th e Prison Journal , 81 , 454 – 472 .
Piquero , A. R. , Jennings , W. G. , & Farrington , D. P. ( 2010 ). On the malleability of self-control: Th eoretical and policy implications regarding a general theory of crime . Justice Quarterly , 27 , 803 – 883 .
Rossi , P. H. , Lipsey , M. W. , & Freeman , H. E. ( 2004 ). Evaluation: A systematic approach ( 7th ed. ) . Th ousand Oaks, CA : Sage .
Smith , M. F. ( 1989 ). Evaluability assessment: A practical approach . Norwell, MA : Kluwer Academic Publishers .
Van Voorhis , P. , & Brown , K. ( 1996 ). Evaluability assessment: A tool for program development in corrections . Washington, DC : National Institute of Corrections . [Unpublished monography] .
Van Voorhis , P. , Cullen , F. T. , & Applegate , D. ( 1995 ). Evaluating interventions with violent off enders . Federal Probation , 50 , 17 – 27 .
Welsh , W. ( 2006 ). Th e need for a comprehensive approach to program planning, development, and evaluation . Criminology and Public Policy , 5 , 603 – 614 .
Welsh , W. , & Harris , K. ( 2004 ). Criminal justice policy and planning ( 2nd ed. ) . Cincinnati: LexisNexis : Anderson Publishing Co .
Wholey , J. S. ( 1979 ). Evaluation: Promise and performance . Washington, DC : Urban Institute .
Wholey , J. S. ( 1987 ). Evaluability assessment: Developing program theory . In L. Bickman (Ed.), Using program theory in evaluation. New Directions for Program Evaluation, No. 33 . San Francisco : Jossey-Bass .
Additional Readings Mercier , C. , Piat , M. , Peladeau , N. , & Dagenais , C. ( 2000 ). An application of theory-
driven evaluation to a drop-in youth center . Evaluation Review , 24 , 73 – 91 . Wilson , D. M. , Gottfredson , D. C. , & Stickle , W. P. ( 2009 ). Gender diff erences in eff ects
of teen courts on delinquency: A theory-guided evaluation . Journal of Criminal Justice , 37 , 21 – 27 .
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group.Created from liberty on 2023-01-26 08:48:26.
Cop
yrig
ht ©
201
4. T
aylo
r &
Fra
ncis
Gro
up. A
ll rig
hts
rese
rved
.

QUALITY: 100% ORIGINAL – NO PLAGIARISM

(USA, AUS, UK & CA PhD. Writers)

CLICK HERE TO GET A PROFESSIONAL WRITER TO WORK ON THIS PAPER AND OTHER SIMILAR PAPERS

The Best Custom Essay Writing Service

About Our Service

We are an online academic writing company that connects talented freelance writers with students in need of their services. Unlike other writing companies, our team is made up of native English speakers from countries such as the USA, UK, Canada, Australia, Ireland, and New Zealand.

Qualified Writers

Our Guarantees:

CLICK TO SUBMIT YOUR ORDER