DD/A MANAGEMENT CONFERENCE, (Sanitized) MAY 1975

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP81-00261R000200060001-7
• Release Decision: RIPPUB
• Original Classification: U
• Document Page Count: 99
• Document Creation Date: December 14, 2016
• Document Release Date: October 7, 2002
• Sequence Number: 1
• Publication Date: June 27, 1975
• Content Type: MF

Approved For Release-2003/01/27 : CIA-RDP81-0026 1R000 09060001-7
STAT
6.. %me
27 June 1975
MEMORANDUM FOR: DD/A Personnel
SUBJECT: DD/A Management Conferen
attending were:
E May 19 7 5 STAT
1. The,Deputy Diredtor for Administration sponsored
a management conference on May 16, 17, and 18, 1975. Those
Harry Fitzwater, D/OJCS
Michael Malanick, D/L
John F. Tietjen, D/MS
D/CO
omas Me, /Fin
STATINTL
2. Mr. Blake opened the conference by noting that
certain events had taken place since he and Mr. McMahon
assumed their current posts in August 1974. He singled
out the Seymore Hersh article in the 22 December 1974 New
York Times and its pervasive impact on the Agency. He
looked back at the 27 September 1974 conference where the
major emphasis was placed on "planning" with high hopes
for establishing a Planning Council which would lay out
long-range plans for the Directorate. As a result of the
Hersh article and the establishment of the President's
Commission and Senate and House Committees to scrutinize
Agency activities, the implementation of a Planning Council
will be postponed. There is no way to predict the outcome
of the Commission and Committees' studies, but it is a
possibility that some changes in mission, function, and
organization will be forthcoming. Given the uncertainty,
to plot the future course of the Directorate would be
most difficult at this time. On the other hand, the Offices
must continue to plan their activities within the context
of their current functions.
Mr. Blake further emphasized the need for open communi-
cations and the responsibility of the Office Directors for
keeping their people informed of Agency issues and problems
as well as positive actions that are going on. While it is
true, generally, that a great deal of time is taken up with
Approved For Releas
John F. Blake, DDA F. W. M. Janney, D/Pers
John N. McMahon, A/DDA Charles W. Kane, D/S
00;/Q1e7 ?,CIA-R > 1-00i6lk2000Z0g060001-7
a..As, All-
 Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7to 9 A?,' - AT r' n~, ~''~ .ab
responding to Commission and Committee inquiries (and FOI
requests) it is of vital importance to press on with Agency
business. To prove that Agency management is not mesmerized
with outside investigations, it is essential that this
conference address specific Directorate issues and problems
from which specific actions will devolve. Those areas
selected for discussion are:
a. Management by Objectives
b. The effectiveness of Directorate Staff
support
c. Implementation of the PASG Handbook
d. The necessity of an "M" Career Service
e. The future of the "MG" Career Service
The above are considered relevant to the continued function-
ing of the Directorate and discussions at the conference
should result in specific recommendations for actions to be
taken.
Mr. Blake concluded by enjoining all of the conferees
to do their jobs to the best of their ability and to have
confidence in the Agency leadership personified in Mr. Colby.
3. Before getting to the discussion items, Messrs.
McMahon and Blake briefed the conferees on the activities
of the President's Commission and the Senate Committee.
Regarding the President's Commission, Mr. McMahon informed
that their work is pretty well completed and that within
the near future they would make their report to the President.
In his opinion the report will be generally favorable.
Concerning the Senate Committee, Mr. McMahon gave some
of the flavor of their activities to this point, including
the deliberations of the use of Agency "monitors" to assist
. W4
witnesses. This is a contentious item, at this time unresolved, =,
although it appears likely that Agency witnesses at certain
times will have some Agency representation. Based on their
actions so far, substantiated by certain memorandums which
have been prepared, it seems quite likely there is a difficult
road ahead. Mr. Blake discussed the more recent hearings with
Mr. Colby. He concluded that a problem could be the struggle
between the Legislative and Executive, with the Agency in the
middle.
Approved For Release 2003/01/27 CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
iat:s.la~i+a+v.oilU Lid vi
Discussion Items
4. Management by Objectives - The Director espouses
management by objectives as the management philosophy in
the Agency. While the DDA has been implementing the system
for the last two years, there is room for improvement.
Perhaps one of the major shortcomings at this point relates
to the need for evaluating the results of our action plans
and the problem of managers being able to quantify their
efforts. Emphasis was placed on the need for norms against
which progress could be measured.
Considerable discussion ensued relating to standards
of performance and the extent to which they have been
established within Offices. During the course of the
dialogue, the word "experience" was frequently used as
the basis for establishing certain standards and norms.
Experience, however, does not provide objective empirical
data to realistically evaluate performance. Emphasis was
placed on the need to synthesize the work of the Directorate
within the MBO framework, which should give a fair indicator
of how well we are doing. A specific objective for FY 76
was laid out which generally received the acceptance of the
Office Directors. It read as follows: Devise systems and
procedures to develop and establish a series of positive
indicators against which progress mission (functions) can
be judged.
Each Office may attack this objective in a different
way. It is not intended that every function lend itself
to quantifiable norms. It is necessary, on the other hand,
that each Office make a start in this effort during FY 76
against which FY 77 objectives may be measured. (It was
STATINTL suggested that meet with the Office Planning
Officers within the next week or two to discuss the imple-
mentation of this objective.)
S. Staff Support - Mr. Fitzwater led the discussion.
He emphasize increased requirements levied upon OJCS and
extended this to the other Offices to a greater or lesser
extent. He followed with a viewgraph depicting the DD/A
Approved For Release 2003/01/27 CIA-RDP81-00261 R000200060001-7
t Cr e. w~ 3 p_' 7
tm-..5. a a u3 E__ 3 l
 Approved For Release 2003/01/27
CIA-RDP81-g0261g0p0200060001-7
v
~d..Z ~:~ i ~L]3
staff organization, pointing to the fact that each element
could levy requirements on the Offices leading possibly to
excessive demands. Although it was generally agreed that
there are increased amounts of work and in some instances
red tape, the Office Directors do not attribute this to
requirements levied by members of the DDA Staff. By and
large, there was agreement that DDA Staff support was quite
responsive. The problem of setting conflicting deadlines
by members of the Staff was discussed and the suggestion
was made that a single point be established to set dead-
lines. The Staff will be alert to this problem and at
least for the time being it does-not seem necessary to
establish a central point. A suggestion was made and
agreed to that when requirements are levied on Office
Staffs, the members of the DDA Staff will alert the Office
Directors at the morning meeting whenever possible.
A question was raised as to the necessity for Offices
submitting weekly reports. Both Messrs. Blake and McMahon
responded that the reports are absolutely essential toward
keeping the DDA informed as to what is going on in the
Offices in more detail than received at morning meetings
.or other staff meetings.
6. PASG Handbook discussed a checklist
which had been competed by each Office identifying the
progress which had been.made toward implementing the PASG
requirements. These will be brought u -to-date and distribu-
tion will be made to all the Offices. walked
through the various sections of the Handbook. Considerable
discussion evolved, especially in regards to grievances and
how they should be handled, and whether or not a grievance
panel is advisable or necessary. In response it was agreed
that a panel is not essential but that grievances must be
handled expeditiously within the chain of command.
Fitness Reports were discussed at length in terms of
the extent to which they serve as a primary resource tool
for evaluating individual performance. It was generally
agreed that the ratings as submitted do not help to identify
the bottom three to five percent. briefed on the
procedures used in the Office of Communications emphasizing
their use of BYCELS. This system documents some twenty
different characteristics which are subdivided into four
ratings. Each characteristic receives a numerical grade
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
PP Tr
STATINTL
STATINTL
STAB NTL
 Approved For Release 20031jB8110
UP
which is weighted depending on the grade of the individual.
While fitness reports are reviewed by the individual employee,
he does not see his BYCELS rating sheet although he is
briefed on the results. Again relating to fitness reports,
Mr. Fitzwater mentioned a continuing problem of ratings within
a specific organization wherein an individual must respond
to many customers as well as his supervisor. There is no
question this problem is real, but it is one for which
ON
STATINTL
there is no ready answer and one must evaluate an employee
as fairly and equitably as possible.
A point was made within the context of competitive
evaluations: the PASG Handbook and its requirements must
be looked on as a tool of responsible management; it is not
a document only to support efforts to identify low perform-
ance individuals or to implement termination proceedings.
A good deal of discussion arose over those portions
of the Handbook relating to clerical careers. It became
quite apparent that it is necessary to define the types
of clerical positions that exist in the Directorate. In
addressing the enhancing of careers of clerical personnel,
it was felt advisable to publish a notice for distribution
throughout the Directorate. It was emphasized that such
a notice should be carefully worded and the terms be
thoroughly defined. The question of precisely to whom
the memo should be addressed was raised. While it was
recognized that a notice is highly desirable, the suggestion
was made and accepted that such notice be withheld until
completes her study on clerical attitudes
and viewpoints.
7. "M" Career Service - Mr. Blake opened the
discussion pointing out t at he was of an open mind as
to whether or not an "M" career service was a worthwhile
entity and asked for free discussion as to what the con-
ferees think of the value of the "M" career service. In
general, most of the Office Directors agreed that there
were benefits to the "M" career service, partly for
psychological reasons indicating that supergrades within
the Directorate belong to the Directorate management pool,
rather than a single office.
Approved For Release 2003/0 $1 00~ .d f? 2QQRho
 Approved efftR 1ease200W01129-Bb 81-p0)61 R000200060001-7
As an action item, the Office Directors were asked to
identify positions at the GS-15 and 16 level that can be
filled by an officer from another Office. This will assist
the Career Management Office in recommending appropriate
assignments and reassignments.
8. "MG" Career Service - Mr. Blake opened this discus-
sion by indicating that c anges within the Agency are causing
new and different requirements for support officers. Whether
or not a Generalist career service continues to be viable is
open to question. There are many capable officers within the
"MG" career service; however, the types of jobs the "MG"
careerists have filled in the past are changing in nature.
There was a strong voice that the "MG" service not be abol-
ished as long as overseas requirements exist and there is
a group of capable "MG" careerists available to fill them.
On the other hand, the "MG" careerists enjoy a relatively
high grade at this time and overseas positions are being
cut or reduced in grade. It was agreed that new members
should not be moved into this service and that the relatively
junior jobs be filled by officers of the DDA Career Sub-
Groups. The suggestion was made that "MG" assignments be
handled at the Directorate level having input from each
Office on the identification of young officers who can
handle these more broadly gauged assignments.
In summary, it was felt that it would be well to con-
tinue the "MG" career service for current "MG" careerists,
but not fill in behind them with younger, more junior
officers who would appropriately remain as members of an
Office Career Sub-Group.
Specific actions suggested were a review of "MG"
positions, toward identifying those which should be more
appropriately Career Sub-Group positions. Also, there
should be an identification of those officers within
Career Sub-Groups who can serve in Generalist positions.
9. Other Business
a. Center for Studies in Iritelli 'ence Mr.
Rodriguez briefed t e group on the Center for Studies in
Intelligence, indicating that four types of activities are
involved: (1) the initiation of research; (2) setting up
Approved For Release 4)03/01/27 : CIA-RDP81-00261 R000200060001-7
n -1
 Approved For Release EEO /O1'27'5.:.dWRT) P 1 Q26`1 0 , 2 O b001-7
and conducting symposiums; (3) sponsoring research; and
(4) establishing discussion group meetings. He described
some of the topics that have been submitted for Center
involvement. Mr. Rodriguez voiced his interest in inviting
controversial non-Agency participation in the activities
of the Center. While this met with general acceptance, it
was pointed out that one must be extremely careful as to
whom is invited in light of today's environment. There
was some discussion as to how proposals for topics of
study at the Center should be evaluated and whether or not
the DDA should be making proposals. It was made clear
that a consultative group be established to review topics,
although membership was left open for further thought.
Insofar as DDA involvement is concerned, it was agreed that
the DDA should not submit proposals for study at this time
but should stand ready to provide assets to DDO officers
pursuing studies relating to staffing, training, and the
like which would impact the DDA.
b. Personnel'Proj"ections - Mr. Janney presented
several charts depicting personnel trends. There is a
hump developing in the 30 to 40 year old age bracket with
limited input (new hires) in the 20 to 30 age bracket.
This points to the necessity for addressing the bottom
three to five percent of personnel and action to move
this group out of the Agency. Mr. Janney reported that
there. was no attrition during April and that the hiring
rate was low. Quite unusual is the fact that clerical
attrition is low. In his general discussion as to the
availability of a managed surplus exercise, Mr. Blake
emphasized that there should be consistent application
throughout the Directorate and that no Office should move
unilaterally. Mr, McMahon cautioned that one must be
careful in addressing a managed surplus since the DCI
has emphasized on several occasions that he does not
foresee such a program. At any rate, this Agency issue
should perhaps be best discussed at a DCI conference or
Management Committee meeting sometime in the near future.
An action item resulting from this discussion r
l
ti
a
e
ng
STATINTL to the statistics provided by Mr. Janney was Mr. Blake's
request that provide similar statistics for
the other three Directorates and total Agency.
2Q0~3/ 1/ : GA- 9P81-90;01 RO0 00060001-7
Approved For ReleX
 Approved se LEO 3/01 2-7--61 1-F;r f#$1f11 1 R000200060001-7
c. Head uarters Buil'din - Discussion involved
the Headquarters building and the encroachment of equipment
into people space. While recognizing this as a real problem,
the fact must be faced that it is highly unlikely that
Congress would consider funding a new CIA building either
for office space or to house equipment. The Office of
Logistics, nonetheless, will continue its Building Planning
Staff to make plans for the future.
10. DDA Conferences - The question was raised as to
how often DDA con erences should be held. Most Office
Directors prefer annual meetings rather than semiannual.
It is likely that the next conference will be held approxi-
mately February 1976, at which time the Senate and House
Committees' recommendations may be made or at least indi-
cated. Mr. Janney suggested that perhaps a conference
every nine months would be about right. At any rate, a
DDA conference every six months is considered excessive.
11. .Summar - Consistent with Mr. Blake's opening
remarks, the following actions were recommended:
a. MBO: Each Office to devise systems and
procedures to develop and establish a series of
positive indicators against which program mission
(functions) can be judged. This is an FY 76
objective against which certain FY 77 objectives
can be measured/evaluated.
b. Office Support: Look into a procedure
by which ea ines are established in a consistent
manner. Also keep the Office Directors informed
when the DDA Staff levies requirements on Office
Staffs.
c. PASG Handbook: At an appropriate time
publish a clerical notice relative to career
aspirations. Provide to each Office the check-
list of all Offices relative to implementation
of PASG requirements.
d. '"M" Career 'Service : Each Office provide
list of GS-15 and positions which could be filled
by officers of other Career Sub-Groups.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
8
r, ^ "1 1
 Approved For Rel9Ai fl loal C1A14 fi 481R U1 0060001-7
e. "MG" Career Service: Identify "MG"
positions which should more appropriately carry
the designation of Career Sub-Groups. Identify
personnel within each Office who can serve in
more broadly gauged Generalist positions.
f. Center for Studies in Intelli ence: For
OTR - Review and make recommendations as to member-
ship of consultative body to review topics for
Center study.
g. Personnel Projections: For OP - Provide
statistics similar to those provided the DDA for
the other three Directorates and the Agency as a
whole.
12. Mr. Blake closed the conference expressing apprecia-
tion for the open and frank discussions among the conferees
and the collegiality evidenced during the conference.
DISTRIBUTION: DD/A Personnel (1-15)
Approved For Release 2003/01/2 : CIA-RDP81-00261R000200060001-7
Atyai8 1- go ~4't ^i m 64 L~~Y Cj` Caa i.v 'Jib
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
AGENDA
DD/A Management Conference
May 1975
STAT
INTL
May 16
4:00
Depart
500
Arrive STATINTL
STATINTL
500
- 6:00 Cocktail Hour
600
Dinner at Club
800
- 9:00 Discussion of Purpose for
May 17
7:00
Conference - Mr. Blake
- 9:00 Breakfast
9:00
- 9:30 Current Status of Commission and
Legislative Committee Reviews -
9:30
Mr. McMahon
- 11:30 Critique of MBO as a Management
Technique -
& STATINTL
11:30
- 12:00 Discussion of Impact of DD/A Staff
Requirements Upon Offices -
(Off
ice Director
to be announced)
12:00
1:30 I
STATINTL
1:30
- 2:30 Discussion of PASG/Directorate
Personnel Handbook related
matters -
STATINTL
2:30
- 3:30 Discussion of the Need for an
"M" Career Service - Mr. Blake
3:30
- 4:30 Presentation of End-of-Year
Personnel Strength Projections,
Directorate E Agency - Mr. Janney
4:30
- 6:00 Free Time
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
May 17
(Cont'd) 6:00 - 7:00 Dinner
STi !INTL
7:30 - 8:30 Directorate Research Participation
in the Center for Studies in
Intelligence - Mr. Rodriguez
& attendees
May 18 7:00 9:00 Breakfast
10:00 11:00
11:00 - 11:30
Mystery Subject - Mr. McMahon
Wind-up - Mr. Blake
12:00 - 1:00 Lunch
1:30 Depart
2:30 Arrive
STATINTL
STAN NTL
STATINTL
STAT
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 STAT Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
MBO is about to begin its third year as a concept of management
in the DDA. In the brief period of time we have used it, attitudes
toward the system have moved from indifference or hostility towards
acceptance and appreciation. There are still problems associated with
the use of MBO as a total management concept. Some problems can be
attributed to the educational and shakedown processes we have undergone
in implementing MBO; some problems can be related to managerial attitudes
towards its use; and others can be related to the environment in which
the system must exist. These problems do not exist in neat compartments,
but are inter-related and often difficult to isolate.
MBO came into CIA almost by fiat when the DCI adopted it as a
management style. In one of his speeches he summarized his approach
to the process saying that managers should be told what their goals
are, what resources they have at hand, and then left alone to work toward
the accomplishment of the goals. Periodic reviews, of course, are needed,
as is a final evaluation, but for the most part managers should be left
to do their own thing. And herein lies part of the Agency's problem.
Mid-level managers and planners indicate that direction from the top
in the goal-setting process could be more forthcoming.
The education of Agency managers in the use of MBO was haphazard
at first. Gradually, training programs were introduced. To improve
managerial understanding, this spring OTR initiated an MBO seminar.
Reaction to the first two runnings of the seminar was favorable,
particularly in pointing up two things: (a) the strengths of the
system; and (b) the shortfalls in its application in the Agency.
One fact emerged.-- MBO was applied differently within each directorate
and the OTR course can present only a very general program which
provides a consistent point of departure. This aggravates some
of the issues raised by past seminar participants -- the multiplicity
of systems within the Agency. Senior and mid-level planning officers
are enthusiastic about MBO, but their enthusiasm wanes in the face
of increasing numbers of reports and reporting systems. Rather than
a systematic, integrated approach to management within CIA, they see
a number of systems often duplicative and layered upon one another.
Here again, there appears to be limited control or management of the
process at the top, and the void is often filled by individual inter-
pretations, a conscious decision to ignore it, or a showpiece attempt
at compliance.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Within the DDA the development of better objectives is in order.
An attempt has been made by the staff to work more closely with planning
officers in establishing meaningful objectives, but there is, as one "MR
might expect, room for improvement. One nagging problem relates to the
fact that objectives may not be attacking the tough issues that an
office may be facing during the next year. An additional concern
has to do with the horizontal communications among the offices
wherein objectives impact two or more offices. The planning that
takes place at the directorate level and below is.at times
frustrated by this less than comprehensive horizontal coordination.
Objectives established by one component may be impacted upon by
another component's requirement, both within and without the DDA.
Often, when these requirements are raised, budgets have already
been approved and MBO programs established, making the process of
reestablishing priorities, changing budgets, and setting new
objectives difficult, if not frustrating. (One might suggest or
consider the establishment of a central planning office which could
oversee and coordinate programs on an Agency-wide basis, thereby
improving the efficiency and impact of the MBO process. The top-
level centralized planning office could also be in a position to
examine and evaluate various management systems and hopefully
develop an integrated system complementing MBO, not paralleling it.)
Evaluation of performance within an MBO system is a difficult
process even when performance is measurable in quantifiable units.
In a service-oriented organization such as the DDA, development of
an objective evaluation technique takes on gargantuan proportions.
Given the fact that only about ten percent of the DDA budget is
controlled or monitored through MBO, it becomes difficult indeed to
evaluate total managerial performance. Attempts to evaluate Office
performance either individually or as compared to the performance
of another Office have been very subjective. To help overcome part
of this problem, it is suggested that more of the DDA budget be covered
by the MBO process. This could be done by tracking MBO through more
of the DDA resource packages. As discussed earlier in this paper,
objectives could be cascaded down through the Office and Division
level and serve not only monitoring purposes, but also as a stimulant
to more junior personnel to become more effective.
The letter of instruction is intended to provide another means
of evaluating personnel performance on an individual basis and should
be tied to organization as well as personal objectives. There is no
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
need to regurgitate a comprehensive, detailed list of actions for
each person to follow on a day-to-day basis. The intent of the
LOI is to provide both supervisor and subordinate with a formalized
(in the sense that it is recorded), mutually agreed upon set of
ground rules. From this basis, each is in a better position to
determine what must be done, what assets are available to do the
work, and a means of evaluating it.
While the evaluation process may still seem subjective in nature,
the very fact that the majority of a manager's programs are scrutinized
on a routine basis should tend to improve his performance and provide
the evaluator with a more detailed knowledge of the programs he must
evaluate.
There follows a number of documents which are considered
pertinent to the implementation and improvement of the MBO system
within the directorate.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
PROGRAM EVALUATION AND MBO
AN 0MB PERSPECTIVE
Remarks by
James W. Morrison, Jr.
Evaluation and Program Implementation Division
Office of Management and Budget
Washington, D. C. .20503
to the
Special Workshop on
"Program Management and Evaluation"
at the
Federal Executive Institute
Charlottesville, Virginia
April. 22, 1975
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
PROGRAM EVALUATION AND MBO - AN 0NMB PERSPECTIVE
Introduction
In April 1973, a new emphasis on governmental
management was launched by Presidential memorandum --
MBO was formally instituted and the Office of Management
and Budget was asked to assist the President in carrying
out this new approach. A little more than a year ago,
the Evaluation and Program Implementation Division was
established in OMB and given responsibility for bringing
about improvements in Federal program evaluation prac-
tices. These two events serve to illustrate OMB's
continuing interest in, and responsibility for, facili-
tating the improved management of Federal programs.
I welcome this opportunity to share some of our
current thinking with you. More importantly, I welcome
the opportunity to obtain your views. I hope this
morning's session will be mutually beneficial I'm sure
it will be helpful to us in. OMB..
In keeping with my understanding of the objectives
of this workship, I've titled my remarks "Program Evalua-
tion and MBO - An OMB Perspective." Thus, I want to
focus primarily on Program Evaluation, and to deal with
MBO essentially as background for the discussion.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
I would like to cover the following points with you:
MBO as a management philosophy and the implica-
tions for program evaluation;
The current national focus on evaluation - some
issues and examples;
? Principal
problems evident in the Federal evalua-
tion environment; and
? OMB's current role and initiatives.
M130 as a Management Philosophy
To begin with, let's briefly explore the meaning of
the term "Management by Objectives (MBO)" as it is being
used in this discussion. Basically, IMO represents a
concept -- a management style. It is.a results-oriented
approach to governmental management. It is intended to
be a simple and flexible management framework for focusing
on the President's program. MBO is based on the premise
that line managers -- translate program managers --
w Ji l l
(1) state the end results they expect to achieve
to do a better job if they:
over a specified time period, i.e., set objectives;
(2) periodically evaluate their performance toward
these objectives; and
(3) institute corrective or alternative actions as
appropriate.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Several key points should be stressed.
the selection of appropriate Although
objectives is an important
.output of planning, MBO emphasizes tota
the idea of accomplisllin management
g intended results -- rather
than simply emphasizing the planning function
or the
tracking process. Further
the concept places major
emphasis on the line or program manager. It requires
the acceptance and active participation of program
management -- not merely a designated staff or adminis-
trative organization.
Second, MBO is not a system. It does not impose
rigid input and output formats or specify precise
processes. As noted earlier, it is a management con-
cept that is intended to be flexible enough to erm'
adaptation at all p zt
levels of program management.
Third,
the 0MB role should be noted. With regard
to Presidential level objectives, 0MB as the Pr '
staff agent has a President s
responsibility to participatd in
management conferences, status reviews, etc
the vitality of ? , t:o ensure
} the concept and to serve as a communica-
tions link. However, the use of these objectives to
manage agency activities and to
achieve the intended results
is clearly within the purview of the agency heads. regard to implementing Y ads. With
g ~1B0 within various levels of
agency management, 0MB is merely an advocate.
Just as
Approved For. Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7 4
the agencies apply their own management styles to their
Presidential objectives, so should they determine when
and how they can best employ the concept internally.
The Implications for Program Evaluation
MBO helped to establish a results-oriented environ-
ment within Federal agencies by focusing on a series of
end accomplishments as opposed to a series of ongoing
activities. The emphasis is on achievement -- not
process. This has stimulated increased awareness of the
difficulties involved in measuring results-oriented
objectives.
tr' Too often the emphasis has been placed on
measuring (quantifying) Federal activity itself, rather
than the results of that activity. However, the effec-
tive management of Federal programs requires all of us
to do a better job of determining the substantive impact
of those programs. In this results-oriented environment,
we must be able to see the linkage between definitions
of missions, setting of objectives, identifying tasks,
and monitoring performance. Program evaluation is the
management process which seeks to systematically analyze
Federal programs to determine the extent to which they
have achieved (or are achieving) their objectives.
(That, incidentally, is the working definition of program
evaluation that I'll be using.)
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
The Current Focus on Evaluation.- Issues and Examples
Before delving too deeply into evaluation as a
process of management, it might be well to take a look
at a couple of national evaluation issues facing 0MB
and the executive branch.
1. Federal expenditures for program evaluation have
risen dramatically in the past few years, while
the benefits of these efforts remain questionable.
Beginning with the proliferation of the social
programs of the 1960's, there has been a continuing
shift in total Federal expenditures toward greater
emphasis on human resources. This shift has pro-
duced increasing demands from the President, the
Congress, and the public for greater accountability
and measurement of program success. In response to
these demands, Federal departments and agencies
have expanded their efforts to perform program
evaluations -- either by in-house staffs or through
contractual arrangements. Just how much is being
spent depends largely on which information-producing
activities are defined as program evaluations. A
reasonable estimate is that "evaluation" expenditures
have increased from less than $20 million in FY 1969
to at. least $130 million in FY 1974. The essential.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7 6
point, however, is that despite the increased
activity, there is widespread questioning of the
usefulness of most evaluations to key policymakers
and program managers.
2. Congressional interest in program evaluation has
markedly increased and is reflected in growing
demands for specific information. Many Congressmen
have, in the past, made individual requests for
objective program performance data. Now, however,
there is evidence of a collective intent by the
Congress to obtain and use program evaluations.
The passage of the Congressional Budget and Impound-
ment Act of 1974 clearly imposes explicit require-
ments for increased use of program evaluations. In
response to that bill, the General Accounting Office
is increasing its evaluation staff considerably.
Also, many executive agencies are now receiving
direct requests from various congressional sub-
committees for detailed program evaluation reports,
etc.
It is, therefore, very clear that Federal program evalua-
tion must be accomplished in a results-oriented environ-
ment that is stimulated, in part, by executive branch
actions but also influenced by many external stimuli
as part of a broader trend toward greater accountability
and ii ppeoveIfF f; ease Q /@1I(?7itCWRQR41r9q O RAW9837.
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Principal Problems Evidenced in the Federal Evaluation
Environment
A close look at the Federal scene reveals several
environmental barriers to the conduct of really useful
program evaluations. So far this morning, I have been
discussing evaluation in a fairly general sense. At
this point it might be meaningful to narrow our focus a
bit and further specify the kinds of information-producing
activities on which I intend to concentrate for the b
balance of my remarks.
As we have defined program evaluation, it is intended
to mean those systematic analyses that are quite structured
and usually result in a formal report of findings. In
terms of a time perspective, these analyses may be
summative (analyses of completed programs) or formative
(analyses of ongoing programs). In terms of scope, or
focus, these analyses may be conducted at the national
level (analyses of a major Federal program in-its totality
or two or more major programs which focus on a single
issue) or they may be conducted at the project level
(analyses of some component of a --i~
major program). In
terms of purpose, or type, these analyses can be generally
categorized as:
Approved For. Release 2003/01/27: CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 CIA-RDP81-00261 R000200060001-7 8
? impact Evaluations - designed to measure the
effect of the program on its target (clients), the rela-
tionship of costs to benefits, and any discernable side
effects. These analyses may also seek to examine the
validity of the program's objectives with respect to the
problems it is intended to address.
? Strategy Evaluations.- designed to compare the
relative effectiveness of different major techniques
(interventions) for accomplishing the objectives of
Federal programs. These analyses are intended to provide
information concerning the most effective mix of services.
? Process Evaluations - designed to measure the
operating efficiency of a program. The focus is on
program activities -- not the personal performance. of
program managers.
Without belaboring these categories, it can be seen
that there are several information-producing activities
that are excluded from our definition. Specifically,
it is intended to exclude:
? Policy Analysis - (The assessment of an environ-
mental condition and options to determine an appropriate
course of action.)
? Audit Purely for Fiscal or Legal Compliance -
(The assessment of the propriety of expenditures or the
conformance of legal requirements.)
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
0 Evaluative Research - (Research which focuses
on strict adherence to experimental design and attempts
to systematically control and vary program characteris-
tics and parameters.)
An understanding of these categorizations, and the
specific exclusions makes it possible for us to discuss
Federal program evaluation practices in terms of opera-
tional.problems. Let's look at some of the more prevalent
ones.
Currently, there are insufficient incentives to
stimulate decisionmaker interest in, and demand for,
program evaluation. The political context of rost
major policy decisions and the historical absence of
decisive quantitative data in advance of decision
points have resulted in most decisions being made
without evaluation information. Increasing public
pressures for "better government" and more efficient
use of resources, together with increased congres-
sional interest in program performance are being
expressed in several ways as we have previously noted.
In view of these pressures, it appears that the impor-
tance of competent, independent analyses of program
impact, effectiveness, or economy will become apparent
to agency heads, senior policy makers, and program
managers alike. There is reason to believe that this
will be less of a problem in the future than it now is.
Approved For. Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Most Federal programs do not have measurable objec-
tives or there does not exist testable assum Lions
linking program activities to accomplishment of
program objectives. Since program goals and
objectives are typically stated in rhetorical terms,
there may not be a way to establish causal links, in
an evaluative sense, to determine the effectiveness
of program activities. Further, there may be few,
if any, significant decisions remaining at the pro-
gram management level. "Managing" the program may
mean, in essence, merely carrying out the designated
program activities. This suggests the need to target
evaluations of existing programs to the areas where
systematic analyses are most feasible and has impli-
cations for the types of evaluations to be undertaken.
Also, this suggests the desirability of building
evaluation components into new programs. This point
we shall discuss further and your views are
especially solicited.
? Pro Tram evaluations, as currently performed are
only rarely useful to key-palicymakers and program
managers. Much of the evaluation work now being
done tends to be lengthy, obscure, and irrelevant
to the real concerns of decisionmakers. This sterns
in part from the way in which evaluation projects
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27: CIA-RDP81-00261R000200060001-7,
are conceived and administered and from the lack of
meaningful interaction between evaluation staffs and
the policy makers and program managers for whom
their work is theoretically being undertaken. These
are essentially management problems which merit
further discussion and, again, your views are actively
solicited.
OMB's Current Role and Initiatives
What then is OMB's current approach to the problems
of program evaluation?
0MB has had a longstanding concern with program
evaluation, primarily as an essential element of the
substantive review of programs when developing budget
recommendations to the President. But, by and large, 0MB
has taken -- and maintains -- the position that agency
heads are responsible for managing the programs under
their jurisdictions, and that program evaluation is a
part of such management. Our role, therefore, is essen-
tially non-directive in character and, as in MBO, pri-
marily that of advocate. However, where MBO is a concept
or a philosophy of management which lends itself to the
individual styles of the agencies, program evaluation
(as defined) is a clearly specifiable process of manage-
ment which has already been institutionalized within
departments and agencies. OMB's responsibilities for
Approved For Release 2003/01/27: CIA-RDP81-00261R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
12
improving governmental management dictate a more activist
posture vis-a-vis program evaluation than for MBO.
Accordingly, OMB's evaluation effort has been
structured so that, over time, it should result in a
significant improvement in the contribution of evaluation
to the decisionmaking processes. The current effort is
designed to address the major issues I discussed earlier
and to assist in solving the operational problems I've
just identified. Two central assumptions underlie the
effort:
Program evaluation results have not been as useful
as they might be to major policy decisions con-
fronted by program managers, agency heads, the
Congress, and the President, largely due to
deficiencies in management processes.
? Given increasingly limited national budgetary and
other resources, and the growing need for objective
information on the efficiency and impact of Federal
programs, measures must be taken to build evalua-
tion into new programs and legislative initiatives.
Thus, we have undertaken activities in the following
areas:
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
amr,
 Approved For Release 2003/01/27: CIA-RDP81-00261R000200060001-7
Understanding and Strengthening Federal Evaluation
Practices
A survey of evaluation activities of the major
domestic agencies was completed in June 1974
to provide baseline data on the nature and
scope of current agency practices. An update
and expansion of this survey is underway, this
time as a joint project with GAO.
A panel of senior evaluation officials (generally
the Assistant Secretary level) was established
in late 1973 and meets periodically to exchange
ideas on evaluation issues and to comment on
specific 0MB evaluation initiatives.
On a very limited basis, some technical assistance
has been provided (on request) to agencies in
structuring evaluation systems.
Two draft policy papers have been developed and
circulated for review and comment by selected
agencies. These papers focus on: (1) problems
associated with the planning and management of
evaluation projects, and (2) mechanisms for
integrating evaluation components into new
programs. The papers are intended as reference
points for OMB initiatives. For example, specific
Approved For, action 2'Ob / i/2iq (ti2 }F94-06Y6)l RO015ZOQMg( 1n7- papers arc:
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
- Meetingswith interested members of the
Congress to encourage the inclusion of
evaluation components in new legislation;
- An OMB Circular to provide general
guidelines for managing evaluation activi-
ties and building evaluation components
into new programs;
- In-depth assessments of selected agency
evaluation activities; and
A comprehensive analysis of the evaluation
training needs of Federal executives, pro-
gram managers, and evaluation staffs.
? Liaison with the Legislative Branch
Continuous contact is maintained with Congres-
sional staff members concerned with evaluation,
with the Congressional Research Service, and
with the General Accouniti_ng Office.
? Building Relationships with the Academic, Research,
and Non-Federal Evaluation Community
The OMB staff has participated in a number of
symposia and conferences on evaluation research,
evaluation management, and the use of evalua-
tion in public policy formulation. These
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
activities offer a valuabl. interchange of ideas
 Approved For Release 2003/01/27 : CIA-RDP81=00261 R000200060001-7
That's essentially where we are -- a description of
OMB's program evaluation activities set against the back-
ground of Management by Objectives. I welcome your
comments and questions.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 The following relates to comments by members of the first MBO class
Apabovetd~s~oelhg~ ~ q/ IA-F~~ 81062~c1 00MQ666091 and below
pin terms o an Pa vantages. his composite is of-interest since
it reflects comments by DDA Planning Officers.
DCI/DDA
What's Happening?
1. Verbally supportive of MBO
2. Financial guidelines are set and
do exist .
3. There are DCI objectives - deadlines
are set but.priority and interrela-.
tionships may be unclear.
MBO as theoretically conceived
basically does not exist.
5. Planning.is mainly short-termed
no long-term planning
6. (a) Rely on budget planning with
al ternat iver for s%'stem .
7. The comptroller pulls planning
together as well as can in the
absence of any executive director
and.or a management committee.
8. The comptroller or monitoring system
is not MBO oriented. Most of it
emphasizes APP-PDP personnel
programs.
9. Performance Evaluation not based
on progress made to goals. Largely
subjective and based on day-to-day
emphasis. No LOI between DCI and DDA.
10. Believes in management by exception
delegation of authority and final audit.
Advantages
1. Emphasis of the support
of the idea but no implemeni
tion. Provides deputies wil
maximum freedom to apply
MBO, verbally support also tf
impetus and getting MBO off
the ground.
Approved For Release 2003/01/27 : CIA-RDP81-002618000200060001-7
 Approved For Release 2003/01/27; CIA-RDP81-00261 R000200060001-7
DDA OFFICE
Mutual goal setting is being
done.
(a) Lower levels formulate.goals
consonant with DCI, DDA goals. Know
goals of top echelons 3.
(b) Challenging,"Mickey Mouse"
Communications: structured around
objectives.
(b) Separate informal communication'
going on, some conflict between
the two.
3. Discipline goals are being set and
Advantages
1. -Timely performance
__..2. Know why not getting befo
with top management
4.. Improve morale involveme.
Know respnsibility and
.problems
performance when that is
ood communication
(a) chance to get acqua
Increase accountability ^+
from performance
deadlines met. People feel they are on tt
hook. Etfcrt be; ng made to "stretch" 7. At-e getting improved p 1l;
people.
Better coordination
y/' 9. Continuity of programs rF'
4. Performance Evaluation
(a) Organization - a problem area. less of whose boss
Problem involves how to integrate
MBO related performances with`over-~ O. -Improved identification c'
.all performance. Know overall managerial talent
organizational evaluation system.
at office level.
(b) Individuals have LOI's evaluation
may not be tied to, organizational
performance.
(c) Managers feel that they don't
know how they are doing their
job.
Approved For Release 2003/01/27 : CIA-RDP81-G0261R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Office and Below
What's Happening?
1. Overall impression
(a) mixed applications
(b) MBO emerging - crawling
(c) MBO fades rather quickly at
this level
2. Goals are set - participation
(a) have ABC system
(b) those originated below and
passed up have a mixed
consensus picture
3. Communication - reasonably OK
4. Performance Evaluation same problem
as at DDA/Office level. (Know
overall system)
5. More than a delta system here
from 10 to 80% of action encompassed
by MBO system
Advantages
1. General, same as at DDA/Office
level
2. Special advantages are:
(a) now management not just
supervising
(b) heightened motivation -
.interest - involvement -
participation
(c) broadens their views not sc
narrow
(d) Less resistance and static
to plans
(e) Especially appreciate chanc
to sound off - be ;~.ard at
this lowerlevel.
Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7
 rp oved For Release 2003/01/27 CIA-RbP81-00261R000200060001-7
4/10/74
ELEMENTS OF PROGRAM EVALUATION
1.
PROBLEM STATEMENT
2.
PROGRAM GOAL
3.
PROGRAM OBJECTIVES
4.
EVALUATION CRITERIA
5.
MEASUREMENT/COMPARISON
6.
DATA COLLECTION
8.
IMPACT (EVALUATION-OF THE EVALUATION)
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
??What are the conditions that have justified the program's
existence?
--How well have these conditions been measured?
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7
PROGRAM GOAL
-What is it about the problem that this program would like to
.change?
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
--Are the objectives stated in measurable terms?
-Are they compatible?
"Are they clearly stated?
"Are they overly rigid?
-?Arp they realistic?
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
EVALUATION CRITERIA (MEASURES OF EFFECTIVENESS)
"Are they appropriate?
"Are they comprehensive?
--Would their measurement indicate success or failure?
*Do they take into account unintended effects?
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
fr
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
MEASUREMENT/COMPARI SON
??Are they feasible?
.Are they efficient?
--Are they appropriate?
--Do they assure validity and reliability?
--Is the design sufficient to meet the knowledge requirements
for the users of the evaluation?
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
DATA COLLECTION
..Is the plan practical?
-Would collection violate right of privacy?
--What level of confidentiality is to be assured?
"Who will collect the data? (in house? other?)
**Is the timetable for collection in concert with the program
activities, i.e., too early or too late?
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-0026TR000200060001-7
--Have study delimitations been identified?
--Is the report prepared in a usable format?
--Have the findings been summarized in a clear and succinct
manner?
-Do the data support the findings?
"Are recommendations provided? -
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RIDP81-00261R000200060001-7
IMPACT (EVALUATION OF THE EVALUATION)
"How did the evaluation aid decision makers?
"Did the evaluation do what it set out to do?
"Who benefited from the evaluation?
Was the evaluation cost effective?
Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7
 o Release 200 /01 /27 ; CIA J DP81-00261 F;0002000600 -7
i 1 ist ra io Keview
JULY/AUGUST 1974
FEDERAL PROGRAM EVALUATION FROM THE OMB PERSPECTIVE
Frank L. Lewis and Frank G. Zarb, Office of Management and Budget
Editor's Note: At the time of preparation of this article,
Mr. Zarb was OMB Assistant Director for Management
and Operations, whose responsibilities included OMB eval-
uation policy. That position is now held by Dr. Robert H.
Marik. To develop and carry out this policy, a new Divi-
sion of Evaluation and Program Implementation was es-
tablished within that office, under the direction of Clif-
ford W. Graves.
Program evaluation represents one of the most
useful and essential tools available to federal
officials for assuring rational policy decisions and
effective program management. The purposes of
this article are to provide some perspective on the
evaluation effort currently conducted within the
Executive Branch and to describe some of the
activities which OMB clans to undertake in co-
operation with the various federal agencies to
improve the quality and usefulness of evaluation a
an aid to the decision process.
Evaluation, as used here, is defined as relatively
structured, systematic analyses of operating pro-
grams designed to assess their impact or effective-
ness in attaining their stated objectives, or to assess
their efficiency. Within this broad definition are
four general categories of evaluation, delineated
generally on the bases of their purpose and
predominant methodology. These are. (1) sub-
stantive impact evaluations, (2) relative effective-
ness evaluations, (3) process or management evalu-
ations, and (4) project evaluations.'
Substantive impact evaluations attempt to
measure the impact which federal programs have
upon their stated objectives. This type of evalu-
ation seeks to determine what the program accom-
plishes, how these accomplishments compare to
their intended purposes. and their costs. The
purpose of' such evaluations is primarily to provide
information for use in major policy formulation.
Relative effectiveness evaluations seek to com-
pare the effectiveness of two or more major
program strategies or approaches in attaining
ultimate objectives within a national program.
These studies are designed to help policy officials
and program managers select the most effective
mix of services to maximize programs' total
impact, such as the mix of skill training, remedial
education, and job search assistance in a man-
power program. However, these studies do not
necessarily measure the impact of the total prr'
gram in absolute terms on its objectives.
Process or management evaluations are designed
to measure the operating efficiency of national
programs. They are intended principally to help
program managers achieve the most efficient de-
ployment of available resources, rather than help
policy officials arrive at major decisions affecting
the scope and focus of the national programs.
Project evaluations are directed to individual,
locally based projects which are components of a
national program, regarding the impact or effi-
ciency of the total national program. Project
evaluations may entail any of the three preceding
types (substantive impact, relative effectiveness, or
process evaluation) as well as project rating-
comparing the effectiveness of one or more indi-
vidual projects against others.
The definition presented above is deliberately
intended to focus upon a relatively narrow sub-set
of a broad range of activities which might be
construed as evaluation in a broader context. It
emphasizes the application of scientific method
(research) to program assessment. Excluded from
this definition are certain activities carried out as
part of normal, day to day functions in connection
with policy analysis and development, budget
review, legislative review, and other similar but less
structured approaches. Also excluded are basic
research, data collection, and audit activities, all of
which are closely related to evaluation but not
cat
Approved For Release 2003/01/27: CIA-RDP81-b
b11560001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Evaluation as a Management Tool
There are neither ready nor easy means for
determining attainment of the basic objectives of
most federal programs. Unlike a business firm with
its profit and loss statement, federal programs
confront a variety of conceptual and measurement
problems when attempting to determine success or
failure. A federal program usually has multiple
objectives,, many of which are never clearly stated
Moreover, which objective, or set of objectives, is
paramount is seldom clear. It may in fact differ
among various policy spokesmen-individual mem-
bers of Congress, the President, and the public at
large. Further, two or more objectives may be
incommensurate-i.e., the attainment of one may
be directly counter to the attainment of another.
The difficulty of measuring attainment of
federal program objectives enhances the impor-
tance of evaluation activities. It is our view that
federal evaluation is, or should be, an essential
management tool for policy officials. Good evalu-
ation studies (especially impact and comparative
effectiveness evaluations) not only help to measure
program results but also assist policy officials in
the specification of program objectives. In addi-
tion to the basic management decisions which
program evaluations support (decisions concerning
efficient deployment of resources, and policy
decisions concerning program level and focus)
evaluations assist policy makers in implementing
difficult decisions.
Evaluation Program Examples
The central importance which evaluation
should play in effective program management
makes the proper management of evaluation activi-
ties themselves highly critical. There is no single
best approach toward carrying out an evaluation
program, and approaches vary from agency to
agency. It may be helpful to review briefly some
examples of current evaluation activity.;
Department of Health, Education, and Welfare4
The authorizing legislation for many HEW
programs includes specific provision for program
evaluation. The bulk of the resources identified for
evaluation are administered by the constituent
operating organizations within the Department-
i.e., the Office of Education, the Social and
Rehabilitation Service, etc.
To provide overall direction and focus to the
department's total effort, a central evaluation
office has been established within the Office of the
Assistant Secretary for Planning and Evaluation
(OASPE). Until recently 25 per cent of all
evaluation funds within the Department were made
available at the discretion of the Assistant Secre-
tary, while 75 per cent was to be used by the
constituent agency. This procedure has been modi-
fied recently, leaving the distribution to be deter-
mined on a less rigid basis.
The Office of the Assistant Secretary for
Planning and Evaluation is assigned the respon-
sibility for developing and coordinating the
Department's overall evaluation plan and re-
sources. This includes those funds specifically allo-
cated to OASPE and those reserved for the oper-
ating programs. The purposes of this coordination
are to insure that evaluation activities are related
to the central policy concerns of the Department,
to eliminate duplication of effort and bring about
coordination of complimentary efforts and to im-
prove the quality of HEW's evaluation activities
while making available to all others the expertise
and insights that are gained in any one area.
Evaluation funds to be used at the initiation
and direction of the Assistant Secretary for Plan-
ning and Evaluation serves severs' important pur-
poses. A major portion of the funds has been
directed to projects relating to policy questions for
which the Assistant secretary for. Planning and
Evaluation has been assigned the lead responsi-
bility. This assures integration of the evaluation
activity into the planning and policy development
process. This system of handling the funds also
serves to facilitate a desired balance of influence
between the agencies and the Office of the
Secretary. It enables the OASPE to negotiate with
the HEW constituent agencies during the complex
process of reviewing and approving their evalu-
ation plans to develop some of the more important
details of specific evaluation projects that the
constituent agencies wish to undertake. Finally,
the funds are used to sponsor major "cross-
cutting" evaluations-i.e., those that bridge more
than a single operating agency's area of responsi-
bility.
As it turns out, much of the evaluation funds
which have been reserved for the Office of the
Secretary (out of a total of $40 to $50 million) is
ultimately returned to the operating programs for
Approved For Release 2003/01/27 : CIA-RDP81-00261R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
use at their discretion on evaluation activities.
However, the initial reservation of such funds,
combined with the overall responsibility for man-
aging and coordinating the Departmentwide evalu-
ation activities, supports the capability of the
Department in achieving a more fully integrated
and coordinated evaluation activity directed to
those areas of highest priority both to the Secre-
tary and to the operating program heads.
The Department of Labor
The Department of Labor also has moved to
strengthen its program evaluation efforts by giving
top-level policy support and by establishing a
centralized evaluation staff in the Office of the
Secretary. Recently, the Secretary of Labor estab-
lished a new emphasis on evaluation in order to
provide departmental management with better
information and empirical evidence for making
policy and resource decisions. A goal was estab-
lished to develop performance information for all
department programs during the next four years.
Each agency was directed to strengthen its pro-
gram evaluation capability. The Assistant Secre-
tary for Policy, Evaluation, and Research in
addition was assigned a number of major responsi-
bilities in order to provide centralized direction
and coordination for departmental evaluation ac-
tivities. These include:
(a) Development of an annual evaluation work
plan for the Department.
(b) Approval of all major evaluation projects,
within the Department.
(c) Review of evaluation project proposals and
provision of technical assistance and training to
program agency evaluation staff.
(d) The conduct of independent evaluations of
program performance.
To help finance these activities, the Assistant
Secretary was given authority to tax agency
evaluation funds with the exact amounts to he
determined annually by the Secretary.
A natural division of labor is developing within
the Department between the central evaluation
staff under the Assistant Secretary for. Policy,
Evaluation, and Research (ASPER) and the pro-
gram agencies. The former is concentrating on
substantive impact evaluations and the program
agencies on other types of evaluations. This
division coincides with the difference in the princi-
pal focus of the Office of the. Secretary and of the
managers of operating programs. The program
agencies are concerned primarily with achieving
efficiency within a given program structure and
funding level which requires process evaluations.
The Office of the Secretary is more concerned
about overall departmental resources and the
allocation of these resources among different
programs; this requires program impact and rela-
tive effectiveness evaluations.
The ASPER evaluation staff are concentrating
their attention at present on developing method-
ologies for doing impact evaluations on a regular,
periodic basis using wherever possible existing data
systems. This typically involves combining two or
more data systems. For example, the staff is
testing the possible use of Social Security earnings
records to evaluate the impact of manpower
training programs. Also, the use of equal employ-
ment opportunity reports filed by employers with
the federal government is being tested to evaluate
the impact of the federal contract compliance
programs on increasing employment opportunities
for minorities and women.
As in I11.W, the central evaluation staff works
in close relationship with the policy and program
analysis staff who are responsible for budgetary
and legislative analysis. The close cooperation
between these two groups assures that the evalu-
ation efforts will be relevant to the major policy
concerns of the Department, and that the results
of evaluation studies will be used in the pokey and
program decision process. However, the desig-
nation of a separate unit within ASPER rc be
responsible for impact evaluations allows work to
proceed on a regular basis, avoiding diversion. of
skilled analytical staff to solve immediate policy
crises.
Department of Transportation
The Department of Transportation represents a
somewhat different approach to managing and
organizing program evaluation than reflected in
the preceding examples. In keeping with the
decentralized management philosophy of the De-
partment of Transportation, there is no attempt to
develop a single unified evaluation program for the
myriad activities under the Department's domain.
The head of each of the Department's seven
operating administrations is directly responsible
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
for developing, executing, and utilizing the results
of its internal evaluation program. The Office of
the Secretary has assumed a positive, but limited,
role with respect to the administration's internal
evaluations to assure that a reasonable program is
established and that progress is selectively moni-
tored. This. is accomplished largely through the
Department's program planning proce.' in which
each operating administration presents in summary
form its evaluation in terms of past year accom-
plishments, curent year plans. and new proposals
for the planning year. Finally, the administrations
are requested to utilize evaluation findings in their
participation in the Department's annual program
planning and budget process.
The primary focus for the central evaluation
responsibility with the Department of Transpor-
tation rests with the Office of Planning and
Program Review under the Deputy Under Secre-
tary for Budget and Program Review. This office
each year identifies a limited number of specific
evaluation studies which reflect top management
concern, the results of which are expected to
impact on major planning, budget, and legislative
formulation decisions. For the most part, primary
reliance continues to be placed upon the individual
operating administrations for the actual conduct
of the studies, with overall direction and support
provided by the Office of the Deputy Under
Secretary.. Since the end of 1971, approximately
30 such evaluation studies have been initiated
under this Departmentwide activity.
In one sense, program evaluation has always
been considered a major responsibility for OMB as
a whole, defining the term broadly. Certainly
program assessment is an essential and crucial
element of the substantive review of programs
performed by OMB when developing recommenda-
tions to the President of an appropriate funding
level for an agency's budget. Similarly, results of
past program assessment efforts of OMB staff
enter into advice rendered with respect to OMB's
legislative and management and organization func-
tions.
However, in terms of the narrower definition of
evaluation presented above, GMB. has not in the
past performed a major role in the conduct of
evaluation studies, nor does it do so currently.
Rigorous, systematic evaluation studies have been
performed by OMB only on an occasional basis in
response to a significant perceived need or new
policy initiative
Moreover, it is not contemplated that the role
of OMB in this respect will change matet,alh in
the future. As L:ggesred above, a major outgrowth
of the efforts to devriop improved planning and ui
budgeting systems in the federal government in
recent year has been a substantial expansion of
analytical capability within the various depart
ments and agencies At this point in time. OMB
takes the position that each agency head is
primarily responsible for managing the programs
under his jurisdiction and that program evaluation
o'nstitutes an essential element of effective man-
agemenf ti follows that each federal agency
should be responsible for carrying out its own
evaluation program.
However, concern has been expressed as to
whether the results obtained from many of these
efforts have been as useful and relevant as they
might be to major policy decisions confronted by
program managers agency heads, the President
and Congress.
What then remains for OMB as an institution
with respect to evaluation? Briefly, OMB will
initiate a multi-agency cooperative effort to up-
rade and strengthen the overall quality and
,elevance of evaluation carried out by the various
ederal agencies and to assure that evaluation
results are utilized in arriving at major policy
decisions.
In pursuing this objective of upgrading the
overall quality of federal evaluations, a wide range
of OMB resources will be employed, working with
the various federal agencies on a cooperative and
mutually supportive basis. The recent reorganiza-
tion of OMB, designed to strengthen its govern-
mentwide management capability, represents a
critical element in its renewed evaluation thrust
The newly established management associates,
working closely with the program examining divi-
sions, will carry out activities directly related to
program evaluation as part of their overall manage-
ment effort. This will include initiatives to help
.mprove the management of the various agencies'
;nternal evaluation programs, and may include
lirect conduct of occasional evaluation studies
-which respond to specific major policy initiatives.
Approved For Release 2003/01/27 : CIA-RDP81=00261 R000200060001-7
 To foSu?.and d*
A3~1G~t`f~i~i RSrt a e~ ,e '6 1ti 7M8I5At P81-00i941RIY 20OW000--7a continuing effort to
a
v ee es a is ed'a central upgrade federal evaluation. In addition to simply
Evaluation Division under the Associate Director identifying resources currently available, it will
for Management and Operations. This division will provide a means for determining which efforts are
develop the internal capability to stay abreast of yielding useful results and where improvements are
latest techniques and developments in the manage- required.
ment and performance of program evaluations. It
will utilize this capability to identify strengths and 3. Cross cutting Budget Review of Evaluation
l Activities
weaknesses in
t f
d
f
curren
e
era
e
forts and support
OMB's total effort to upgrade the overall quality
of federal evaluation.
Major OMB Evaluation Initiatives
The following discusses some specific major
evaluation initiatives which OMB has initiated, or
is now considering, to carry out its broad mission
of upgrading the state of the art of federal
program evaluation. These specific initiatives
should not be viewed as isolated activities under-
taken in isolation from other OMB functions.
Rather, they are intended to be a part of continu-
ing OMB activity directed at strengthening govern-
mentwide management capability.
i . Consultation with Senior Evaluation Officials.
OMB has recently prompted the establishment,
on a very informal basis, of a panel of senior
federal executives to serve as a forum fo, the
mutual exchange of ideas and developments within
the federal evaluation community. This group.
comprised of individuals familiar with and respon-
sible for evaluation activities within their own
agencies, should be instrumental in fostering the
development of new ideas and approaches to
evaluation and in facilitating the exchange of
information among one another. This group will
be especially helpful to OMB, both in guiding and
directing OMB's own effort and in providing a
mechanism for transmission of findings to the
federal evaluation community.
2. Inventory of Federal Evaluation Activities
Resources currently available for conduct of
program evaluation are substantial, but scattered
through the federal establishment. There exists no
ready means of identifying, describing, and
measuring these activities in a comprehensive and
consistent manner. To fill this gap, OMB will
initiate a comprehensive survey of federal evalua-
tion programs that will inventory and catalogue
the universe of such activities.
The results of this effort will serve as a basic
OMB currently conducts annual reviews of
agency programs when preparing the President's
budget submission to Congress. Evaluation activi-
ties, though large in total, normally represent a
relatively small part of any single agency's entire
budget. As a result, the budget examining process
frequently accords less attention to this manage-
ment function than might be desired in light of its
potential significance for the efficient operation of
the larger operating program.
To rectify this discrepancy, the new Evaluation
Division will direct a cross-agency budget review to
assess the adequacy of funding levels for evalua-
tion programs in a number of federal agencies. The
survey of evaluation programs discussed above will
serve as a useful input to this ,yf'tort The built-in
knowledge of OMB staff especially those in the
examining divisions. will also be utilized to the
full.
4. Review and Comment on Agency Evaluations
Plans
Most agencies currently performing evaluation
activities to any significant extent prepare formal
plans on a periodic basis to direct and guide
individual studies to high priority topics. OMB will
urge that those not already doing so initiate a
planning process, and subject them to interagency
review and comment, utilizing the panel of evalua-
tion executives discussed above.
The purpose of this exercise will not be to
impose a new rigid planning requirement on the
agencies, nor will OMB attempt to exercise direct
control or veto over the content of agencies' plans.
Maximum flexibility wilt be left to each agency
concerning the form and content of such plans.
The major objectives will be (a) to assure that
there does, in fact, exist some planning process
with respect to each major evaluation activity. (b)
to facilitate exchange of various agencies' plans in
order to minimize duplications and maximize
complementarity, and (c) to enable OMB to
influence initiation of occasional studies of par-
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 ApPro8 F
% Releeaase 2003/01/27 : CIA-RDP81-00261 R000200060001-7
ticular inters to an the White House. est bl; h d f
a
h
5. Continuing Legislative Review
The ability to develop useful and timely infor-
mation on a program's impact can be significantly
enhanced by building in an evaluation capability
from the program's inception. OMB intend, to
carefully review all new legislative initiatives to
assure that adequate provision is made for evalua-
tion. and that the evaluation capability be in place
as soon as the new program begins operation.
6. Relationship to Management by Objectives
A may; : management initiative recently
launched by OMB is the identification of major
program objectives of presidential interest among
the major federal agencies and development of a
routine system for tracking their attainment. This
process. commonly referred to as management by
objectives IMBO), has a direct bearing upon OMB's
evaluation activity in two general respects
First. the MBO process will help guide evalua-
tion efforts to tune highest priority policy issues.
The process of identifying and specifying major
short term management objectives for individual
programs will inevitably lead to identification of a
number of major economic or social objectives
which do not readily lend themselves to measure-
ment by routine management information sys-
tems. For instance, we may be able to determine
through existing management systems the total
numbers of persons treated by a drug rehabilita-
tion center. However, such information does not
necessarily tell us how many drug or narcotic
addicts are actually rehabilitated, nor what the
cumulative impact is upon the society at large. It is
to questions of this nature that evaluation activi-
ties are appropriately addressed.
Accordingly, we will review each major objec-
tive identified under MBO to assure that the
broader social implications are specifically identi-
fied and, where appropriate, identify the major
evaluation efforts necessary to measure their at-
tainment. The review of. agency evaluation plans
discussed above will enable us to determine where
new or redirected evaluation efforts are required in
this connection.
The second area of MBO/evaluation relationship
relates to the management of the evaluation
program itself. In addition to identification of
specific management initiatives associated with
large operating programs, MBO objectives will be
is a or t
eir evaluation components-i.e., 0 0
completion of specified major studies with signifi-
cant policy results, or improvement in the overall
management of the evaluation activity. U111,w
7. Monitoring Individual Studies
On occasion. major evaluation studies will be
undertaken by individual agencies which are of
critical concern to the White House and OMB. On
a selective basis, the OMB will identify those of
particular interest and monitor them closely to
assure that the analysis is adequately reflective of
presidential priorities, and that adequate attention
is accorded the evaluation findings upon comple-
tion. The recent evaluation of federal housing
programs leading to a new federal housing policy
conducted by HUD serves as an example. Al-
though HUD was appropriately responsible for
conduct of the study, OMB worked closely with
HUD during the implementation stage.
Current Perspectives on Federal Evaluation
Activities
The evaluation program outlined above for
OMB should permit us ultimately to speak de-
finitively about how well the federal evaluation
effort has performed, which approaches generate
the best results, and what its contribution has been
as a management tool. Currently, our knowledge is
not sufficient to permit this, except in a limited
sense.
Nevertheless, on the basis of limited experience
and investigation to date, we are able to offer
some evaluation efforts, the difficulties encoun-
tered, and some principles for overcoming these
problems.
srriers to Application of Evaluation to Policy
-,visions
The enormous difficulty in developing program
.;tluation products which can make a significant
,antribution to policy decisions should be recog-
4iized at the outset. There appear to be four major
problems in this regard.
Timeliness. To be useful, it is essential that
results of evaluation be available when decisions
are actually made. The short time frame within
which major policy decisions are made does not
allow policy officials to await completion' of a
long, carefully conducted study
Approved For Release 2003/01/27 : CIA-RDP81=00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
2. Reliability. However, it is important to produce
conclusions that will support a credible manage.
ment decision. Client groups and other program
beneficiaries can be expected to attack evalua-
tions which conclude that a program is ineffec-
tive, and such attacks will normally be directed
at the reliability of the data and methodology
employed.
3. Relevance. Too frequently. =tie criticism has
been made that evaluation studies are not
directed to the problems most relevant to major
policy issues.
4. Utilization. Even if the preceding problems are
adequately resolved, there remain formidable
problems in assuring that evaluation products
are utilized in the decision-making process.
There frequently is no assurance that evaluation
products will be made available on a routine
basis to decision makers. Moreover, in ome
cases there may be strong incentives-on the part
of program managers to suolimate evaluation
products which cast doubt on a program's
worth Hither in terms of its intrinsli value or
how well it is administered.
Suggested Management Principles
The problems discussed above do not lend
themselves to solution by any select few manage-
ment or operating decisions, if for no other reason
than that the effort to address one is often at the
expense of others. For instance, if timeliness is of
paramount concern, reliability through adequate
methodology may suffer.
Individual agencies' past efforts to improve the
quality and usefulness of their evaluation appear
to have emphasized a mutually supportive set of
operating and administrative procedures designed
to address each problem individually, but in
concert with other actions so that attainment of
related objectives is not subverted. The following
outlines a tentative set of management principles
which appear to emerge from these efforts. They
should not be interpreted as exhaustive or defini-
tive. A major thrust of OMB's evaluation role in
the immediate future will be to test these and
other principles with a view toward enhancing our
collective ability to generate useful evaluation
products.
I I. Development of Evaluation Capability at Initia-
tion of Program
The problems confronted in obtaining reliable
information upon which to base program evalua-
tions are frequently a major impediment to con-
duct of reliable program evaluations. These prob-
lems are greatly magnified if the data on program
operations and results must be reconstructed from
old records and past experiences. Therefore pr,.
gram evaluation capability should he nuilt into the
administrative machinery from the outset of each
new federal program. This will help assure that
data collection and assessment is performed on a
current basis, and that the results are available
sufficiently early to influence policy decisions
affecting the program's scope and direction subse
quent to its initiation.
2. Staff Adequacy
This is perhaps the single most important
element to a successful evaluation activity, and the
one least realized Staff adequacy must be deter
mined in relation to the specific mission of the
unit (contract, direct study, or oversight) and is
dependent on two variables--sufficient numbers
and overall quality (ability, training, background.
experience).
3. Planning Process
An adequate planning process to outline both
the major focus of evaluation activity, as well as
specific evaluation projects to be undertaken, is
essential to assure relevance to policy decision
needs. In this connection, it is crucial that those
involved in the policy decision process, both
decision makers and staff, be formally injected
into the development, review, and approval of
such plans.
One highly rigorous approach to the planning
process has been advanced by the Urban Insti
tute.s In effect, the Urban Institute urges that
systematic models be developed by evaluation
planning staff to assess beforehand the most
desirable types of studies needed and the most
cost-effective means of obtaining the needed data.
Only after making such estimates, in relation to an
assessment of the likelihood of affecting major
decisions and the probable value to be derived
from such decisions, can intelligent decisions be
made on the precise data collection and analytic
efforts which should be undertaken.
The Urban Institute's proposal may be more
systematic and rigorous than necessary. But it
Approved For Release 2003/01/27 : CIA-RDP81-002618000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
points up the need and significance of an active
planning process to guide each agency's overall
evaluation effort.
4. Organizational/Administrative Integration with
Related Activities
As discussed above, a number of activities are
closely related to evaluation but are not defined as
evaluation in this discussion-e.g., policy analysis.
However, to assure that evaluation activities are
focused on the high priority areas for maximum
relevance, and that each area is mutually suppor-
tive, certain steps can be taken to assure opera-
tional integration.
One of the most fruitful aspects of organiza-
tional integration is between evaluation staffs and
policy analysis units. While each has a particular
objective and operating style, the interdependence
of each is strong. A similar relationship can
profitably be developed with units responsible for
program data reporting systems, though perhaps
on a less integrative basis.
5. Relationship of Evaluation Unit to Program
Administrator
A critical dilemma faced by any evaluation
effort is the likelihood of arriving at adverse
judgments concerning individual program accom-
plishments. Program managers naturally resist such
conclusions and can be expected to argue against
their validity. This leads to the conclusion that, to
assure adequate objectivity, evaluation capability
should lie outside the direct line of command of
responsible program managers.
It can be argued that this will result in
alienation of the cognizant manager from the
results of evaluations and consequent resistance to
adoption of concrete findings. It is also argued
that if evaluation is, in fact, a management tool,
the program manager most directly responsible for
operations should possess his own evaluation
capability.
A suggested principle to be followed in the face
of this dilemma is that evaluation capability
should exist generally at two levels, performing
generally different types of evaluation. Program
managers should have the capacity to undertake or
fund evaluation studies designed to measure pro-
gram efficiency and to identify major operating
problems. Capability for basic impact evaluations,
especially designed to yield "go-no go" conclu-
sions, should be performed by units outside the
scope of the immediate program manager and at a
hierarchical level above the program manager.
typically at an assistant secretary level. This should
assure sufficient "clout" to obtain needed cooper-
ation from program operators, objectivity in per-
formance of studies, and economies of scale to
conserve limited evaluation resources.
6. Follow-up Procedures
A prevalent criticism of many federal evalua-
tion activities is that studies, once completed, are
"put on the shelf" and ignored. In fact, there exist
powerful incentives to avoid wide publication of
evaluation studies which yield negative findings.
To offset this built-in resistance, procedures
should be adopted to assure the routine extraction
from evaluation studies of major operating and
policy implications and to pursue the course of
their implementation.
The approach taken by several agencies is to
prepare for major decision makers (agency or pro-
gram head) a policy implications memorandum
upon the completion of a major study. A com-
plementary approach is to assign one or more staff
members to "follow-up" in a systematic way the
major recommendations for implementation.
7. Operating Practices in Contract Administration
Units engaged primarily in contracting for
evaluation studies frequently involve themselves
almost exclusively in the ministerial functions of
contract administration. Aside from the obvious
impediments such practice has to effective recruit-
ment of qualified staff, the practice almost guaran-
tees, in the large, studies irrelevant to the needs of
the funding agency because contracting organiza-
tions normally are not sufficiently close to the
federal agency to comprehend its needs fully.
To assure maximum relevance, the following
general practices should be followed:
(a) Requests for proposal (R FPs). should be
highly directive indicating riot simply the general
questions for which answers are sought, but
references to the problems being encountered by
the program, and an initial outline of how the
study should proceed.
(b) During conduct of the study, one or more
federal staff should be assigned to work with the
contractor on a continuing basis, almost as a part
of that contractor's own staff. In this way, the
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 deA*r dcFaReWatW2008901&7udCnb4sRDP81-0Q2Ee11RQ(1 2QIQ6 W Ieim analyses discussed
perception of its reliability, is vested in staff of the
federal agency.
(c) As suggested above, upon completion of
the contracted study, the federal staff should
prepare its own implications analysis.
Another approach which has been suggested,
but on which there has been little experience, is to
use contractors primarily for specialized data
collection purposes, leaving to the federal staff the
primary responsibility for analysis. The logic for
this approach is that federal staff are best
equipped to perceive the decision needs to which
the analysis is designed to contribute. Contractors
are in a favorable position to collect special data
because of sudden start-up and contraction prob-
lems which federal agencies do not possess suffi-
cient flexibility to handle.
8. Need for Both Long Term Basic Evaluation and
Short-Term Reactive Evaluation Capability
As suggested above, a fundamental issue con.
cerning the appropriate focus of evaluation is
whether it should focus on short-term analysis
designed to meet immediate policy needs or
proceed within a longer time frame to assure
adequate quality and methodology. Resolution of
this issue can perhaps best be realized by following
both courses of action simultaneously:
(a) Basic long-term evaluation can frequently
best be performed, at least in part, through
contract arrangements. However, it may be a
mistake to treat each evaluation study as a
separate, independent, and self-sustaining entity.
An alternative strategy is to develop an accretion
of knowledge over time for a given program or
functionally related group of programs. In this
way the attacks on the validity of individual
studies can be handled best-i.e., while individual
studies might be attacked on methodological
grounds, it is more difficult to sustain criticisms
across the full scope of evaluations performed on a
group of programs. The approach also recognizes
the difficulty of securing unambiguous results
from individual studies, even with the most careful
methodology. This approach implies still greater
importance and significance for adequate planning
of evaluation activities..
(b) Short-term analysis responsive to immedi-
ate needs of major policy officials generally can be
performed best by federal staff. Such studies will
frequently be less well structured and sophis-
under (a) above. However, valid insights and
impressions, based upon intensive scrutiny and
examination, can be derived in many instances
that will meet short-term decision needs. These
studies should, to the extent possible, rely upon
the longer-term analyses already completed, or
surface problems or issues which require longer-
term analysis.
Summary .
The major problems associated with program
evaluation in terms of their impact on policy
decisions appear to be four-fold: (1) timeliness, (2)
quality, (3) relevance to policy needs, and (4)
utilization.
A productive administrative approach toward
coping with these problems is, of necessity, multi-
faceted. A set of operating principles is suggested
as a composite, though not exhaustive, package to
achieve a well-managed evaluation. activity. These
principles can be summarized as (a) development
of built-in evaluation capability from the inception
of each new federal program; (b) adequate staff to
assure quality of output; (c) sufficient planning to
focus attention on the high pay-off areas; (d)
follow-up procedures to assure that results are
utilized; (e) organizational alignments to assure
full integration with the policy decision process
and related and supportive activities; and (f) a
proper balancing of long-term and short-term
efforts (in-house and external) to meet both
quality and timeliness needs. In short, effective
management of evaluation is contingent upon
simultaneous efforts on a number of fronts.
The validity of each of these principles, de-
velopment of others, and detailed approaches to
attaining each will require substantial further
consideration, and specific implementation ap-
proaches will vary among agencies. The effort of
OMB in the coming months and years will be to
test and advance these principles in general and in
specific agencies, utilizing the management initia-
tives discussed above.
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Notes
1. The discussion in this section is directed primarily to
evaluation as it should exist, which may not conform
to the existing situation in all cases. For instance, in
the definition of evaluation, reference is made to
programs' "stated" objectives. However, as suggested
later, a major obstacle to performance of useful
evaluations is the absence or a clear statement of
objectives. Similarly, the types of evaluation described
in this section are not necessarily performed by some
agencies, but, in our view, should be.
2. The types of evaluation described generally conform
to those which have been described in other studies on
evaluation. See particularly Joseph S. Wholey, John
W. Scanlon, Hugh G. Duffy, James L. Fukumoto, and
Leona M. Vogt. Federal Evaluation Policy: Analyzing
the Effects of Public Programs (Washington, D.C.: The
Urban Institute, 1971).
3. The discussion of evaluation systems provided below
are based upon descriptions provided by each agency.
The descriptions are intended to provide an illustra-
tion of some existing federal evaluation programs.
However, these should not be construed necessarily as
models of examples of ideal approaches to federal
evaluation.
4. For a more complete discussion of HEW's evaluation
system and examples of evaluation studies, see:
Laurence L. Lynn, Jr., "Notes from HEW," Evalua-
tion, Vol. 1, No. I (Fall 1972), pp. 24-28.
5. John W. Scanlon, Ralph E. Schwartz, John D. Waller,
Thomas W. White, Reynold B. Madro, and Joseph S.
Wholey, An Evaluation Strategy and Plan for the
Department of Labor (Washington, D.C.: The Urban
Institute, January 1973).
I
L
? ORGANIZATION AND MANAGEMENT EVALUATION
? OPERATIONS AND SYSTEMS ANALYSIS
? MANPOWER MANAGEMENT SYSTEMS DESIGN
? POSITION CLASSIFICATION AND PAY PLANS
^ TASK ANALYSIS AND CAREER LADDER DESIGN
? RESOURCE PLANNING AND COST ANALYSIS
? FACILITIES PLANNING
IS GOVERNMENTAL SERVICES RESTRUCTURING
? INFORMATION SYSTEMS DESIGN
7600 OLD SPRINGHOUSE RD., MCLEAN, VA 22101
(703) 8812320
LOS ANGELES, CAL OAK BROOK, ILL
/ X,%X
Task Analysis AtIirmatwe Action
Job Restructuring Perlormancn Evaluation
Upward Mobility Manpower Plann,n9
Personnel System Modernization
1 7 1 7 Massachusetts Ave., N.W., Suite L L 5. Washington, D C 20036
1202) 4835000
681 Market SI., Suite 523. San Francisco, Caldorrna 94105
[phone]
THE JACOBS COMPANY
PUBLIC MANAGEMENT CONSULTANTS
Staff Traieilp and Teelldeal Ani since
In
HUMAN RESOURCE MANAGEMENT
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 STAT Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
6 'Iay 1975
ic: oRAa I.)[ .t i C);t: Assistant for I esources
SUBJECT . DIVA : ianagement Conference M May 1975
I. In connectioi2 wit'll the forthcoming I)1)/A ''ianag efiierht Conference,
you asked that. I prepare a paper that addresses the :impact of short
deadline requirements levied on [iD/A components by the DU/A staff.
The approach I used in gathering data. 1,%,as simple and direct; I con-
ta.cted the Executive and Planni.n~- Officers in the Directorate and
asked for their candid and thoughtful assessments concerning the
tasking from the I)%i/A staff. To begin, I determined that a tote cI?tete
situation would encourage candor; I did not ask for written views.
Thus, what follows is based on notes and my impressions formed during
the course of discussions.
2. At the outset the officers interviewed recognized that the
AK` ency is undergoing an extraordinarily difficult period as a conse-
quence of the investigations into Agency activities by the Presi-
denti,al Coarmlission, Senate Select CcT,:mittee, ctc. Thus, the
requires eats levied In support of the investi ations, which normally
carry a short fuse, are not a factor in this gaper.
7. Only one Office, OJCS, reported that it 1',%as experiencing
difficulty in responding to tasking from the DD /A staff. I hasten
to adci, however, that a pattern of concerns did surface in the
other C {{ices, a.lthou} h few were tied to quick response situratiens.
Sc1, tee of those are the age-old bur -a4 oos steeped in personality
ei.ffe.relices, others are prob loins which, I believe, ar a -;erhent may
wish to contemplate. The ();ICS problem is one of long standing.
Having served a tour as a. staff officer in CJCS some years ago, I
recall vividly the miserly allocation provided for staff support,
The Office continues to increase in 'f/C) ?-- up 78 positions i.n the
last several years -- while the management support for the Office
has remained static. The Office is contl.110. ly beset by burgeoning
regm_re:< r.ts for amore and faster computer service. Consequently,
when 'I'/C) increases arc approved, the slots arc thrust into the
operation to satisfy customer demands. This coupled with 1;lana e
melat?s growing information needs, e. ., iit,), PUTT, more detailed
Program Calls, etc., pat the mariaz f',l,le nt'staff personnel in a
Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
 Approved For Release 2003/01/27 : CIA-RDP81-00261 R000200060001-7
posture of being on the five-yard line wit l tell yards to go on [ourth
uuw 1, 1.e., p lint for:Iatlon. TIe tasks levied by the DD/A staff are
not over A- oworing in and of themselves, nor does C'JCS get o, disp ro-
>ortionate share of tasks . The rrob leer then, as I see it, is one of
resource allocation within COCS. (OJCS is due a 14-positicil T/O
increase co; urencin~ FY-76 alai, as I understand it. none of these
i;osi_tions are presently pro(;rarruied for management support.)
4. Aside from OJCS, no major problems surfaced with regard to
Di;/A staff tasking. Each component reported that it had adequate
staff to handle DD/A requirements and that tasks were neither
excessive in number nor unreasonably short on response time.
Obvious ly, there are times when problems do arise with respect to
meeting ceauliries; however, the opinion voiced airiest unanimously
was that the DD/A staff was reasonable in negotiating revised time-
f,nmes. As I indicated earlier, several areas of concern did emerge
which I believe warrant mana