INTELLIGENCE MONOGRAPH NATIONAL ESTIMATES: AN ASSESSMENT OF THE PRODUCT AND THE PROCESS

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80-00630A000300040001-3
• Release Decision: RIPPUB
• Original Classification: C
• Document Page Count: 127
• Document Creation Date: December 15, 2016
• Document Release Date: September 24, 2002
• Sequence Number: 1
• Publication Date: April 1, 1977
• Content Type: NE

CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
G7
cD
2
flJL3/i\L
NATIONAL ESTIMATES:
AN ASSESSMENT
OF THE PRODUCT
AND THE PROCESS
a ~ CENTER FOR THE STUDY OF INTELLIGENCE
CENTRAL
ENTRAL INTELLIGENCE AGENCY
CENTRAL LLIGENCE AGENCY
1977 TR/IM 77-03
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
CONFIDENTIAL
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
THE CENTER FOR THE STUDY OF INTELLIGENCE IN OTR OPERATES
A RESEARCH AND DISCUSSION PROGRAM KEYED TO THE PROCESSES
AND FUNCTIONS OF INTELLIGENCE. THE OBJECTIVE OF THE CENTER
IS TO CONTRIBUTE TO THE PROFESSIONAL UNDERSTANDING AND
TO THE RECORD OF THE ART OF INTELLIGENCE. RESEARCH PROJECTS
ARE UNDERTAKEN BY INTELLIGENCE "FELLOWS"--VOLUNTEER
OFFICERS FROM ACROSS THE AGENCY ON FULL-TIME DETAIL TO
THE CENTER. INQUIRIES ABOUT THE CENTER PROGRAM, OR COMMENTS
ON THIS REPORT ARE INVITED BY THE DIRECTOR/CSI, EXTENSION I
25X1
25X1
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
April 1977 TR/IM 77-03
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
PREFACE
SUMMARY
1. SETTING THE STAGE
II. ESTIMATES AS AN INSTRUMENT OF POLICY
SUPPORT
Page
The Traditional Doctrine 23
How Estimates Fared 25
The Negative Side
The Positive Side
Other Findings
Why the Gap? 34
Some Unrealized Assumptions
Other Reasons
The Market for Estimates 40
What Should Estimates Do?
For Whom Should Estimates Be
Written?
How Should Estimates Be Written?
III. IMPORTANT ELEMENTS IN THE PRODUCTION PROCESS
FOR NATIONAL ESTIMATES 51-79
Self-Initiated Estimates 51
Terms of Reference 52
Multidisciplinary Work 54
New Analytical Methodologies 57
Competing Analysis 59
Net Assessments 63
Coordination 66
Dissent 70
Summaries 72
Classification 73
Written and Oral Presentation 74
"Post-Mortems" 76
Post-Publication Update 79
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
Page
IV. ORGANIZATION FOR THE PRODUCTION OF ESTIMATES 80-91
Internal Collegial Review 85
Review by Outside Consultants 87
Changes in Current Practices 89
V. THE USER'S ROLE IN ESTIMATES 92-102
Closer User-Producer Links: Risks
and Benefits 94
Providing for Closer Contacts 97
VI. THE FUTURE ROLE OF ESTIMATES 103-107
APPENDIX A: Study Methodology 108-113
Consumers 109
Producers 110
Use of Estimates and Other Writings 111
APPENDIX B: Selective Bibliography 114-117
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
The impetus for this study derived mainly from two factors.
First the intelligence community had never taken a comprehensive
look at the role of national intelligence estimates in the
decision-making processes of government. Secondly, estimates
have been increasingly subject to criticism in recent years,
and a serious effort to evaluate them seemed in order. Con-
sequently, a study team composed of experienced Agency officers
was assembled and work begun in July 1976, under the auspices
1
of CIA's Center for the Study of Intelligence.
The intelligence community produces a wide variety of
products which make judgments about the future, ranging from
crisis reports and short-term analytical articles through
National Intelligence Estimates (NIEs) and similar interagency
products. The study deals only with the last of these, i.e.,
interagency papers produced by National Intelligence Officers
The team was composed of member of the former
Board of National Estimates and a National Intelligence Officer
(Ret.); of the former Office of Current Intelligence,
Directorate of Intelligence, and presently Director of the Center
for the Study of Intelligence in the Office of Training; II
25X1A Il of the Directorate of Operations, who served with the Office
o ational Estimates in its early years and recently com leted
a tour on the National Security Council Staff; and of
the former Office of Current Intelligence, now assigned to the
Current Reporting Group, Directorate of Intelligence.
i
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
25X1A
25X1A
25X1A
25X1A
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
(NIOs) on behalf of the Director of Central Intelligence (DCI)
(although some findings of the study may be germane to other
products). Included are NIEs and Special NIEs, now produced in
considerably fewer numbers than in former years. Also included
are National Intelligence Analytical Memoranda (NIAM) and
Interagency Intelligence Memoranda (IIM), which have existed
only since 1973. The NIAM, infrequently used, is lengthier
and more detailed than the typical NIE. The IIM is produced
under the aegis of the DCI, but is not passed upon by the
National Foreign Intelligence Board (NFIB) and is a less
formal and more flexible publication than the NIE.
The study methodology adopted placed considerable emphasis
on conducting interviews with a wide range of users and pro-
2
ducers of estimates. Valuable supplemental and background
information was acquired from existing classified and unclas-
sified literature about estimates and from published estimates.
Devising a methodology for evaluating estimates themselves
proved impracticable, for the reasons given in Appendix A.
Appendix A also contains a fuller explanation of the method-
ology used in the study, as well as information on the 66
interviews conducted by the study team. A selective bib-
liography is included as Appendix B.
It is important to remember that the interviews were con-
ducted prior to the change in Administration which occurred
in January 1977 and that the comments apply to the intelli-
gence and policy processes which existed at that time.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
It did not prove easy to generalize about the responses
of those interviewed. Time did not permit us to cover all the
issues with each of the individuals interviewed. The responses
covered a wide spectrum, and the respondents did not fall
neatly into categories; typically, reactions to estimates
were mixed. It was also necessary to allow for personal dif-
ferences, e.g., essentially the same view might be expressed
by one person in low key and by another with vigor and passion,
or differences in view might be disguised by subtleties of
language. Further, a study of this kind tends to invite a
critical approach, as does the asking of questions in an inter-
view situation. Efforts were made to guard against these and
similar hazards, and to concentrate in the report on certain
recurring themes which rather clearly emerged from the interviews.
The report covers three broad topics: the product and
its audience, the process of producing estimates, and the
relationship between producer and user. The first topic is
considered in Chapter II, which sets forth the views of users
concerning the quality and utility of estimates, and con-
trasts these views with the traditional perception of the
role and function of estimates. This chapter takes up two
other matters: what estimates should attempt to do for the
reader, and for whom they should be prepared. The estimative
process is discussed in Chapters III and IV. Chapter III
discusses certain specific aspects of the production process,
such as methods of presentation and the coordination process.
Approved For Release 2002//11/04 : A-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Chapter IV covers the question of the organization for pro-
ducing estimates. The relationship between producers and
users is taken up in Chapter V, which discusses, inter alia,
the benefits and risks of close association, and the means
by which the needs of users can be conveyed to producers of
estimates. The final chapter has some thoughts on the future
of estimates.
Gratitude is tendered to the many people who agreed to
he interviewed and who so generously contributed time and
thought to this study. It could not have been prepared with-
out them.
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
I. SETTING THE STAGE (pp. 19-22)
National Intelligence Estimates (NIEs) have changed over
the years from short, narrowly focused papers dealing with
near-term problems to include more comprehensive, analytical
studies of longer-range issues. This change resulted from:
--greater analytical capabilities;
--an enlarged data base; and,
--changing requirements, as readers grew more
numerous, more sophisticated, and more demanding.
The role and importance of estimates in policy making has
varied with Administrations, particularly in response to:
--the structure of the policy making machinery
and the place accorded estimates;
--the attitude of top policy makers toward intel-
ligence; and,
--the quality and relevance of estimates as
perceived by the principal users.
Reaching their zenith in the early 1960's, estimates sub-
sequently declined in prestige and drew increasingly sharp
criticism. The criticism contributed to the decision of the
Director of Central Intelligence (DCI) in 1973 to abolish the
Board and Office of National Estimates (ONE) and entrust the
preparation of estimates to a group of National Intelligence
Officers (NIOs).
Approved For Release 2062A1lO4 NCIA=RDP8O-OO63OAOOO3OOO4OOO1-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
II. ESTIMATES AS AN INSTRUMENT OF POLICY SUPPORT (pp. 23-50)
The criticism of estimates has continued since the change,
however, and this chapter sets forth the views of policy makers
and other consumers on the present quality and utility of esti-
mates and comments on how and for whom they should be written.
The Traditional Doctrine (pp. 23-24)
National estimates were intended at their outset to be the
most authoritative appraisals available to the top levels of
government on foreign developments of national security concern.
Issued by the DCI, they were to be forward looking and predictive,
rather than historical and descriptive, of high quality and
objectivity, and national products with respect to subject
matter, audience, and process of production. The interviews
for this study were conducted with this doctrine in mind and
the results were measured against it.
How Estimates Fared (pp. 25-34)
Against the traditional standard, estimates did not fare
well. Although highly praised by some users, and found use-
ful in one way or another by most others, they were judged
in the aggregate to fall well short of the traditional ideal.
They clearly have not played the important role envisioned
for them in the national security decision process.
The Negative Side (pp. 25-28). Estimates seldom reach
the top levels of their intended audience--the President and
members of the National Security Council (NSC). Such of their
Approved For Release 2002/1C194 9 IUi I- E '80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
content as does is usually included in memoranda or briefings
prepared at lower levels, where estimates are widely received,
and read to varying degrees. The complaints about estimates
focused on quality and relevance; estimates were criticized
for being:
--irrelevant to, or oblivious of, the specific
policy problems of the readers;
--insufficiently analytical and overly descriptive;
--conservative and imprecise in their judgments
about the future;
--inadequate in explaining judgments and con-
clusions, and in discussing the alternatives
considered and discarded; and,
--unable to contribute much that is unique or
not already known to the policy maker, par-
ticularly on political subjects.
The Positive Side (pp.28-30). Some respondents were high
in their praise of estimates and others were on the whole well
satisfied with them; almost all found them useful in one way or
another. For example:
--those with strongly positive views, including
two former cabinet members, tended to be less
concerned about the relevance of estimates to
immediate policy issues, and valued them for
their presentation of a disinterested view;
--others with a positive view expected less of
estimates, and were not troubled by the
deficiencies perceived by the more critical;
--most users valued estimates for their balance
and professionalism, because they pulled
together all that was known about a subject,
and because they helped assure the reader
that he had considered all the factors
bearing on a problem.
Approved For Release 2002/11/04: CIA RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Other Findings (pp. 30-34). Estimates got good marks
for objectivity--some users considered this one of their
principal virtues. Most users who said they had observed
bias, considered it a comparatively minor problem, easily
discerned and adjusted to, and the consequence of human
imperfection rather than of deliberate intent.
It proved difficult to get the views of users on dif-
ferences in the quality and utility of estimates since the
1973 change in the production system, because of the turnover
of officers in policy positions. There was, however, a fair
degree of consensus between users and producers on two points:
--the NIO system has produced a modest improve-
ment in the relevance of estimates and some-
what greater improvement in-the responsiveness
of the system; but,
--estimates are more uneven in quality than those
produced before 1973, because of the new drafting
procedures.
The interviews revealed different reactions to different
kinds of estimates:
those on military, scientific, technical and
economic subjects were better received than
those on political subjects, not because of
differences in quality, but because most
users were less able to handle the complex
data, perform their own analysis, and reach
their own conclusions.
We found little or no support for criticism heard in
recent years concerning:
--the proliferation of intelligence publications
containing estimates;
Approved For Release 200111A" ~ TA-DP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--the issuance of departmental as well as national
estimates; or,
--the absence of an explicit scale of probabilities
in estimates.
Why the Gap? (pp. 34-40)
The substantial disparity between traditional expectations
and what users said is attributable in large part to deficiencies
in the product:
--the failure to be fully responsive to the policy
question;
--the failure to be sufficiently venturesome; and,
--inadequacies in drawing implications for U.S.
policy.
Some Unrealized Assumptions (pp. 35-38). There are
other reasons for the disparity, which individually and
collectively are of considerable importance. A basic one
is that the traditional doctrine puts an unrealistic burden
on the DCI and the intelligence community. It rests on some
unstated and unrealized assumptions.
One is that estimates would have a major influence on
the formulation of national security policy:
--in fact, estimates have played only a modest
role, partly because
--security policy is not directly driven by facts,
analyses and resulting judgments; it is the com-
plex product of an often lengthy and untidy pro-
cess, in which many other considerations come
into play.
A second assumption is that policy makers would seek and
welcome the contributions of estimative intelligence, even
when they cast doubt on current policy:
Approved For Release 2002/11/ T1-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--in fact, policy makers often believe that they
can estimate as well as the intelligence spe-
cialist, if not better; also,
--estimates may say unwelcome things and cast
those who produce them in the role of
troublemakers.
A third assumption is that the relationship between policy
and intelligence would be close, and communication free and
complete:
--this relationship has usually not existed; it
has been very weak in recent years; moreover,
--there is an absence of structure for system-
atically insuring that estimates are part of
the policy process.
Other Reasons (pp. 38-40). Two other circumstances have
contributed to the failure of estimates to play their prescribed
role. One is that the foreign policy establishment tends to be
highly operational, and to focus on the short term and highly
specific matters immediately before it. However:
--estimates were originally intended not merely
to support day-to-day operations, but as con-
tributions to the formulation of basic, long-
term national security policy;
--from this perspective, the difficulty may be
with the way policy is formulated. Government
institutions, such as State's Policy Planning
Staff, which were designed to assist with long
range policy, usually focus instead on short
term issues.
Secondly there has been suspicion and distrust of estimates
at the top, and this has had serious effects on their use.
--Unless estimates are welcomed and read at the
top, they are not likely to be taken seriously
elsewhere.
CONFIDENTIq
Approved For Release 2002/11/04: CIA-I P8O-OO63OAOOO3OOO4OOO1-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
The Market for Estimates (pp. 40-50)
Even the most critical users believe that estimates have
a role to play in the policy process.
What Should Estimates Do? (pp. 42-46). Users believe
that estimates should:
--identify policy problems not yet in the fore-
front, and pose questions about them;
--sort out the facts in complicated situations
where such facts are elusive, apparently con-
tradictory or fast changing;
--identify and evaluate the forces at work and
their interplay, and discuss how their con-
tinuation or manipulation could affect the
final outcome;
--judge the consequences of ongoing developments
for U.S. policy; and,
--judge foreign reactions to U.S. policies, present
or contemplated.
There was a relative lack of interest in specific pre-
dictions of future events such as coups, elections, or changes
of government.
Most striking about these comments is their reaffirmation
of the traditional doctrine, with its emphasis on the analysis
of forces, trends and their implications for the U.S. in a
context analytical and forward looking, rather than descriptive
and current.
For Whom Should Estimates Be Written? (pp. 46-48). The
nature of the audience is important, for it affects the way
7
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP8O-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
estimates are organized, their level of detail and how they
present facts, analyses and conclusions.
Some of those interviewed thought that estimates should
be written with readers at the very top in mind--the President,
the NSC, and other cabinet members--not because estimates would
regularly be read there, but because so aimed, they would catch
other readers along the way.
-But the perspectives and needs of those at
the top are not necessarily the same as those
of the individuals who support them.
--Some estimates will be read at the top, par-
ticularly those that the DCI believes should
be read there, and which he urges upon the
senior policy makers.
Most respondents believed that estimates should be aimed
at the Assistant Secretary of State, White House Staff or com-
parable level, and we agree.
--These are the officials who set the terms for
the policy debate by formulating the options
and alternatives, who enjoy the confidence of
policy makers at the highest levels, and who
constitute the highest level combining expertise
and the power to act on many problems.
How Should Estimates Be Written? (pp. 48-50). Such
readers are able and knowledgeable, have access to the same
material as estimators, and feel competent to reach judgments
on the basis of their own analysis. If estimates are to
appeal to this audience they must:
--emphasize analysis rather than description,
show the relationships among data, analysis
and conclusions, and describe the thought
process by which the estimators came to
their judgments;
Approved For Release 2002/11/640NtiA- 066-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--explain what issues were contentious and what
was disgarded and why, and set forth any dif-
ferences of opinion;
--describe continuity and change as compared
with previous estimates, and identify earlier
material now judged incorrect; and,
--clearly state the implications of their
analysis and conclusions for U.S. policy.
III. IMPORTANT ELEMENTS IN THE PRODUCTION PROCESS FOR NATIONAL
ESTIMATES (pp. 51-79)
This chapter discusses the views of producers, some users,
and the study team on aspects of the process by which estimates
are produced.
Self-Initiated Estimates (pp. 51-52)
About half of all national estimates are initiated by the
intelligence community and most producers and users who com-
mented on the subject found this satisfactory. A very few
felt strongly that, to avoid irrelevant papers, estimates
should be produced
study team believe
estimate when they
for U.S. policy.
Terms of Reference
terms
of an
ents.
The degree to
of reference
only on request, but the majority and the
that producers have a duty to initiate an
perceive a development of significance
(pp. 52-54)
which users participate in preparing the
is likely to determine the real relevance
estimate to the needs and interests of its main recipi-
A formalized procedure providing for such consultation
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
should be instituted. The risk of predetermining the outcome
by a conscious or unconscious "loading" of the questions
appears minimal and manageable.
Multidisciplinary Work (pp. 54-57)
Users and producers alike agreed on the need for more
cross-disciplinary or analytically-integrated estimates, and
on the difficulty of doing them. A common complaint was that
readers were left to synthesize separate sections on political,
economic, military and other subjects. Bureaucratic separa-
tion and increasing specialization among analysts were cited
as the causes for this situation. There is, however, little
agreement on how best to accomplish good multidisciplinary
synthesis. At a minimum, it probably requires bringing
various analysts together under an effective project leader
for wide-ranging "synthesizing discussion" before drafting
begins.
New Analytical Methodologies (pp. 57-59)
Although some critics fault the estimative process for
not incorporating more quantitative, mathematical, and
systems-oriented methodologies, we found little support
for this charge. There was a great deal of skepticism about
the use of computers and other new tools, and producers saw
some risks in the use of new methodologies. Nonetheless, it
is important to keep up with the state of the art, and some
new techniques appear to have at least limited applicability.
10
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Competing Analysis (pp. 59-63)
Another concept that has received attention recently
is the creation of "competing centers of analysis." The only
extensive effort to provide an alternative analytical approach
was the widely-publicized and controversial "B-team" experi-
ment. Undertaken last year during the preparation of the
annual estimate on Soviet strategic forces, it involved a
team of experts from outside the intelligence community.
There are many practical problems in such an effort, however,
and the best insurance of proper analytic competition probably
lies in the skill, perception, objectivity, and intellectual
rigor of the estimative manager rather than in any organiza-
tional technique.
Net Assessments (pp. 63-66)
Net assessments involving the U.S. require access to
data on U.S. forces, weapons and capabilities. Those inter-
viewed agreed fully that the intelligence community should
neither conduct them nor include them in estimates, because
of the inordinate risk of transforming estimators into
advocates or opponents of particular U.S. weapons systems
or policies. Net assessments comparing the capabilities of
two or more foreign countries are an acceptable and at times
essential part of national estimates, and there is need for
more of them.
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Coordination (pp. 66-70)
This process brings together experts from the various
agencies to debate and revise the text of a draft estimate.
Coordination is central to the. concept of national intelli-
gence, allowing the DCI as the nation's chief intelligence
officer to set forth his views while requiring other par-
ticipants either to agree or to express and explain their
disagreements. Substantively, coordination brings the
talents of the entire intelligence community to bear, makes
coverage of the pertinent issues more likely, and, at its
best, helps to define and sharpen issues.
The best way to avoid the pitfalls of coordination, such
as masking divisions and fuzzing conclusions, is to select a
chairman for coordination meetings who is tough-minded, inde-
pendent, judicious and skilled in running a meeting, and to
remember that consensus is often not what users want or need.
Dissent, Summaries and Classification (pp. 70-74)
Consumers welcome the presentation of conflicting views
on controversial and complex matters, and they are sufficiently
sophisticated to distinguish between dissents reflecting sub-
stantive differences and those taken primarily to support
bureaucratic positions. The present trend of incorporating
dissents in the text of estimates, rather than in footnotes,
is clearly favored by users, some of whom want also to see
an elaboration of the rationale behind a dissent.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Consumers emphasized that a tightly written and accurate
summation of the key conclusions is the best possible device
for ensuring high-level attention to the message of an estimate.
Many did not object to lengthy estimates so long as they led
off with a crisp, well-written summary. The importance of a
good summary can hardly be overemphasized.
Policy makers strongly favored the lowest possible security
classification for estimates as a means of widening their
audience and enhancing their utility. Where possible, highly
classified material seems best handled in separate, more
restrictively distributed annexes, so that a lower classifica-
tion can be given to the basic estimate.
Presentation, "Post-Mortems" and Updates (pp. 74-79)
While recognizing that both written papers and oral
briefings have certain advantages, almost all consumers com-
menting on the two methods clearly preferred to get most if
not all estimative material in written form. The bulk of
any effort to improve presentational formats ought therefore
to go into making written estimates as succinct, readable,
and responsive to different levels of need as possible.
Post-mortems can be useful if done sparingly, and if
they include feedback from consumers.
There was almost no support from consumers for a regularly
scheduled revision and update of estimates, except for the
annual Soviet strategic estimate, NIE 11-3/8. In our view,
estimates should be updated only when significant changes
have occurred.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
IV. ORGANIZATION FOR THE PRODUCTION OF ESTIMATES (pp. 80-91)
The current system for producing estimates gives each NIO
a larger amount of authority and responsibility than any one
individual had under the previous system, and it encourages and
requires a close working relationship between the NIO and the
main producers and users in his area.
Both of these results were highly valued by most of the
users who commented on the subject. Some of those interviewed,
however, felt that the present system places too heavy a burden
on the NIO, and that it suffers by lack of provision for col-
legial review, as once supplied by the Board of National
Estimates.
One benefit of the present system, in the eyes of some
producers, is that it puts the drafting responsibility on
analysts who deal. with a subject on a day-to-day basis.
Others believed, however, that the analysts' lack of expe-
rience in estimative writing and the conflicting demands on
their time from other tasks often resulted in a lower quality
estimative product.
Changes in Current Practices (pp. 89-91)
Efforts have been made to respond to some of the criti-
cisms, including the authorization of a panel of outside
consultants to review estimative drafts. Without offering a
detailed blueprint, the study team believes that additional
steps should be taken. These are:
Approved For Release 200:/Y /bPDIAZikbP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--to establish a body of generalists to serve
as an internal collegial. review group;
--to establish a small group of analysts to
do the initial drafting of estimates as a
full time task; and,
--to activate the authorized panel of outside
consultants.
These steps would help insure high quality and the
thorough and complete treatment of a subject. They would
provide an additional means of getting objective and dis-
interested national estimates, and would highlight the
primacy of the DCI. They should not be allowed to impinge
on a strong virtue of the present system: its improved
ability to maintain contact with the consumer and to
insure the pertinence and responsiveness of the estimative
product to the consumer's needs.
V. THE USER'S ROLE IN ESTIMATES (pp. 92-102)
If estimates are to be useful and relevant, producers
must clearly understand the needs of users. Such under-
standing can best be acquired by direct communication on
matters of scope, timing and the issues to be addressed.
An effective dialogue between producers and users would
seem to require:
--clear evidence of interest by the President and
senior policy makers in the use of estimates;
--a recognized procedure for fitting estimates into
the national security decision process; and,
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--active participation by senior policy and intelli-
gence officials.
To date, these conditions have been met only partially and
sporadically. The National Security Council Intelligence
Committee (NSCIC) was set up in 1971 to provide guidance by
users on their needs and to evaluate intelligence products,
but proved ineffective. Since it was abolished in 1976, there
has been no formal mechanism for user-producer exchange.
Despite what the NIOs have done to bridge the gap, intelligence
production and collection are still determined more by what
the producers think is needed than by direct requests or by
guidance from users.
Some believe that intelligence analysts should maintain
a certain remoteness from decision makers to keep intelli-
gence untainted by policy pressures. But most users and
producers took a different view, considering a close rela-
tionship mandatory--its primary benefit being a clearer,
more realistic appreciation by each of the other's capabilities,
limitations and needs.
Providing for Closer Contacts (pp. 97-102)
The major obstacles to closer user-producer relations
are lack of time, physical separation, and a view among some
policy makers that the producers of intelligence are well
enough informed and sufficiently competent to determine on
their own what users want and need. Most users agreed,
however, that efforts to improve communication are desirable.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--One solution sometimes suggested is the creation
of a forum such as the NSCIC, but without its
liabilities. This course is worth trying, but
not a great deal should be expected of it.
It is quite clear that the improvement of communication
will depend mostly on the efforts of individuals, and that
the main burden will remain on the producer.
--Several policy makers stressed the importance
of developing personal relationships of mutual
confidence with intelligence producers;
--Users should do all they can to insure that
the producer is aware of trends in policy
and to understand what estimates can and
cannot do; producers must help them acquire
this understanding and absorb what is
readily knowable about policy concerns;
--More tours for selected intelligence officers
in policy offices would be helpful.
VI. THE FUTURE ROLE OF ESTIMATES (pp. 103-107)
As its power and self-sufficiency become more circum-
scribed, the U.S. will be increasingly dependent on accurate
estimates of the possible plans and actions of its adversaries
and friends. Thus, estimates will have a highly useful role
for the foreseeable future and should get high priority in
the overall intelligence effort.
Estimates will be more difficult to prepare in the
future. With the growing complexity and interrelatedness
of the world's military, technological, economic, political
and social affairs, it will be harder to understand and to
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
foresee crucial developments, to be clear and precise, and
to phrase estimative judgments in simple declarative sentences.
The likely shrinking of intelligence resources will be coupled
with a requirement for estimates on a wider variety of sub-
jects aimed at more and different consumers, including
Congress and perhaps even the public.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
SETTING THE STAGE
National Intelligence Estimates (NIEs),are almost as old
as "central intelligence" itself, having been systematically
3
produced by the United States Government since 1950. In
October of that year, General W. Bedell Smith became Director
of Central Intelligence (DCI) and established an Office of
National Estimates (ONE), together with a nine-man Board of
National Estimates under
University.
In the early years, estimates tended to be short, spare,
narrowly focused, and concerned with events in a very short
time frame. NIE #1, written in November, 1950, was entitled
Prospect for Communist Armed Action in the Philippines During
November. Others written in the first year or two were very
cautious about looking more than a month or two ahead. Esti-
mative judgments were often tendered with little of the under-
lying rationale provided in the text.
Probably the first "national estimate" was prepared under
the direction of Admiral Roscoe H. Hillenkoetter as DCI in
March 1948. Its subject was Soviet intentions in the Berlin
crisis, and its production was a milestone in interdepartmental
cooperation and a forerunner of the system adopted in 1950.
See "March Crisis 1948, Act I," by iri Studies
in Intelligence, Vol. X, No. 4 (Fall 196 , p. 13.
25X1
25X1
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
As analytical capabilities and the data base grew,
estimates also changed. Greater emphasis was placed on the
presentation of data and on the development of the argument.
The forces at work in a situation were discussed and trends
identified and evaluated. Estimates were produced whose
objective was to educate and to provide background.
These changes were stimulated in considerable part by
changes on the demand side. From the beginning, the market
for estimates went well beyond the top policy makers and
their senior aides, and it grew greatly with the passage of
time. It came to include readers with widely varying back-
grounds and requirements. Many of them required sophisticated
analysis, and became increasingly unwilling to accept the
judgments of others without a backdrop of fact, argumenta-
tion and analysis.
In this respect, a major turning point occurred with the
advent of the Nixon administration in 1969. Skeptical from
the first about the estimative product, its senior officials
demanded a complete justification of the judgments rendered,
particularly for military estimates. They also required that
sufficient data and analysis be presented so that the reader
could reach his own conclusions from the material presented.
Estimates had their clearest formal role in the decision-
making process during the Eisenhower administration, when the
National Security Council (NSC) was a more structured and
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
important policy making body than at any time before or since.
The NSC Planning Board was the place where NIEs were brought
together with policy inputs and combined for use by the NSC.
Not every NIE had an impact on policy, but there were clearly
defined institutional linkages connecting intelligence with
the policy process. The potential of those arrangements was
not fully realized, however. Estimates were not consistently
read by high-level officials, and there is not much evidence
that they had a major influence on policy.
With the decline in the role of the NSC machinery under
the Kennedy and Johnson administrations, the systematic
structured connection of estimates to policy was reduced.
Nonetheless, the true heyday of estimates may have been in
the early 1960s. It was then that the White House and impor-
tant Cabinet members asked for and used more estimates in
formulating policy than at any other time. The DCI and senior
intelligence officers had access to the President and other
top policy officials, and the DCI personally played a crucial
role in inserting estimative products into the policy process.
Also, at about that time rapidly expanding technical collection
capabilities began to pay rich dividends, which were reflected
in more comprehensive and sophisticated estimates, especially
on technical and military subjects.
The influence and importance of estimates began a decline
in the mid-1960s. Complaints about shortcomings in estimates
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP8O-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
have been voiced with increasing frequency, particularly
since 1969. Although the themes had been heard in one form
or another from the very beginning, the complaints built to
a crescendo in the late 1960s and early 1970s. There appear
to be a number of reasons, including internal governmental
strains and divisions over Vietnam, a lack of access and
influence with the White House by the DCI, and a belief by
consumers that estimates were increasingly out of touch with
the needs of the executive.
Partly in response to these complaints, DCI William Colby
abolished ONE and the Board of National Estimates in November
1973, and established the system currently in effect. Overall
responsibility for producing policy-relevant estimates, and
for maintaining close contacts with producers and consumers
of intelligence, was vested in some dozen senior National
intelligence Officers (NIOs), who reported to the DCI and
were responsible for particular geographic regions or sub-
stantive specialties. The actual drafting of estimates was
diffused: NIOs could call on intelligence production com-
ponents anywhere in the community to furnish drafters, but
in practice most drafting of non-specialized estimates fell
to officers in CIA's Directorate of Intelligence.
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
ESTIMATES AS AN INSTRUMENT OF POLICY SUPPORT
In setting about to study the role of estimates and in
evaluating their success in supporting policy, it is first
necessary to understand the extant doctrine of estimates--
that amalgam of directives, pronouncements, and history
which describes what estimates are supposed to be and do,
and for whom they are supposed to be written. This chapter
begins with that task. It follows with a discussion of what
consumers say about their use of estimates and the value they
place on them in order to see what differences exist between
doctrine and fact.
The Traditional Doctrine
Part of the traditional perception of the role of
estimates resides in the word itself. As distinct from
reports and studies, estimates were to be forward-looking
rather than historical and descriptive. Their ultimate
purpose was to make judgments about the shape of the future
on matters of concern to U.S. policy--that is, to estimate.
National estimates were intended to have a position
of primacy among the various intelligence products prepared
for decision makers. Estimates were to be the best and most
authoritative appraisals available on subjects of national
23
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
security concern. As such, they were to be national products
in three senses: their subject matter was to be suitable for
assisting in the formulation of national security policy;
their audience was to be at the top levels of government--
including the President, members of the NSC, and similar
officials; and the process of producing them was to engage
the entire intelligence community, so that its combined knowl-
edge could be brought to bear. Nonetheless, the estimate was
to remain the product of the DCI, who was charged by statute
with the production of national intelligence. Other partici-
pants in the process could disagree with the DCI, could
attempt to persuade him to change his mind, and could express
their dissents in the estimate itself, but they could not
impose their views on him, or put him in the position of
being a dissenter from the majority view.
The process was intended to achieve two other results.
The first was quality sufficiently high to engage the most
senior policy makers and to illumine their concerns. The
second was objectivity and disinterestedness. This was to
be achieved, in the first instance, by involving the entire
community, rather than any single agency, in the production
of an estimate. Beyond this, as the DCI's estimate, it was
separated from the purview of intelligence agencies whose
parent organizations had policy responsibilities, and assigned
to an individual who did not, and whose responsibilities were
to the NSC and to the President.
Approved For Release 2002/~Y1f4IDc'T P80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
How Estimates Fared
Against these standards, estimates did not fare well in
the view of the users interviewed. Estimates were often found
wanting in quality and in relevance to the concerns of the
policy maker, and they did not play the role that had been
envisaged for them in the formulation of policy. Although
some individuals were high in their praise, and others found
estimates useful in one or another respect or in particular
situations, most users had critical remarks to make. In the
aggregate, the quality and utility of estimates were viewed
as falling well short of the traditional ideal.
The Negative Side. The first thing to be said is that
most estimates do not reach their intended audience. If they
are called to the attention of the President and NSC members,
it is usually through memoranda written by subordinates who
paraphrase the contents and add comments of their own. Some-
times marked or annotated copies are sent along. Below this
level, estimates are widely received and frequently read.
Even here, however, their content is often absorbed indirectly,
through briefings, conversations, and memoranda.
As to the quality and utility of estimates, perhaps the
most fundamental complaint was that they were not tied into
the ongoing concerns of the policy maker. They were frequently
criticized as irrelevant, and for showing unawareness of the
specific policy problems facing the reader. As one respondent
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
put it, the producers of estimates doubtless wanted to make
them relevant, but apparently did not know how to go about
it. Another complained that on the big foreign policy issues,
estimates very often overlooked some of his problems and did
not hit on his operational needs.
Such critics viewed estimates as documents which supplied
general background and little more. For the most part, they
placed little value on such background and could often not
spare the time to read estimates. As one user put it, a lot
of estimates wound up in his Saturday reading file, and were
never read. Another allowed that estimates did perform an
educational function for staff people, but he also said that
they were never a guide to action.
Estimates were frequently criticized for being insuf-
ficiently analytical and overly descriptive. Many thought
them lacking in boldness, and as more concerned with the
comfortable present than with the unknown future. There
were complaints about the "fuzziness" of their judgments
about the future. One respondent said that many of the
papers he got were "conservative"; the analysis was not
taken very far, implications were not drawn, and there was
a fear of making predictions.
Skepticism was also expressed about the quality and
usefulness of the predictions that were made. Many readers
were simply unwilling to accept a "faceless" bureaucratic
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
product whose authors and their capabilities were unknown to
them. They stressed the importance of the personal element
in receiving and accepting advice, and of knowing something
about the cast of mind and the thought processes of those
who proffered it. They felt that a written product would
inevitably fare poorly in competition with aides and con-
fidantes, unless it was skillfully laid out and of exceptional
quality.
Estimates were also faulted for insufficiently explaining
their judgments and conclusions, and for not discussing what
alternatives were considered and why they were rejected.
Consequently, readers often felt that they were being asked
to accept conclusions from "on high." Unpersuaded by analysis
and argumentation, and lacking personal knowledge of the
authors, many were unwilling to do so.
Skepticism was fed by a widespread belief that estimative
products contributed little that was unique in making judg-
ments about the future. Most policy makers--even the very
senior--read considerable quantities of raw traffic, and
consider themselves well informed. Many felt that they were
in as good a position as the intelligence specialist to make
estimates, and frequently in a better one, because of their
greater first-hand knowledge of particular countries and
leaders. This belief was far stronger on political subjects
than on subjects of a scientific, economic or military nature.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
Past errors also played a role. Some cited specific
cases in which they felt that more accurate judgments could
have been made, even given the hazards of estimating and the
benefit of hindsight. Such remarks require careful inter-
pretation, however. They concerned mainly the general impact
of estimates (e.g., they were too bland or too reassuring),
or inadequacies in the portrayal of broad forces, trends, and
implications. There was a tolerance of failures to make
accurate predictions of specific events, such as a change
of government or the outcome of an election. Our respondents
recognized that such predictions are difficult, and were
disposed to forgive failures, providing the basic analysis
had been sound, and pertinent factors had been considered
and illuminated. Indeed, a sizeable number of our respondents
felt that specific predictions of specific events should not
even be attempted.
The Positive Side. Some respondents took a strongly
positive view of estimates and the estimative process, and
advanced few or none of the criticisms set forth above. In
effect, they felt that estimates met most if not all of the
desiderata of the traditional doctrine. On the whole, they
were less concerned than others about direct relevance to
immediate policy issues, less operationally oriented, more
interested in the "big picture," and more receptive to
material which educated or provided background. They placed
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
greater value, as one put it, on getting a "tapestry," and
on the usefulness of an informed and disinterested view from
outside the policy process.
Two of the three former cabinet members we interviewed
fell into this category. One described the role of estimates
in supporting policy as fundamental, and valued them for their
objectivity and professionalism. The other said that estimates
helped the policy maker to go through a mental checklist of
what he should consider before making an important decision.
He believed that they gave the policy officer a lot of infor-
mation that he could tuck away and summon when he needed it
most, and when he had the least time to acquire it.
The rest of those with a positive view fell into a dif-
ferent category. Not as sweeping or categorical in their
praise as the group just discussed, neither were they
troubled by the kinds of deficiencies perceived by the more
critical. In a sense, they had more modest expectations for
estimates, and these expectations were on the whole realized.
For example, one senior military officer told us that esti-
mates do pretty well in covering the subjects he is inter-
ested in and said that he is not too unhappy with the way
things are, perhaps because he expects less than others.
Another respondent said that he found estimates useful on
specific occasions and in certain contexts, and the work
objective and thorough.
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
Many respondents found estimates useful because they
pulled together in one place all that was known about a sub-
ject. This was meant in one or both of two senses. First,
the estimate was perceived as a product which took into
account all the available sources on a particular subject.
Second, it was valued as the work of a body of experts from
outside the policy process, who had given considerable thought
to the subject. These qualities were found useful by many who
faulted estimates on one or more of the grounds mentioned above.
In the same vein, estimates served for some as a kind of
security blanket. That is, by providing a systematic and com-
prehensive treatment of facts, trends, and issues, they helped
assure the reader that he had considered all the factors
bearing on a problem. If an estimate confirmed his own
analysis and conclusions, so much the better. It could
then be used, if need be, to convince a skeptical superior
or a doubting colleague. Indeed, some estimates were requested
for the very purpose of persuading colleagues and superiors of
the rightness of the requestor's views, and were used for that
purpose.
Other Findings. During our interviews, we solicited
views on the objectivity of estimates, and the presence or
absence of bias. This is a matter of considerable importance,
particularly in view of the virtues traditionally ascribed to
the product and the process.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
On this score, estimates fared well. Indeed, a few
respondents considered the objectivity of national estimates,
and the fact that they were prepared outside the policy process,
as among their principal virtues. Included, as noted, was a
former cabinet member. Numerous others praised the balance
and professionalism of the product. Some who did so had been
critical of estimates on other grounds. For example, one
individual observed that most of what he read was balanced
and professional, even though these qualities led to what he
called blandness and to a lack of focus on the issues which
count.
Some of our respondents did say that they had observed
bias in estimates, either currently or in the past. With two
exceptions, however, they viewed this as a comparatively minor
problem, rather easily discerned and adjusted to, and as the
consequence of human imperfection rather than of deliberate
intent. As one person put it, the record varies, everyone is
a prisoner of his own stereotypes, but the intelligence com-
munity makes a reasonable effort. Another said that it was
not possible to get totally away from subjectivity or from
vested interests; he said that he did not currently see hidden
policy biases, although he felt that vested interests had shown
through in the past.
In only two cases did an alleged lack of objectivity bulk
large in the comments of our respondents. In one case, a former
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
senior official felt that estimates frequently and with intent
superimposed viewpoints not derived from the facts, or allowed
facts to be overruled by the conventional wisdom. In the other,
our respondent in effect imputed bias to estimates because their
conclusions did not agree with his, and, as far as we could tell,
for that reason alone.
We attempted to get a sense of whether users of estimates
perceived any differences in the quality and utility of esti-
mates as the result of the change in the method of producing
them made in 1973. This proved difficult, partly because of
turnover at the policy making level and the unfamiliarity of
many with the process and the product before 1973. There was,
however, a fair degree of consensus on two points. The first
was that the NIO system had produced modest improvement in
the relevance of estimates and somewhat greater improvement
in the flexibility and responsiveness of the system. Con-
versely, estimates were thought to be more uneven in quality
than before 1973, a result which some attributed to the new
procedures for drafting them.
The interviews revealed what appeared to be systematic
differences in reactions to different kinds of estimates.
The so-called "hard" estimates--those on military, scientific
and technical, and economic subjects--tended to get a better
reception than those on politics or foreign policy. This was
apparently not a function of differences in intrinsic quality,
NFI
Approved For Release 2002/11/0
4'CIA-R~P80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
but rather of differences in the policy maker's familiarity
with the complex and difficult subject matter, his ability to
do his own analysis, and to reach his own conclusions. Also,
many judgments in military estimates are computational or
represent the extrapolation of comparatively hard information.
By comparison, judgments on political matters, or on intentions,
seem fuzzier, less precise, and less supportable, and often are.
Another finding is advanced with caution. On the whole,
there appeared to be more satisfaction with estimates at the
country director or equivalent level than at higher levels.
Caution is necessary because our sample at the country director
level was small. We mention this matter only because the impli-
cations are considerable, and because it may warrant attention
at some future time. Perhaps higher level policy makers need
a product so sensitive to their particular concerns that it is
virtually impossible for a producer many steps removed to
satisfy them effectively, without new arrangements for exchanges
on needs and capabilities. The country expert and the producer,
by contrast, may more nearly share perspectives, as well as
views on the evidence and the conclusions that can be drawn
from it.
We failed to find among consumers certain concerns that
criticisms of estimates in recent years had led us to expect.
For example, consumers did not consider that they were getting
confused signals as the result of the proliferation of intelligence
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
publications that cross their desks, estimates among them.
The existence of departmental as well as national estimates
on some matters also did not seem to present a problem. In
general, users seemed to appreciate getting a variety of view-
points, so long as they were analytical and interpretive and
not just a duplicate catalogue of facts or narrative description.
Similarly, there did not appear to be much concern with
the present style of estimative writing, or with the lack of
some explicit scale of probabilities. The use of traditional
expressions such as "probably," "likely" and so forth seemed
satisfactory, though they admittedly do not convey the same
meaning to all readers. The experimental efforts to provide
explicit quantitative rating scales for probability apparently
have not struck any very responsive chord with users. They
were never mentioned as examples to be emulated.
Why the Gap?
Why was there such a disparity between what estimates are
supposed to be and do and what was said about them by those
interviewed? The answer must begin with a mea culpa. Much
of the problem lies with deficiencies in the product, and in
the failure to provide the users of estimates with the kinds
of product they believe they need.
Much of what was said above speaks for itself. It seems
clear that estimates have frequently not understood the policy
question, that they have been insufficiently venturesome, and
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
that they have not done enough about discussing the implications
of their analysis and conclusions for the U.S. and for U.S.
policy. Estimating is a difficult and hazardous art, and the
temptation to be descriptive and current, rather than analytical
and forward looking, has not always been resisted.
The answer does not end with a mea culpa, however. There
are a number of other circumstances which account for the
failure of estimates to achieve the eminent position contem-
plated for them. Individually and collectively, these are of
considerable importance; what is more, they explain in part why
the quality of estimates has frequently left something to be
desired.
Some Unrealized Assumptions. A basic difficulty is that
the doctrine for estimates represents an ideal, which puts an
unrealistic burden on the DCI and the intelligence community.
Specifically, the doctrine rests on some unstated and unrealized
assumptions, which have borne little resemblance to reality,
and which should not have been expected to. These are:
1) that intelligence in general, and estimates in particular,
would have a major influence on the formulation of national
security policy, if not the major influence; 2) that policy
makers would seek and welcome the contributions of estimative
intelligence even when they cast doubts on current policy; and
3) that, consequently, the relationship between national policy
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
and national intelligence would be close and intimate, and
communication on capabilities and needs free and complete.
As to the first of these assumptions, national estimates
have played only a modest role in the formulation of policy.
Almost certainly, this was an inevitable development. National
security policy is not directly driven by facts, analyses, and
judgments, and knowledge does not lead to action in any simple
way. What also come into play are competing bureaucratic
pressures, a wide range of domestic considerations, the con-
cepts, perceptions and stereotypes held by the policy makers
themselves, and a host of essentially political inputs.
Policy making is a complex and often lengthy and untidy
process. Many policy decisions--perhaps most of them--are
not made at a particular time or place, as the result of a
particular act, or as the product of a formalized, established
process. Policy may emerge incrementally, or as the result of
a series of specific decisions, no one of which appeared por-
tentous at the time it was made. It may be generated by efforts
to prepare a speech or a statement to the press. Under these
circumstances, there are some real and difficult questions
about how estimates are to impact on the policy process, and
at what point.
Intellectually, the producers of national estimates have
long recognized the limits of their role, as these limits have
been borne upon them by experience. Yet, the older notion
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
has been hard to shake, so firmly has it become part of the
mythology.
As to the second assumption, it was overly optimistic to
assume that policy makers would represent a ready and even
eager market for estimates. As noted earlier, policy makers
read raw intelligence traffic, consider themselves well informed,
and often believe that they can do as well at estimating as the
intelligence specialist, if not better. Moreover, because of
the past accomplishments of intelligence itself, policy makers
have access to a corpus of fact, analysis and judgment on
national security issues from a national vantage point--a
resource taken for granted now, but not available in earlier
years. Further, they have access to a continuing stream of
other analytic and estimative material, including departmental
estimative products. Many users attach no special value to a
coordinated, national product as compared with a departmental
issuance, and some prefer the latter.
Additionally, if national estimates do their job, they
may wear out their welcome. Thus, one of the functions of
estimates is to help policy makers think about their problems
and to address, and if necessary to challenge, their assumptions
and judgments. In the process, an estimate may say things the
policy maker does not wish to hear and which make life more
difficult for him. On divisive issues, or where the political
or budgetary stakes are high, producers of estimates can be
37
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
cast in the role of interlopers and troublemakers, and they
run the risk of being considered not on the team. Much depends,
of course, on the attitudes existing on both sides, and on the
presence or absence of mutual confidence and trust.
Finally, the idealized relationship involving free and
frank interchange between producer and user does not exist,
and events of the past few years made it less of a reality
than it used to be. There is, moreover, a conspicuous absence
of structure for systematically insuring that estimates are a
part of the policy process, and that they can make whatever
contributions their own virtues allow. The reference here is
only in part to a formal bureaucratic mechanism; more funda-
mentally, it is to a concept of how estimative intelligence
and policy are to complement one another, and how information
on capabilities and needs is to be exchanged.
Other Reasons. The failure of estimates to play their
prescribed role is in part the result of two other circum-
stances. First, the foreign policy establishment is, on the
whole, highly operational. This means that it is concerned
with the policy matters immediately before it and that these
are often short term and highly specific in nature. Frequently,
users judge an estimate by whether it meets such operational
needs; if it does not, it is deemed irrelevant, and it remains
unread, or is read and ignored. For example, one user complained
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
CONFIDENTIAL
that a paper on a country of interest to him had not established
the relationship of what was said to certain issues then under
negotiation with that country.
One solution, of course, is for the intelligence estimator
to do the fast footwork required to inform himself of what is
needed and to supply it. Under the NIO system (described in
Chapter IV), attempts have been made, with some success, to
get a better feel for the requirements of users and to devise
more flexible and responsive estimative formats. The Inter-
agency Intelligence Memorandum (IIM) is one such form. Typically,
the IIM addresses matters of current concern to the contemplated
users and it can be more quickly and easily coordinated than an
NIE, because it does not require a formal sign-off by the
various intelligence chiefs.
Although operational relevance is important, there is another
aspect to the matter. Estimates have been visualized by those
who produce them not merely as products which supported day-to-
day operations, but as contributions to the formulation of
basic, long term national security policy. They were to assist
in developing concepts and in establishing broad, general direc-
tions for policy. Accordingly, it was at one time thought that
a major consumer of national estimates would be the Policy
Planning Staff of the Department of State, performing in the
role that was originally contemplated for it, and not in the
operational role it has often subsequently assumed. From
this perspective, it is not enough to say that estimates are
39
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
frequently deemed irrelevant to policy requirements. Perhaps
the difficulty is with the way policy has been formulated, and
with the atrophy of the functions and institutions that esti-
mates were supposed to serve.
The other circumstance is that the reception given esti-
mates is very much affected by the attitudes which develop
among those who receive and use them. These attitudes tend to
be set at the top. Thus, if intelligence is welcomed and read
at the top, and taken seriously there, it will be read and
taken seriously throughout the government. The reverse is
also true. It became clear, beginning in 1969, and perhaps
earlier, that intelligence in general and estimates in par-
ticular were not held in high esteem at the top, and in a
degree quite out of proportion to what seemed justifiable.
There was deep suspicion and distrust, not only of the esti-
mates, but of those who produced them. This attitude was
bound to permeate much of the executive branch, and it did,
and the production and use of estimates has yet to recover
from its effect.
The Market for Estimates
The comments of our respondents on the quality and value
of estimates laid the basis for another category of questions.
These concerned the nature of the market for estimates. Was
there a continuing requirement for national estimates? If
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
so, for what kind? Was their main value in their analysis, in
their judgments about the unknown future, or in something else?
Where was the market? Should estimates be written for policy
makers at the very top, or for some other audience?
On the broad question, the users' view was that national
estimates do have a role to play in the policy-making process.
This was true even of those who had been most critical of the
past product; their problem was not with the legitimacy or
usefulness of the estimative function, but with the way they
considered it to have been carried out. Our respondents
believed, to be sure, that policy makers would rely heavily
on their own judgment, and on the judgment of trusted aides.
But they also saw a place for estimative products which do
certain categories of things.
Before getting into what these things are, a brief digres-
sion is in order. As some respondents suggested, the require-
ment for estimates is probably greatest when a new Administration
takes office and for a period of time thereafter. A new group
of top policy makers needs to inform itself, and to familiarize
itself with the issues. Old concepts and policies must be re-
examined and new ones formulated and tested. Once the new team
has absorbed the larger picture, and established the main lines
of policy, the requirement lessens. At the extreme, as one
respondent put it, estimates are not missed when things are
set and there are no changes in direction.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
What Should Estimates Do? As to what estimates should
strive to do, the responses depended, to a certain extent,
on the kind of estimate involved. The role of estimates in
sorting out the facts, for example, was more highly valued
on military or technical subjects than on matters of internal
politics or foreign policy. This reflected in part the greater
detail and complexity of the issues, and in part the lesser
confidence of users in their own ability to do the job.
The responses covered a fairly wide range, but they again
tended to cluster around recurring themes. One was that esti-
mates should help to identify policy problems, and to pose
issues and questions about them. "Identify" is the key word.
That is, estimates were deemed useful if they surfaced prob-
lems which had not occurred to the reader, or were not cur-
rently in the forefront of his attention, and if they helped
him to understand what their policy impact might be. An
example would be a paper which foresaw basic changes in the
Sino-Soviet relationship, with a consequent impact on U.S.
relations with both countries.
Estimates were also valued when they could sort out the
facts and issues in a highly complicated situation. Here the
key word is "complicated." What our respondents meant was
that in some cases, the facts are so elusive, so apparently
contradictory, or so fast changing, as to be unintelligible
to anyone whose principal exposure to them is from reading
42
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
raw traffic or perusing current intelligence publications. In
such circumstances, they saw a role for estimative intelligence
in sorting out the facts, the issues they raised, and the pos-
sible consequences for policy. An example would be a paper
written in the early stages of the civil war in Angola, which
described the strength and composition of the opposing forces
and the state of play in the political arena in that little-
known country, before tackling the question of probable future
developments and their impact on the U.S.
Another perceived function was that of identifying and
evaluating the forces at work in a situation. In essence,
this is an. aspect of analysis. The interest was in what was
driving events--in the causal forces whose continuation or
manipulation would affect the final outcome, and in the inter-
play of those forces. An example would be an examination of
the internal political and economic forces which accounted
for the position taken by various foreign countries in inter-
national monetary negotiations, and an appraisal of how these
forces could be manipulated or circumvented to serve U.S. ends.
Closely related, there was also a desire for analysis
of trends and their probable outcome. That is, there was a
requirement for estimating. The focus was not, however,
on specific estimates of specific future happenings. It was
rather on how events could be expected to develop as the
forces at work played themselves out, and how changes in the
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
forces at work would affect the ultimate outcome. An example
would be an examination of probable future developments on
the Italian political scene and their impact on Italian policy
toward NATO, the Common Market and other institutions of
interest. to the U.S.
Still another perceived function was that of judging the
consequences of ongoing developments for U.S. policy. This
meant, in effect, supplying the "so what" of that which had
come before. It was perceived as the step which converted
analysis into a coherent statement of what the interplay of
forces and their probable development meant for U.S. interests
and U.S. policy. It included the identification of forces and
trends which could undermine the premises on which U.S. policy
was based. There is no need to cite examples; it would be a
rare estimative product which did not contain this dimension,
as the above examples suggest.
Finally, estimates were valued when they were able to
judge the reactions to U.S. policies or actions, ongoing or
contemplated. Here, U.S. policy was taken as given, and the
estimative input was not to its formulation, but to a judg-
ment of its effects. In part, our respondents had in mind
so-called contingency estimates, which attempt to describe
what would happen if the U.S. were to do this or that. An
example would be a paper which examined the reactions abroad
to a contemplated arms control proposal, and which assessed
its prospects.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
CONFIDENTIAL
Apart from these more or less recurring themes, certain
users saw other values in estimates. Worth mentioning here is
the interest expressed by some in getting long term analyses
which they did not have time to do for themselves. Almost
invariably, they had in mind a division of labor, rather than
the performance of a function which they thought estimative
intelligence particularly well equipped to carry out. Such a
requirement has interesting implications, for it views estimates
coming from outside the policy process as a mere convenience to
a policy maker who is too busy, and not as publications with
a special contribution to make.
There were several reasons for the relative lack of
interest in specific predictions. Some thought that estimates
placed too much emphasis on prediction at the expense of
analysis and the development of conceptual frameworks.
Others simply placed little value on the prediction of events
such as coups, elections or changes of government. Still others
valued predictions not for their own sake, but because they
forced estimates to move away from the factual and the descrip-
tive and to focus on analysis and the general shape of the
future. Overall, there appeared to be considerable skepticism
about the capacity of the intelligence community to foresee
specific future events, going both to the inherent difficulties
and to doubts that it possessed any special qualifications.
Taken as a whole, what is most striking about the com-
ments received is their reaffirmation of the traditional
45
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
function of the national estimate. Emphasized were the same
desiderata set forth in the doctrine: a focus on the analysis
of forces, trends, and implications for the U.S., in a con-
text that is analytical and forward looking rather than
descriptive and current.
For Whom Should Estimates Be Written? The nature of the
audience for estimates is a matter of considerable importance.
It affects the way they are organized, their level of detail,
and how they go about presenting facts, analyses and conclusions.
Some of those we interviewed thought that estimates
should be prepared for readers at the very top--the President,
members of the NSC, and others at the cabinet level. This was
usually not out of confidence that estimates would regularly
be read there, but in the belief that estimates aimed at such
customers would catch others along the way. There is some-
thing to this view, but it misses an important point--that
the perspectives and needs of those at the top are not neces-
sarily the perspectives and needs of those who support them.
It is almost certainly too much to expect that busy
policy makers at the highest levels will read estimates
regularly. Nonetheless, they will be customers under some
circumstances. Estimates on particular subjects will from
time to time commend themselves, such as those on Soviet
strategic objectives and military forces or those making a
significant contribution to the conduct of important
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
CONFIDENTIAL
negotiations. So will products of exceptional quality and
insight. Particular individuals will value estimates for a
variety of reasons--because they come from outside the policy-
making establishment, because they inform and illumine on
matters of concern, or because they contain agreeable con-
clusions useful in the policy debate.
The estimates most likely to be read at the highest
levels, however, are those which the DCI believes should be
read there, and which he urges upon senior policy makers.
It is one thing for an estimate to be issued in the name of
the DCI. It is quite another for the DCI, as a trusted senior
aide of the President, to commend it, and to press it upon
others. If the DCI has the necessary access and enjoys the
confidence of the President and his chief aides, then any
given estimate has a far greater chance of being read at the
top, and consequently, throughout the government. "Any given
estimate" is the way to put it, because the use of the DCI's
position, prestige and access, if it is to be useful and
effective, cannot become routine. Nor can it be fully
effective unless the DCI involves himself in the process
of producing estimates, so that he can sell what is truly
his, and help insure that estimates are relevant and of high
quality.
The reception afforded estimates at the top will also be
affected by the attitudes toward them that exist or develop
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
there. We have already suggested that over the past few years,
there was a disposition to discount the importance of the
estimative function, and a deep skepticism about the value
of the product. In this respect, too, the DCI plays a crucial
role. His personal qualities, his access, and the confidence
that the President has in him can help establish a receptivity
for the product, which its own qualities would then have to
sustain.
Most of our respondents felt that estimates should be
aimed at the Assistant Secretary of State, White House Staff,
or comparable level, and we agree. These are the officials
who set the terms for the debate on matters of national
security policy by formulating the options and alternatives.
They are the trusted aides, referred to earlier, who enjoy
the confidence of policy makers at the highest levels, and
who are turned to for advice. On many problems, they are
the highest level which combines expertise with the power to
act. Readers such as these are able and knowledgeable. With
access to the same material as is available to the intelli-
gence community, they feel able to do their own analysis,
and to reach their own conclusions about the shape of the
future. They will not accept unsupported judgments, in
the form of "we believes" or otherwise.
How Should Estimates Be Written? If estimates are to
appeal to this audience and be useful to it, they must be
CONFIDENTI L
Approved For Release 2002/11/04: CIA-IDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
prepared with these considerati.ons in mind. As we gleaned it
from our interviews, they must:
--emphasize analysis rather than description,
and carry the analysis to the edge of the
unknown;
--show the relationship among data, analysis
and conclusions and the thought processes
involved, so that the reader can make his
own journey to a different destination if
he believes it necessary; this also en-
hances the credibility of the product and
helps to compensate for its "facelessness";
--explain clearly what issues were most con-
tentious at the coordinating table, and why;
--discuss what was discarded and why; this
not only helps build confidence in what
was retained, but may persuade the reader
that his views, if different, might better
be discarded;
--state any differences of opinion, and on
what they are grounded, and avoid the
temptation to consider consensus desirable
per se;
---state the implications of the analysis and
conclusions for the U.S. and for U.S. policy;
and
--describe the continuity or change in the
current estimate as compared with its
predecessors, particularly the change,
and set forth what earlier material is
now considered wrong, and why.
One difficulty with this list is that it makes for longer
papers, and longer papers tend not to be read. We mentioned
this dilemma to a number of those we interviewed, but beyond
agreeing that it existed, they had little to offer. It is
also true, however, that papers which slight the above
49
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
desiderata will probably not be read either. Thus, the pro-
ducers of estimates will have to find ways of being complete
and persuasive, yet readable and spare. To the extent that
length cannot be avoided, quality will compensate for it,
among the readers we are now talking about.
At higher levels, the problem is more serious. To pro-
vide for the occasions when estimates will be read there, they
must contain appropriate syntheses and summaries, with invita-
tions and directions for dipping into detail if such is desired.
Summaries are also useful to readers at all levels who for one
reason or another do not desire detail in particular cases.
The importance of an effective summary cannot be overstated;
it was a point made to us, without any prompting, by some of
the most senior policy makers with whom we talked. Of this,
more is said in the following chapter.
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
III
IMPORTANT ELEMENTS IN THE PRODUCTION PROCESS
FOR NATIONAL ESTIMATES
The preceding chapter discussed the views held by con-
sumers on the quality and utility of estimates. Some consumers,
as well as producers, also had views about specific aspects of
the estimative process. In this chapter, those views are dis-
cussed in detail, along with perspectives developed by the
study team.
Self-Initiated Estimates
Roughly half of all the national estimates presently pro-
duced are initiated by the intelligence community itself, and
most producers and users with whom we talked seemed satisfied
with this fact. A very few felt strongly that work on an
estimate should be initiated only at the specific request of
a consumer, primarily to avoid the production of irrelevant
or unneeded papers. The vast majority, however, believed
that intelligence producers have the duty of initiating an
estimate when they perceive the development of a significant
problem for U.S. policy. We agree, and would add that this
appears particularly important at times when the policy making
apparatus is preoccupied with some special problem, such as a
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
major and prolonged international crisis, or at the beginning
of a new Administration.
Two important corollaries should be kept in mind. The
first is that the most useful estimates seem to have been
produced in response to specific requests by policy makers
who really wanted and needed them. Cranking out a lot of
self-generated estimates does risk overloading the circuit,
thereby diluting user receptivity, and wastes time and money.
Also, a self-initiated product typically has a harder time
being accepted by busy consumers psychologically unreceptive
to thinking about issues ahead of their time, particularly
if the product arrives unanticipated. The second corollary,
therefore, is that any estimate generated by the intelligence
community stands a better chance of being read and used if
it is discussed fully from the outset with its intended
recipients. Prior to doing this, the producers should focus
carefully on their proposed effort to determine whether it
will. be perceived by users as related to ongoing or potential
U.S. policy concerns. If the proposed product appears to
have little relationship to such concerns, it is not likely
to be worth the effort.
Terms of Reference
On any estimate, there ought to be clear agreement between
producer and user on the major questions to be addressed. The
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
device for securing this agreement is the "terms of reference"
for the estimate.
The production of an estimate generally begins with the
elaboration by the NIO or an interagency group of the topics
to be explored and the scope of the estimate. Currently,
producers take part in the process, and the policy customer
may also, either directly or through his agency's intelli-
gence organization. To the degree that producer and user can
initially agree on the questions to be answered, and can
throughout the production process jointly further refine the
scope of the paper, the resulting estimate can be made more
relevant to the specific needs and interests of the main
recipients.
In general, the more direct the communication between
producers and users on the scope of a paper, the better. A
formalized, regular procedure providing for ample initial and
subsequent consultation between estimators and policy making
customers should be instituted. This process should not, how-
ever, be allowed to become so formal and bureaucratic as to
greatly lengthen the time that it takes to do an estimate,
for such delays would themselves detract from the utility of
the product.
Encouraging producers and users to discuss the scope of
the paper is not to ignore the danger that either might
attempt, consciously or unconsciously, to predetermine or
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
alter the outcome of the estimate by posing the issues in
terms of "loaded" questions. This appears to be a manageable
risk, and is likely to be outweighed by the benefits of a
close and. continuing exchange. The best insurance against
bias is probably a careful and continuing review of the
process by managers and participants. We found little, if
any, concern among producers or users with the possible
slanting of estimates as a result of a "cooking" process
involving the way the issues are posed.
The need for an estimate having been established and its
scope worked out, the producer is next faced with a number of
decisions on procedural aspects of the process. These involve
such specific questions as how to handle the multidisciplinary
aspects of the paper, if any; the use of special methodologies,
especially those of a quantitative nature; and whether pro-
vision should be made for the presentation of competing views
on key issues, all matters on which critics have faulted the
producers of estimates. The succeeding pages discuss these
matters, along with other more general issues asspciated with
the production process, including the use of net assessments,
coordination, and the handling of dissent.
Multidisciplinary Work
There was general agreement among consumers of estimates
that the intelligence community has done far too few cross-
disciplinary or analytically-integrated studies in recent
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
years. None of the dozen or so consumers with whom this
specific subject was raised believed that estimates are now
adequately interdisciplinary. Several thought that estimates
were particularly poor in blending economic with political
factors. Two felt that the intelligence community as a whole
had done very little interdisciplinary work. One commented
that NIE 35/36-1-76, The Middle East Military Balance, 1976-
1981, represented a "good try" in the multidisciplinary
realm. This estimate attempted to interweave political,
economic, and military factors into an appreciation of the
relative strengths of various countries in the mid-East.
Producers were less critical. Some maintained that esti-
mates have contained interdisciplinary analysis all along,
that consumers can be assured that estimates take account of
all the main factors in the situation, even if some were left
unstated. Several veteran intelligence officers believed
that the ONE staff and Board had, in its heyday, constituted
the multidisciplinary group par excellence. Other producers
complained that the present product usually simply splices
together separate sections of political, economic, or military
analysis, leaving the reader to do his own synthesis of the
relationships. It is the failure to produce this synthesis
that spurs the criticism from the consumers.
Almost everyone critical of the present interdisciplinary
effort agreed that, while the community must do better, the
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
task will not be easy. There is, for one thing, little agree-
ment yet on how best to accomplish good multidisciplinary
synthesis or integration. Moreover, differing bureaucratic
priorities and increasing specialization among analysts have,
until recently at least, presented real problems in useful
interdisciplinary work. For another, the notion that multi-
disciplinary analysis is "a good thing" has become perhaps
overly fashionable in the intelligence community and outside
it, and climbing recklessly on the bandwagon could easily
lead to a drop in quality. This would be unfortunate,
because the principle is sound and the need to apply it real.
To produce a good interdisciplinary product, analysts
and experts working on it must be organized in a way calcu-
lated to bring their expertise to bear in a multidisciplinary
manner. This probably necessitates placing the analysts in
an environment with an effective task or project leader where
meaty, wide-ranging discussion can take place prior to actual
drafting so as to focus the drafting on the key questions from
an interdisciplinary point of view at the outset.
The discussion should be aimed at identifying the key
issues and the cross-relationships among them, i.e., it should
be a "synthesizing discussion." The objective is to produce
4
New organizational arrangements within the Directorate of
Intelligence of CIA are intended in part to meet these problems.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
in the drafting itself a narrative designed around the illumina-
tion and analysis of the key issues, one that interweaves,
according to their significance and interrelationships, political,
economic, sociological, military and scientific factors bearing
on the issue. It is this subtle, hard to produce synthesis
that is the essence of a good interdisciplinary effort.
New Analytical Methodologies
Another question is whether additional analytical leverage
can be gained on estimative problems by reaching beyond tra-
ditional techniques and utilizing quantitative, mathematical
and systems-oriented methodologies.
The alleged failure by producers to use new methodological
tools of analysis has been cited by some critics as a serious
flaw in the estimative process. Our interviews reflected
little support for this charge. No consumer we talked with
showed any deep-seated conviction that the intelligence com-
munity was seriously remiss on this score. While virtually
everyone agreed that the community ought to evaluate new
analytical methods as they come along, no one argued that
the product would be improved by their wholesale adoption,
especially in estimative production, where short deadlines
and the need for simplicity and relevance often work against
complicated methodologies and presentations.
We found a good deal of skepticism among consumers and
producers about using computers and other "new" methodological
57
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
tools in preparing political or general estimates. Most
described themselves as "willing to be persuaded," but as
one producer put it, a long time is usually required before
one can be sure that a new analytical technique will have an
intelligence payoff, and this means that inevitably "you chase
up some blind alleys." With limited analytical resources this
can be an expensive proposition.
Other producers pointed out that using new methodologies
entails some risks. A strange new technical vocabulary--a
frequent by-product--may well turn off the reader. Over-
reliance on new techniques in writing estimates might imply
greater precision in judgments than the facts warrant. And
not all new analytical techniques--not even those specifically
designed for political analysis--can be successfully adapted
for estimative use. Many demand more information than analysts
have or can get. Many are quantitative, and not all intelli-
gence issues lend themselves to such treatment.
Nevertheless, keeping up with the state of the art in
analytical methodologies is an important obligation of all
intelligence analysts, and several "new" techniques appear
to promise at least limited applicability in estimating.
25X1
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
two possibilities; there are and will be others. The important
thing is to strive to be open-minded and alert to possibilities
as well as limitations. At this point, the new analytical
methodologies, particularly the more radical and complicated
ones, appear more suitable for conducting the basic analytical
work supporting the production of estimates than for use in
estimates themselves.
CompetingAnalysis
"...the issue of consensus versus competition
in analysis represents a persistent conceptual
dilemma for the intelligence community. Policy-
makers tend to want one 'answer' to an intelligence
question, but at the same time they do not want
anything to be hidden from them ..... Some members
of the intelligence and foreign policy communities
today argue that the consensus approach to intel-
ligence production has improperly come to sub-
stitute for competing centers of analysis which
could deliver more and different interpretations
on the critical questions on which only partial
data is available."
--Final Report, Senate Select
Committee on Intelligence
Activities, 1976, Book 1, p. 272.
The provision of competing centers of analysis is a con-
cept that is getting considerable attention at present. The
objective is to reduce the chances of overlooking or ignoring
some vital aspect of an important analytical or estimative
matter.
In one sense, the principle is no stranger to the intel-
ligence community. Currently, each separate intelligence
component with competence in a particular field finds itself
in competition with others to convince the audience that its
Approved For Release 2q11 I :TTf RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
views are correct. Thus, each agency serves in a very real
sense as a competing center of analysis. If this were not
so, there would be no need for a coordination-dissent pro-
cedure in the national estimative process.
Some critics of the process argue, however, that it is
not enough to have competition over ideas within and among
the intelligence agencies themselves. They charge, in effect,
that the intelligence community could become--or has become--
a sort of closed corporation in which the free interplay of
ideas and theories is no longer possible and important alter-
native approaches are ignored. They allege that problems of
"group think," of reinforcing consensus, and of mind set or
bias often prevent the adequate exploration of analytical
alternatives and the formulation and presentation of alter-
native estimates. Critics cite the inability to provide an
environment wherein analysts can easily depart, if they like,
from the standard view on an issue as a main reason for the
failures of intelligence to do such things as predict the
outbreak of the Arab-Israeli war in 1973.
We encountered only limited support for these assertions
among consumers. Many of those commenting nevertheless made
it clear that they wanted and expected all the informed views
they could get, especially on the more important intelligence
subjects. Some intelligence components do have a virtual
monopoly of expertise on certain economic and scientific or
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
technical subjects. This makes for considerably less competitive
analysis on these subjects than many consumers--or producers--
would like to see.
In the effort to leave no analytical stones unturned, one
relatively inexpensive approach is the use of a so-called
"devil's advocate." The concept provides that someone be made
responsible for building a case opposed to that in a draft
estimate, so that two sets of analyses and conclusions can
be considered and weighed before the estimate is finally
approved. The approach was tried during the production of
NIE 11-3/8-75, Soviet Forces for Intercontinental Conflict
Through the Mid-1980s, and we were told by some familiar with
the exercise that it was very useful. Several of those
interviewed favored using this process--none was opposed--
but said that it should be employed only on the most. impor-
tant estimates. We agree. Clearly, there is a continuing
need for efforts of this sort that force estimators to search
out and seriously consider information and hypotheses that go
against their own theories and views of reality. It would
appear important, however, to confine use of the technique
to important areas of estimates where there is substantial
uncertainty and debate. It should not be used to help con-
struct straw men.
The only experiment involving a large scale effort to
provide an alternative analytic approach was the so-called
61
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
"B-Team" exercise, used during the preparation of NIE 11-3/8-76,
Soviet Forces for Intercontinental Conflict Through the Mid-
1980s. The team was put together as a result of the dissatis-
faction of certain members of the President's Foreign Intelli-
gence Advisory Board with earlier estimates in this same annual
series. Producers to whom we spoke before or during the experi-
ment were generally neutral or slightly pessimistic in their
views about its usefulness; consumers were guarded in their
reactions. The widely publicized charges and counter-charges
about the experiment, and the fact that it has been so recently
completed, make it difficult at this point to evaluate its
strengths and weaknesses or to offer an opinion about its
overall value. Perhaps the only comment that can fairly be
made now is that the idea of turning loose a fully staffed
and funded "competing center of analysis" on an estimative
problem is laudable in purpose, but likely to be expensive
and difficult in execution. The competing teams must have
the same data, the same rigorous analytical procedures, and
the same degree of relentless objectivity if the exercise is
to work at all. The hue and cry following the "B-Team" experi-
ment suggests just how tricky a proposition this is likely to be.
In all this, the problem is to encourage alternative
analysis without artificiality and without a drop in the
quality and coherence of the product, in a way that stimulates
thinking rather than emotion, and within the constraints of
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
available time and resources. Ways must be found for encouraging
the development and presentation of alternative views, and the
time and means to give them a fair hearing and to incorporate
them in the final text as appropriate. We suspect that the
basic answer, as with so many other elements that make for
success in the business of estimating, lies in the skill,
perceptiveness, and intellectual rigor of the estimative
manager, which are probably more important to success than
any organizational techniques that might be developed. This
is not to say, however, that there may not be room for more
formal arrangements, if the issue is important enough and if
other factors seem to warrant such an approach.
Net Assessments
Although the use of net assessments is not frequently an
issue in the production of estimates, it presents philosophi-
cally troublesome problems and deserves discussion.
The net assessment function, and the part that intelli-
gence should play in it, are subjects of continuing confusion
and controversy. Much of the confusion derives from the lack
of a commonly accepted definition of a net assessment. The
controversy is largely over where in the national security
bureaucracy the function can most efficiently be performed,
and with the least risk of having it distorted by policy biases.
In the aspect that raises issues for intelligence, net
assessment means pitting on paper, in carefully comparable
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
form, the strengths and weaknesses of an adversary power against
those of the United States, in order to determine which is the
stronger and in what respects. This is usually, though not
necessarily, in terms of military forces. In the lingo of war
gaming, a closely allied concept, net assessment poses "red"
or enemy forces against "blue" or friendly forces. The rub
is that those who do the assessing must have complete access
to data on U.S. capabilities, present and prospective, as well
as data on the capabilities of the adversary. Compiling the
former, of course, lies well outside the intelligence com-
munity's responsibilities and prerogatives.
Of the producers and users of estimates with whom we
discussed net assessments involving the U.S., none felt that
the intelligence community should conduct them or include
them in national estimates (although one saw the DCI as a
possible producer, in some other context). To do so, they
reasoned, would be to run an inordinate risk of transforming
estimators into advocates or opponents of particular U.S.
policies. They recognized that if net assessments are to
be done, the intelligence community must furnish data and
judgments on the capabilities of the adversary, but felt
strongly that this should be the limit of the community's
involvement. They believed that the overall responsibility
for net assessments involving the U.S. must reside outside
the community proper as it does at present. They did not
64
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
appear to rule out the presentation of comparative data on
the U.S. and other countries in estimative products, or
analyses involving comparisons, so long as the purpose and
result was the elucidation of foreign capabilities, and not
that of making net judgments.
We find these views compelling. Getting involved on the
U.S. side of a net assessment could rob the national estima-
tive apparatus of one of its most important assets: its
detachment from policy advocacy and from domestic disputes
over the relative merits of proposed new weapons systems.
With the U.S. intelligence effort already weakened by develop-
ments in recent years, such involvement, apparent or real,
could be disastrous. In making comparative analyses, which
are quite another thing, estimators must preserve the dif-
ficult but important distinction between that which helps
the reader to better understand the capabilities of foreign
powers, and that which becomes a net assessment in fact if
not in name.
Net assessments can involve comparisons between
foreign countries only. In such cases, it is not only
acceptable for estimates to include them but imperative, if
they are to do their job properly. Several consumers told
us that they would like to see more such net assessments,
and we agree that it would be useful. Their preparation
will probably be a growing assignment for the intelligence
65
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
community in the future, as power becomes more diffuse and
as more blocs of states and regional complexes develop that
have importance for U.S. national security.
Coordination
"...the process of coordinating a paper could
be rewarding. The atmosphere becomes uncongenial
to special pleading and to the urging of a paro-
chial point of view by a particular agency ..... But
in addition to knocking down parochial prejudices,
the process had a more positive aspect. It was
a forum where people from all over town could
exchange views, add to the store of community
knowledge, and refine and sharpen their assess-
ments of the course of events."
--Sherman Kent, The Law and Custom
of the National Intel igence
Estimate, 1976, pp. 93-94.
In the last sentence quoted above, Kent was speaking
only of coordination. He might just as well have been dis-
cussing the value of the whole estimative process from the
point of view of the producers. The estimative process,
including the drafting phase, is one of the few that requires
a group of participants to work their way through an analytic
problem, pulling together into a structured, reasoned whole
what might otherwise remain as disparate strands of thought,
impression, or opinion. In short, it gives the producers a
base from which to compare, agree, and disagree. In the
highly segmented world of most intelligence production today,
with its frequent emphasis on quick reaction, such a result
is important, regardless of the success of the product in
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
predicting the future or its immediate contribution to the
policy process.
That part of the estimative process that brings the experts
on one subject together from separate agencies to scrutinize,
debate, and revise the text of a draft estimate--the so-called
coordination process--is in many ways the most important single
step in the entire process. Often, it is also the most frus-
trating. At its best, the interagency coordination meeting
enriches and adds new substantive dimensions and insights to
the draft. At its worst, it dissolves into interminable
wrangling and stalemate, or even more unhappily, produces
sterile language and flabby judgments in the name of consensus
--and to get the messy business over with. Most intelligence
officers who have participated more than casually in the estima-
tive process will readily recall attending both types of session.
Many users of estimates told us that it mattered little
or not at all whether a paper was coordinated; for them, it
was what the paper said that counted. If this attitude is
prevalent, why bother with coordination at all, given the
complexities of the process and the appreciable chance that
it will have negative as well as positive effects?
There are good reasons for doing so. The principal
reason is that the coordination process goes to the heart
of the concept of national intelligence. It allows the DCI,
as the nation's chief intelligence officer, to express his
views on the issues in a paper which is his and his alone,
Approved For Release 200C29 ' /b~E'IA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
while requiring other participants either to agree with him
or to express their disagreements and the reasons for them.
Perceived deficiencies in the process ought not to obscure
this essential point. What it means, substantively, is that
coordination brings to bear the talents of the entire intel-
ligence community, provides a means of getting a variety of
views into the paper, makes it more likely that the pertinent
issues will be covered, and, at its best, helps to define the
issues and to sharpen them. Additionally, the give and take
of coordination permit the participants to test their data,
their assumptions and their hypotheses. It is thus incon-
ceivable, despite the very real difficulties in producing
coordinated assessments, that any top level decision maker
could settle for less than a thorough interagency estimate
on subjects of national importance.
The pitfalls in the coordination process are never-
theless numerous and important. Coordination can mask the
very real and deeply-felt divisions that sometimes exist
within the intelligence community on controversial issues.
It can fuzz up clearly drawn, analytically sharp conclusions
and projections, and cause an important point to be lost.
Divergent views may be given short shrift by a "reinforcing
consensus," and "submerged in a sea of conventional collec-
tive wisdom." Haggling over major or minor issues can delay
the completion of a paper well past the time when it would
be most useful to its chief recipients.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
Curing these problems is much harder than pointing them
out, but a few general suggestions can be offered. One of the
most important has to do with the official who serves as chair-
man of an interagency coordination meeting. Whether the process
proves product-enhancing or stultifying largely depends on this
individual. To the extent that he or she is tough-minded,
independent, and judicious, and is skilled in running a
meeting and in separating serious substantive differences
from trivial semantic hangups, the process is likely to be a
fruitful one.
It should also be borne in mind that a consensus is often
not what the user of an estimate is looking for or needs. Con-
sumers at all levels frequently told us that reaching a con-
sensus was a part of the estimative process that ought to be
played down. Several took pains to tell us that they were
much more interested in seeing the final paper accurately
reflect the range of views within the intelligence community
on an important point. Some also wanted to get more of a
flavor in reading the paper as to how the coordination process
went: which issues elicited the most arguments among the
interagency representatives, and which theories or lines of
reasoning were advanced at the coordination table but dis-
carded, and why?
Another problem that bothered several interviewees was
the internal difference of views that occasionally surfaced
69
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
on a subject among different components or officials of the
same agency. This causes delays in the coordination process
that would probably be reduced if participating agencies were
to meet internally to sort themselves out on the issues before
each round of interagency meetings took place. CIA, as the
agency with prime responsibility for producing national intel-
ligence has a special obligation to do this, but State, Defense
and other members of the community should do more of it as well.
Dissent
In the estimative process, coordination and dissent are
two sides of the same coin. At their best, they help make an
estimate more thoughtful and more useful to the policy maker
than the product of a single agency. A well drafted dissent
focuses a different view in such a way that the facts, assump-
tions, and arguments employed are comparable with those for
the position being dissented from. More fundamentally, the
dissent mechanism is, as Sherman Kent puts it, "the indis-
pensible corollary to the DCI's primacy" in the estimative
field. If all estimates had to be negotiated to a compromise
totally acceptable to all the participating agencies, the
result would be pap; if the DCI were to insist that all
agencies accept his views, the intelligence community would
quickly disintegrate.
The formal dissent is a device that has been used rather
sparingly on most estimative subjects. One detailed review
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
of 83 estimates produced between 1951 and 1964 on a single
5
geographic region found only six formal dissents in the lot.
That dissents have not been used more frequently may be partly
due to an exaggerated notion that policy makers prefer single
"bottom line" judgments. More recently, it has become clearer
that the presentation of conflicting views on controversial
or complex matters is not merely tolerable to the consumer,
but that he actually prefers it. Several of those interviewed
emphasized this preference.
There is, of course, a world of difference between the
value of a dissent taken to reflect an honest substantive
difference with a predominant view and one taken primarily
to support a bureaucratic position. There are many examples
of both in the estimative record, and this seems bound to
continue. But an increasing sophistication in elaborating
dissents has characterized the process in the 1970s. To the
degree that this is continued, it should be increasingly
possible for the user to recognize readily--as many already
do--which dissents need to be taken seriously and which can
be dismissed.
One practice that should be pursued is the incorporation
of dissents directly in the text instead of in footnotes.
5
The Arabs and the Estimates, ONE Staff Memorandum 58-65,
4 Nov 65, TS.
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
The textual dissent is now frequently used, and most of the
consumers who addressed this subject clearly prefer it. They
feel that it provides a much better balance among competing
ideas, and a better means of comparing them, than is possible
if only one view is carried in the text and dissenting views
are relegated to footnotes.
Some consumers told us that they would like to see all
the elements of.the intelligence community holding a particular
dissenting view clearly identified with the dissent. They
also want to see an elaboration of the rationale behind the
dissent. We agree that such refinements would further improve
the mechanism of dissent and make estimates more complete and
more useful.
Summaries
There was a strong feeling among the consumers inter-
viewed that a good summary is an absolutely essential part
of any estimate. Again and again, consumers emphasized that
a tightly written summation that manages to convey accurately
an estimate's key conclusions in comparatively brief form is
the best possible device for ensuring high-level attention to
the "message" of an estimate. Many added that lengthy esti-
mates did not bother them per se--indeed, some users said
they wanted and needed ample substantiating detail--so long
as a crisp, well-written summary led off.
72
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
A very intensive, deliberate effort to draft such a
summary needs to be untertaken for each estimate produced.
Those responsible for reviewing the final draft of the paper
should ensure that the summary is of the highest quality and
accurately captures the flavor and tone of the estimate it-
self. This is, of course, a tall order, but it is difficult
to overstress its importance if estimates are to be well-
received.
Beyond matters of process affecting the substance of an
estimate, the producers of an estimate have decisions to make
bearing on the classification of the product, the manner in
which it should be presented, and the possible scheduling of
a so-called "post-mortem." The question may also arise as to
whether an updated version of the product should be produced
at periodic intervals. These matters are discussed below.
Classification
Some consumers were strongly of the view that estimates
tend to be overclassified and thereby lose some of their
potential audience and thus some of their utility. Both
working-level users and officials high in the policy-making
apparatus said, in effect: the lower the classification,
the better. As in the past, highly classified material will
have to be treated in estimative products. It seems only
sensible, however, to adopt a general rule that highly classified
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
material, especially backup detail and supporting rationale,
be relegated whenever possible to separate annexes or appen-
dices. In this way, the basic estimate can be held to (or
sanitized to) a lower classification, permitting the essentials
to get across to a wider audience.
We heard one other suggestion that struck us as useful
in this regard: the periodic publication of brief abstracts
of all the estimates written during a given period. If held
to the Confidential or Secret level, such abstracts could
receive wide dissemination, and apprise potential users (many
of whom would have the necessary clearances) about documents
of possible relevance to them. This could be particularly
helpful to users of military/strategic or scientific/technical
estimates, most of which require special handling or physical
storage in remote vaulted areas. A publication of this sort
might also aid in reducing the number of people who auto-
matically get copies of published estimates.
Written and Oral Presentation
"Whatever the format and procedures, important
intelligence should be presented in a way that can
lead to discussion and questioning before decisions
are made, so that the dangers of the policy maker
misunderstanding the judgments (especially those
expressed as probabilities) and the implications
of such intelligence are reduced to a minimum."
--William J. Barnds in the
Report of the Commission on
t eOrrganization of the
Government for the Conduct
of Foreign Policy, June 1975,
Volume 7, p. 32.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Although few would go so far as to agree with Marshall
McLuhan that "the medium is the message," both consumers and
producers alike seem strongly of the view that the presenta-
tional medium chosen and the skill with which it is used are
very important in getting across the message of an estimate.
But, as with alternatives elsewhere in the estimative business,
no one method of presentation pleases or meets the needs of
everyone.
The basic choices now available are essentially two: a
printed paper, or a "live" oral briefing. Each has certain
rather obvious advantages and limitations. Most consumers--
seven of the ten who commented specifically on this point--
clearly preferred to get most or all their estimative material
in written form. Of the remaining three, one felt that esti-
mative conclusions should be offered in both written and oral
form, one felt that oral briefings of estimates should be made
available "more often," and one felt that the presentational
format didn't matter much. All producers and former producers
who commented said that they thought the intelligence community
Additional possibilities, including closed-circuit TV,
video disk, graphic display of text or computer readouts are
becoming increasingly feasible. In general, these seem better
suited for use among specialists at the working level, than
for communicating detailed judgments on complex subjects to
top policy makers.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
ought to explore more vigorously the feasibility of providing
selected estimative conclusions orally to some consumers.
We agree that the oral mode of communicating estimative
material should be investigated, and used more often if there
is sufficient consumer demand for it. Certainly, the pro-
ducer of estimates should make available the services of a
briefer as required. But oral briefings do not seem to offer
any kind of panacea and, except for unusual circumstances,
should supplement rather than replace a written text. Com-
municating complex, sophisticated judgments and the supporting
rationale orally is very impressive when it is done well, but
it is difficult and subject to pitfalls, especially as regards
nuances and qualifications, and it requires special skills that
may not always be available. Even when they are, the intended
audience may be too busy to give the briefer the time or
attention needed to do an effective job. Our view is that
the brunt of any effort to improve presentational formats
ought to go into making written estimates as succinct,
readable, and responsive to different levels of need as
possible. Efforts should also be made to produce, to the
degree that resources permit, several versions of key esti-
mates for consumers with need for different levels of detail.
Over the years, the term "post-mortem" has been used in
the intelligence community to describe two quite different
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
processes. It may mean the formal qualitative review of an
estimate or a series of estimates some time after publication
to judge the accuracy of the predictions and the soundness of
the analysis. It may also mean the review of an estimate soon
after publication to investigate the adequacy of the informa-
tion on which it was based and to identify the need for future
collection or analysis to fill the gaps.
Post-mortems of the latter type were once quite common,
but were later abandoned, because they became highly pro-
forma and routinized, and failed to accomplish their purpose.
The only recent example of which we are aware was conducted
on NIE 11-3/8-75, Soviet Forces for Intercontinental Conflict
Through the Mid.-1980s. It was carried out as an experiment
by the various departmental representatives who had just com-
pleted the estimate in order to identify gaps in intelligence
and areas of special concern that required more attention next
time around. The participants were pleased with the experiment.
We agree with those holding that the device can be help-
ful if used sparingly and on estimates of prime importance.
We think a post-mortem should be undertaken shortly after an
estimate is completed, should involve both producers and con-
sumers, and should be focused on identifying substantive gaps
and problem areas rather than on Monday-morning-quarterbacking
the estimate's judgments and predictions. The role we see
formal post-mortems performing best is that of helping close
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
the circle in the intelligence production process. By pointing
up substantive weaknesses and missing pieces of the puzzle, the
post-mortem could become a valuable tool in helping set realistic
priorities and requirements as guides for intelligence collection
efforts.
One useful variant would not await the completion of an
estimate, but would be carried on concurrently with the drafting
as an integral part of the process. There is much to the propo-
sition that the most efficacious time to identify deficiencies
in information and analysis, and to initiate corrective action,
is at the time the analytical process is concentrated on a
substantive matter during the process of drafting.
Post-mortems on the quality of estimates have been rela-
tively rare, particularly in recent years. Most recent post-
mortems of this kind have been in response to "intelligence
failures." They have encompassed not just estimates, but
the entire range of intelligence coverage on a subject, from
special memoranda to current intelligence reporting, situation
reports and even individual, raw intelligence reports. The
post-mortems on the October 1973 Arab-Israeli war and on the
Cyprus crisis in 1974 are examples. The producers who com-
mented on post-mortems concerning quality favored them as
helpful if they were used only on estimates of major impor-
tance and if they included provision for feedback from
consumers.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
Post-Publication Update
We encountered almost no support for the notion that
estimates ought to be revised and reissued at fixed intervals,
except in the case of the annual Soviet strategic estimate,
NIE 11-3/8, on which there was substantial user demand for a
new version each year. One Department of State country director
did express an interest in an annual estimate on his country,
even if nothing significant had occurred in the interim. But
his country is a "denied area" as far as reporting channels
are concerned, and he was the only user who took this position.
Our own view is that estimates should be updated, as a rule,
only when a significant change in the situation or in the
questions to be addressed requires it.
The problem is that there is at present no systematic
method for determining whether such changes have occurred.
In our view, all estimates should be reviewed periodically
to decide whether an update is required. Such a review would
involve a sort of post-mortem to determine whether the existing
estimate is deficient in scope, analysis or in the pertinence
and accuracy of its judgments. Systematic stock-taking would
not only help insure that particular estimates remain up-to-
date, but might also provide more general insights into past
mistakes which can be minimized or avoided in the future.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
ORGANIZATION FOR THE PRODUCTION OF ESTIMATES
The ultimate quality and utility of an estimate depend
primarily on the personal skills and insights of those who
produce it. But their ability to use these qualities effec-
tively is determined in no small measure by the organizational
environment in which they work. The present chapter focuses
on certain aspects of that environment.
The question of how the machinery for producing estimates
ought to be structured was among the topics most frequently
raised in our interviews. Producers, not surprisingly, had
stronger views than consumers. While we encountered a wide
variety of views, most were variations of two basic proposals:
revert to the former ONE system or something like it, or
retain and perhaps modify the current NIO system.
Under the NIO system, the estimative process can begin,
as it has all along, with a request from a consumer. The
appropriate NIO makes a recommendation to the DCI, who
approves or disapproves. If he approves, the NIO becomes
responsible for the proposed estimate, the choice of NIO
depending on the subject matter involved. The NIO can also
initiate a proposal, subject to the approval of the DCI, and
the DCI himself may do so. The NIO may or may not choose to
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
convene a meeting to decide on terms of reference or to receive
oral contributions, but almost always does so. Such meetings
involve representatives of the agencies and departments of the
intelligence community most interested in the completed estimate
and best able to contribute to it, and sometimes include policy
makers to help assure that the proper questions are addressed.
The NIO selects a drafter (or drafting team) from among
the same agencies, in consultation with the parent agency or
component, and research and writing get under way. The
drafters work under the guidance of the NIO; written con-
tributions may or may not be requested from the participating
agencies. The draft, when completed, is circulated to the
participating components for review prior to its consideration
and coordination at an interagency meeting. This meeting, of
representatives of the intelligence chiefs of the participating
components, is usually chaired by the NIO, but on occasion he
designates someone else. When coordination is completed, the
draft is reviewed by the DCI (who may also have involved him-
self at earlier stages) and considered by the National Foreign
Intelligence Board. Once approved by the DCI, with the agree-
ment or disagreement of other NFIB principals shown, the esti-
mate is published and disseminated.
The system in the former Office of National Estimates
differed in important respects. Except for three major
military estimates (which were drafted by CIA teams), drafts
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
were prepared by a small staff of analysts in ONE, who devoted
their full time to such work. These drafts were reviewed by
the Board of National Estimates, a small group of distinguished
generalists responsible to the DCI. The Board provided a col-
legial review, designed to ensure that the proper questions had
been asked and answered, that there were no gaps in the treat-
ment of the subject, and to test and if appropriate to challenge
the judgments in the draft. Drafts were then sent for review
to the participating agencies. Subsequently, representatives
of the chiefs of these agencies met with the Board to comment
on the draft and to coordinate it, prior to its submission to
the United States Intelligence Board (now NFIB).
There is still another difference in the two systems.
The staff and Board of ONE typically spent far less time on
liaison with consumers than do most NIOs, who are explicitly
charged with making such liaison an important part of their
duties. This lesser emphasis was in part to ensure that the
separation between intelligence and policy was maintained, a
separation regarded at that time as highly important by many
in the intelligence community.
The current system gives each individual NIO a larger
measure of autonomy and responsibility for the production of
estimates than any one individual had under the ONE system.
It also encourages and requires a close working relationship
between each NIO and the main producers and consumers in his
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
particular area. The former results in more structural flexi-
bility and responsiveness than was possible under the ONE sys-
tem. The latter makes possible better communication between
producers and. consumers. Both results were highly valued by
most of the consumers who commented on the subject. A third
major benefit, in the eyes of some producers, is that the NIO
system puts the drafting responsibility on analysts who deal
with a subject on a day-to-day basis, and who, theoretically
at least, have the greatest expertise on that subject.
At the same time, many producers and some consumers were
quite vocal about what they saw as shortcomings in the NIO
system. One frequently heard criticism was that estimates
before 1973 were of more uniformly high quality than those
produced now. Another was of the ad hoc nature of the
drafting process, whereby analysts with little or no expe-
rience in drafting estimates are asked to do the job. It
was pointed out that the best substantive expert in town is
not necessarily able to write a good estimate, and that even
if he is, he may be so busy responding to a host of demands
on his time that he is unavailable to the NIO or forced to
divide his time among different tasks and slight his drafting
assignment.
The system, as seen by some, also places an extraordinarily
heavy burden on one person--the NIO. This individual must take
on almost single-handedly not only the substantive responsibility
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
for the accuracy and thoroughness of the estimate but also
the managerial and diplomatic chores involved in steering it
through the coordination process. At the same time, he must
cope with the numerous other responsibilities of his position.
There is no collegial review apparatus to spot weak or missing
links in the paper, or to challenge the conventional wisdom,
as was the case under the ONE system. Finally, a single NIO
is probably less able than a group to deflect advocates of
particular bureaucratic viewpoints.
Efforts have been made to respond to some of these prob-
lems. An Estimates Advisory Panel made up of distinguished
individuals from outside the intelligence community has been
approved though not yet activated; it would supply an element
of review. The recent reorganization of the production offices
in CIA's Directorate of Intelligence was undertaken in part to
put more analytical strength into the Agency's estimative
effort and to make it bureaucratically easier for substantive
experts to be freed from competing demands on their time while
working on estimates.
We seriously question, however, whether these steps,
laudable though they may be, go far enough. If the President,
his top advisors in the national security and foreign affairs
field and the American public are to be confident that esti-
mates contain the very best intelligence judgments that can
be made, further organizational improvements need to be
undertaken.
Approved For Release 200 N~tiDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Internal Collegial Review
A small group of distinguished generalists of proven
wisdom and ability, chosen from both within and outside the
intelligence community, ought to be reinserted into the
estimative process as a collegial review body.
In making this recommendation, we are aware of the sharply
contrasting views that have been expressed about the work of
the former Board of National Estimates, which performed the
collegial review function until it was abolished in 1973.
Its detractors charged that the institution was out of touch
with the needs of customers and that it encouraged an undue
search for consensus at the expense of sharpness in estima-
tive language and judgments. Its defenders maintained that
its labors resulted in a more thoughtful and sophisticated
product, and that it helped guarantee thoroughness and
objectivity of analysis and presentation. At any rate,
since the demise of the Board there has been no collegial
review of estimates, save the necessarily brief and often
cursory one that occurs when the National Foreign Intelligence
Board meets to consider a draft.
Arguments over the value of a collegial review of esti-
mates by a body of generalists continue to this day. Most
of the consumers whom we interviewed and who commented on
the subject clearly supported the concept. Producers were
mixed in their reactions. Some of those at or near the
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
working level tended almost instinctively to oppose it. As
one put it, in extreme terms, "the less review the better;
the customers' reaction is the only review we need." There
tended to be greater support for the concept among senior
intelligence officers, present and past, many of whom felt
that collegial review by generalists is a vital part of the
process that should not be neglected any longer.
We find the arguments of those favoring collegial review
persuasive. Better quality could ensue from the review of a
draft by a group of thoughtful, analytically incisive indi-
viduals who are sharp enough to catch faulty logic, wishful
thinking, and inconsistencies. More thoroughness could
result from the examination of a draft estimate by a group
of generalists, each bringing to bear his own approach and
knowledge, so that the draft is considered from a variety
of angles. Greater objectivity could be attained because a
group independent of any bureaucratic loyalties would be
better able than a single individual to fend off pressures
to alter judgments or avoid hard calls. Also, generalists
can call attention to some important element in a situation
that the experts are apt to take for granted and say nothing
about, or to facets of a subject that they might have overlooked.
Members of the collegial review body ought to be drawn
from various professions in and out of government, with the
widest possible representation from the major academic
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
disciplines used in the estimative process. Inclusion of a
political economist, a scientist, an economist, a military
specialist and a sociologist or anthropologist might be a
good start. Members should be appointed by the DCI, and be
responsible only to him.. To prevent bureaucratic hardening
of the arteries, members should ordinarily not serve for
more than four or five years. The review body should be a
full time operation, and should be close to the DCI both
organizationally and physically. Beside examining estimative
drafts, the group might generate or discuss ideas for new
estimates, review old ones to see if they need to be updated,
and perhaps stimulate and comment on thoughtful single agency
products inappropriate for national estimative treatment.
Review by Outside Consultants
The Estimates Advisory Panel (EAP) could, if implemented,
serve as an adjunct to the collegial review body, and possibly
as an eventual source of recruits for it.
As originally conceived, the EAP was to consist of about
three dozen carefully selected experts, primarily if not
entirely from outside the intelligence community. Members
would be invited by NIOs to gather in small sub-panels to
review specific estimative projects while they were under
way and to provide comments and recommendations.
The use of outside consultants on estimative problems
is not new, dating back to the beginnings of national estimates
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
in 1950. Those familiar with the history of their use felt,
for the most part, that only indifferent success had been
achieved overall, but that sometimes the results were useful.
Intelligence producers who commented on the EAP were only
mildly optimistic. They pointed out that its real value
would be directly proportionate to the caliber of the people
enlisted, and expressed skepticism that enough able people
could spare the time to make the idea work well.
Their caution seems well founded, for the problems in
using outside consultants are substantial. A consultant
must be willing and able to spend a good deal of time going
over the intelligence data on which an estimate is based;
this can be a real problem for busy, prominent specialists
with their own careers and interests. Another problem is
that the consultant may find it difficult to cope with the
rather special world in which intelligence producers operate.
As one senior official said, "if it is hard for a producer
to get into the mind of a policy maker, it will be three
times as hard for a consultant to do it."
Nevertheless, the EAP concept seems worth a try. Con-
sultants have performed a useful role in the estimative
process in the past, and can do so again. At the least,
assuming that top notch people are recruited and used, the
EAP could help allay concerns that national estimates are
not the result of the best thinking that can be obtained on
a subject.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Changes in Current Practices
There remains the question of what changes should be
made in the basic practices now employed for producing
national estimates. We recommend two.
The proposal for inserting a collegial review body into
the process, as recommended above, could or could not consti-
tute a basic change, depending on the way it is implemented.
On the one hand, a collegial review group could serve in an
advisory role to the NIOs, with the latter free to accept or
reject its advice and recommendations. On the other hand,
acting for the DCI, it could have the last word during the
production process on matters of scope and content, much as
the former Board of National Estimates did. The ultimate
responsibility and authority for the scope and content of
national estimates would of course be the DCI's, as it has
been all along.
In making our recommendation for a review group, we had
in mind the latter alternative, for a number of reasons. If
the review group were only advisory, it would be no more than
a group of internal consultants. We doubt that it could then
be justified, particularly as a full time body, and it might
have difficulty in acquiring and holding the caliber of people
required. Substantively, it would not have the authority to
go along with its responsibility for insuring, through a
collegial approach, high quality and the thorough and com-
plete treatment of a subject.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
A collegial review group with authority would serve two
other purposes. It would be an additional means of insuring
the objectivity and disinterestedness of national estimates,
qualities that should be among their prime contributions to
the policy-making process. This contribution is an important
one, for the comparative advantage of intelligence officers
vis-a-vis their policy making counterparts lies not in their
being more able, better informed, or better intentioned, but
in the vantage point from which they do their work and in
their separation from the policy process. Second, a review
body with authority over substance would serve as an additional
means of both highlighting and exercising the primacy of the
DCI in the production of national estimative intelligence.
We urge a second basic change in current procedures,
which we consider to be as important as the first. That is
the establishment of a small staff of analysts who would be
charged with the initial drafting of estimates. It would
have no other duties. The staff, like the collegial review
group, would be an instrumentality of the DCI. Care should
be taken to insure both the substance and appearance of this
relationship, and the separateness of the staff from the
constituent elements of the intelligence community, for the
reasons already suggested in the preceding paragraph.
There are good reasons for having a full time staff to
draft estimates. The drafting of estimates requires particular
90
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
skills, which are not easily found or developed. Native
ability and substantive expertise do not guarantee them, for
they involve the crafting of a distinctive art form--the
national estimate--and the development of perspectives and
approaches suitable for it. Experience strongly suggests
that these skills are best developed through specialization
and experience, i.e., by the repeated exposure of an indi-
vidual to the actual task of drafting. The present method
of drafting has created some problems, as mentioned earlier,
and almost certainly accounts for much of the "unevenness"
in the product frequently mentioned to us.
We recognize that it is untidy to make the above two
suggestions for change in the present system of producing
estimates, and to go no further. It seemed better, however,
to confine ourselves to a discussion of principles, rather
than to try to set forth a detailed blueprint of what the
estimative machinery might look like after the changes were
made. In the same vein, we make an additional recommendation.
According to the comments of those we interviewed, and. by our
own reckoning, the NIOs have made a most valuable contribution
in carrying out their explicit responsibility for maintaining
contact with the principal users of estimates. This has
helped to make estimates more relevant and responsive to
their needs. We urge that provision for this kind of
dialogue be maintained.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
This study has stressed that if estimates are to be
relevant and useful, producers must have a good understanding
of the needs of users. Common sense suggests that such under-
standing can best be acquired if producers and users are
reasonably closely linked, at least to the extent of com-
municating directly on matters of scope, timing and the
issues to be addressed in an estimate. On this basic pre-
cept there is no real dispute, but there are differences of
opinion on how to define "reasonably closely" and on whether
the benefits of close association outweigh the risks.
A number of policy makers saw little need for any
additional effort to provide guidance to the producers of
estimative intelligence. Some thought that if an intelli-
gence producer were competent and well informed, he could
determine on his own what users needed and wanted. Most
policy makers, however, considered that efforts to improve
communication were necessary and desirable. Generally, they
felt that the NIOs had done much to bridge the gap between
policy officers and the producers of intelligence, but that
much remains to be done.
If there is to he an effective dialogue between the
producers and users of estimates, it would seem necessary
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
that certain conditions be met. The most important, rioted
before in this study, is clear evidence of interest on the
part of the President and other senior policy officials in
the utilization of estimative intelligence as part of the
policy-making process. The President in particular, as the
chief "user" of intelligence, sets the tone. Second, senior
officials on both the intelligence and policy side must
actively foster the dialogue and devote part of their time
to producer-user relationships; the key individuals are the
DCI and the President's National Security Advisor. To date,
these conditions have been satisfied only partially and
sporadically.
In November, 1971, the National Security Council Intel-
ligence Committee (NSCIC) was established to provide user
"direction and guidance on national intelligence needs," and
"evaluation of intelligence products from the viewpoint of
7
the user." It was made up of senior policy makers, plus
the DCI. Largely because of a lack of top-level attention,
the NSCIC languished, holding only one substantive meeting
during its entire five-year existence. Its second Working
Group, established in late 1974, made only a small start in
educating users about the products and capabilities of the
intelligence community. The NSCIC was criticized for its
White House press release, November 5, 1971.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
inactivity by both the Murphy Commission and the Senate Select
Committee on Intelligence.
By the time the Select Committee submitted its report,
the NSCIC had been abolished. Executive Order 11905 of
February 18, 1976, requires the DCI to consult with users
of intelligence to ensure the "timeliness, relevancy, and
quality of the intelligence product" and to "establish such
committees of collectors, producers, and users of intelligence
to assist in his conduct of his responsibilities as he deems
appropriate." In implementation of this order, the DCI
charged the National Intelligence Officers with determining
consumer needs for intelligence and gave his Deputy for the
Intelligence Community the responsibility for the evaluation
of intelligence products. At present, there is no formal
organization of users to provide guidance to the producers
of intelligence. It is probably fair to say that the col-
lection and production of intelligence are governed more by
what producers think is needed than by guidance from users.
Close User-Producer Links: Risks and Benefits
There are two basic schools of thought about the proper
relationship between intelligence producers and policy makers.
The first holds that intelligence analysts ought to maintain
a certain remoteness from those involved in decision making
"so as to keep intelligence pure and untainted by policy
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
8
pressures," as Hanson Baldwin put it. The other asserts that
close cooperation between intelligence and policy officers is
not only permissible but mandatory, if both are to do their
jobs properly. Most of the consumers and producers we inter-
viewed held the latter view. They felt that the risks of a
close relationship, although not to be taken lightly, are
nevertheless tolerable in view of the potential benefits to
both sides.
The chief concern of those who worry most about the risks
is that a close association will in time make the estimators
too sensitive to the user's interests and attitudes. As one
former senior intelligence officer put it, there is a. distinct
danger that the producer will get so wrapped up in day-to-
day support of the policy maker that he will become, con-
sciously or unconsciously, a member of the policy team. By
so doing, he risks losing his objectivity and his ability
to step back, take the longer view, and go beyond the opera-
tional considerations of the moment to product a paper that
provides longer range perspectives and perhaps challenges
the postulates on which policy is based. For their part,
policy makers can legitimately be concerned that intelligence
producers will, intrude into their territory if the relationship
Hanson Baldwin, "The Future of Intelligence," Strategic
Review, Summer, 1976.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
with them becomes quite close--though no policy makers that
we talked to complained of this having happened.
The primary benefit of a close working relationship, as
we and our respondents see it, is a clearer, more realistic
appreciation by each side of the capabilities, limitations
and needs of the other. The producer of intelligence would
have less need to guess about what is wanted by the users of
his products. He could acquire a better appreciation of the
problems preoccupying the policy maker, and what issues are
paramount in his thinking. As one of our respondents put
it, it is necessary for the producer of estimative intelli-
gence to get inside the mind of the user; this cannot be done
without close and continuing association. Such association
would facilitate the production of papers which are relevant
to the needs of the policy maker, and responsive to them.
For their part, many policy makers have an imperfect
view of what the intelligence business is all about. More
particularly, they are unaware of what estimates can and
cannot be expected to do for them, and of the difficulties
that are sometimes involved in zeroing in from a distance
on matters of interest to them. There also appears to be an
insufficient understanding of the process of producing esti-
mates and of the nature of the product that results, with the
consequence that an estimate is viewed by some as no different
from any other intelligence product. Perceived deficiencies
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
in the product have contributed to this viewpoint. So has the
failure to understand that a national estimate is the voice of
the DCI, that coordination has many positive benefits, and
that the vantage point of the intelligence estimator permits
him to make a distinct, if not unique, contribution.
Providing for Closer Contacts
We found a general feeling among both users and producers
that providing for the right kind of communication between
them will not be easy. One very obvious constraint on both
sides, but especially on policy makers, is a lack of time.
Policy people who admit to neglecting their relations with
intelligence producers complain that there simply is not
enough time in their operational and crisis-oriented schedules
to fit in regular meetings with intelligence analysts, no matter
how desirable this might be. Additionally, the fact that users
and producers are physically located up to an hour apart is a
strong deterrent to impromptu meetings when time is available.
A solution sometimes suggested for bringing intelligence
and policy people together more regularly is the creation of
an organization like the NSCIC, but without that group's
liabilities. Among the few producers and users who commented
there was skepticism, however, about the efficacy of a new
attempt to provide a formal, institutional solution. Such
skepticism is understandable in the light of experience.
Unless vigorous and continuing leadership is exerted from
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
the top, a new organization might soon atrophy. As the pres-
sures of more immediate operational concerns grew, top-echelon
members of the group would be increasingly tempted to designate
lower ranking officers to attend meetings. This would reduce
the organization's power to make decisions and to provide
guidance.
This alone is not reason enough to shy away from a new
effort to establish such a group, and in fact such an effort
ought to be made. Even a lively second or third echelon users'
organization would provide a forum for airing the concerns and
needs of policy makers. It would be wise, however, not to
expect too much of such a group, given the problems and
pressures described above.
Among the subjects it might tackle are the following.
On estimates:
--topics for national intelligence estimates;
--terms of reference;
--priorities;
--the evaluation of published estimates; and,
--the role of estimates in support of "opera-
tional" requirements and in crisis situations.
More generally:
--intelligence research programs;
--significant gaps in intelligence coverage;
--the allocation of intelligence resources as
it affects the final product;
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
--conflicting demands on intelligence;
--imbalances in the intelligence effort; and,
--the broad capabilities, limitations and
problems of intelligence collection, analysis
and production, as they affect the ability
of intelligence to support policy makers.
Even if a formal group is established, it seems evident
that the improvement of communication between policy officers
and intelligence officers will depend mostly, as it has all
along, on the initiatives taken by individuals to get to know
one another better. Furthermore, it is clear that most of the
burden will be on the producers of intelligence. As one quite
senior State Department official put it: "the policy maker
does not communicate with intelligence--you know that. The
intelligence officer has to anticipate and do everthing.
The policy man will not tell him enough, early enough."
Another senior customer said that he found it difficult to
provide advance guidance, and that intelligence should not
look for it. He added that the intelligence producer should
instead derive guidance from a greater interaction with users,
so that he could identify independently what was needed.
Certain related themes emerged from our interviews.
Several policy makers stressed the importance of the per-
sonal relationship between producers and users. That is,
only if there were relations of mutual confidence would a
policy maker be prepared to express his innermost thoughts
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
or to welcome periodic get-togethers at which perceptions
could be compared. There were also suggestions that the
intelligence community do market research, in a rather
formal way. As one senior policy maker put it, intelligence
has to find ways of getting its message across, and this is
a marketing problem.
Within the framework of an expanded dialogue, each side
needs to work harder on certain aspects of the relationship.
The user, busy though he is, should do all he can to ensure
that the intelligence producer is aware of trends in policy
that may create new requirements for estimates. He also
ought to strive to find out more about what intelligence
can, and cannot, do for him. This would involve a far
better understanding than many policy makers now have of
just how the intelligence community functions.
Producers have a responsibility for seeing that policy
makers acquire such an understanding. The main users of
estimates are busy people. Moreover, there is fairly rapid
turnover in many of the key positions in the national security
and foreign policy field. Under these circumstances, the
intelligence community has probably been remiss in not having
developed a program for acquainting its customers with the
capabilities and limitations of estimative intelligence.
When this is done now, it is on an ad hoc basis, with little
consistency or continuity. As simple a thing as the production
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
of a package of sophisticated briefing material would probably
help, especially if it were delivered personally by an intel-
ligence officer with whom the user had or was developing a
working relationship.
Producers also owe it to themselves, and to the ultimate
utility of their product, to work harder than many now do to
absorb what is readily knowable about current policies and
the preoccupations of policy makers. An alert intelligence
officer can acquire a substantial understanding of at least
the thrust and direction of U.S. policy in his area simply
by keeping his eyes and ears open. Several producers
admitted as much, and ventured that this was often enough
to permit them to do their jobs. And, of course, it is the
duty of the producer of estimates not only to tell the con-
sumer what he wants to know, but also what the producer
thinks the consumer ought to know.
This gives rise to two other thoughts on this subject.
The first is that those charged with producing estimates can
learn much about the concerns of policy makers, particularly
those at the Department of State, from their colleagues in
the Directorate of Operations (DDO). Officers of the DDO
maintain continuing contact with their opposite numbers at
the Department
Also, DDO division chiefs meet regularly with the Assistant
Secretaries for their areas. At a minimum, the results of
25X1
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
such meetings ought to be passed on to the producers of
estimates; it would be even better if the latter were invited
to participate as appropriate.
Second, one of the best ways for intelligence producers
to get to know the needs and concerns of users is to work in
or close to a policy position for a time. We found that the
most perceptive and pertinent comments on user-producer rela-
tions during our interviews came from the relatively few
individuals who served in both types of position. But such
cross-fertilization should be practiced carefully to avoid
conflicts of interest and the blurring of the lines of separa-
tion that both users and producers believe it essential to
maintain. There are also serious impediments to assigning
policy officers temporarily to intelligence roles. For care-
fully selected intelligence officers, however, a stint in a
policy office could be highly useful as a device to foster
better mutual understanding.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
What we have heard and read during the course of this
study convinces us that national intelligence estimates will
have a useful role to play in the policy-making process for
the foreseeable future. Whether they actually play such a
role will be largely up to the intelligence community itself.
Far from having outlived their usefulness, estimates would
seem to be more urgently required than ever. There is good
reason to think this is true, to one degree or another, of
9
most kinds of finished intelligence. But because estimates
alone attempt to discern what lies ahead, they surely should
get high priority in terms of our total intelligence effort.
Other reasons for their potential importance are not
hard to adduce. The world with which intelligence and policy
must deal is still changing with breathtaking speed. Rela-
tions between and among nations and people are becoming more
and more complex. As they do, it is getting steadily more
difficult for the policy maker, pressed by a great many more
immediate concerns, to find the time to take the long view.
See The Future Market for Finished Intelligence, (S)
a CSI s tudy issu~" n A ugust T7-6; -fir a more etai ed look
at finished intelligence needs in the years to come.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Nor can he usually afford to sort out for himself the many
interacting factors that will affect the prospective behavior
of a foreign entity. And unless he is directly involved in
such matters, he will rarely be able to stay abreast of new
and impending scientific or technological changes that may
impact upon his responsibilities.
This is where intelligence comes in--or should, because
these are the sorts of things that useful estimates come to
grips with for their readers. Estimates have done this well
at times and poorly at others, but will need to be much
more effective and consistent from now on. The task will not
be easy. For one thing, the growing complexity and inter-
relatedness of the world's military, technological, economic,
political, and social affairs will make it much more difficult
to be clear and precise, or to phrase estimative judgments in
simple declarative sentences. More and more tangible and
intangible factors will have to be considered before reasonable
conclusions can be formed. This will make it increasingly
important to weigh carefully all the alternative views on
crucial issues and to articulate them clearly. It also
means that the underlying data base is likely to become
larger, or clouded by conflicting information. Yet to
ignore or pass lightly over parts of the data would be risky.
For another, it is likely that the resources of the
intelligence community, like those of the nation as a whole,
Approved For Release 2002/FYYI: l-MJP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
CONFIDENTIAL
will continue to be stretched to accommodate other urgent
needs. This may well mean that estimates will have to be
produced more economically, in a shorter time span, and for
a wider audience. Eventually this seems bound to entail pro-
ducing fewer estimates. Producers and users will no longer
be able to enjoy the luxury of trying to know everything
about everybody, but will have to concentrate on those
countries and problems that affect our security the most.
This means a greater willingness to say "no" to requests
for estimates on less essential matters.
"Quality" should become the watchword, because the need
for estimative excellence will continue to grow as U.S. power
and self-sufficiency become more circumscribed. Our national
well-being will depend more heavily on knowing accurately
what our adversaries and allies may be up to. This does not
necessarily mean refining the ability to predict specific
events. It will, rather, mean being close to the mark in
forecasting trends, measuring capabilities of important foes
and friends, and assessing their intentions.
Even when quality must be cut back, the plate will remain
full. Surely there will be no scarcity of important estima-
tive issues ahead, including some that are critical and a
few that are explosive. Unforeseeable crises will occur and
troublesome developments are bound to arise. Many topics of
present concern will remain and become even more difficult
to handle,
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Assuming that well thought-out, relevant estimates are
produced, there need be few worries about a market. The most
important customers will continue to be the President and his
chief national security advisors. To the degree that these
leaders find estimates useful, and to the extent that esti-
mates have an established place, formal or informal, in the
policy-making structure, their market will be assured. The
frequent turnover of military and political decision makers
at intermediate and lower levels is a further guarantee that
a potential audience for estimates will always exist. The
DCI, however, will continue to be the key person in getting
estimates read and taken seriously. If the DCI is convinced
that estimates are as good as they can be and tries to con-
vince others, estimates should have few problems in reaching
the right people.
And the "right people" may well turn out to include a
much larger number in the future than it has up to now.
Congressional interest in the conduct of foreign and national
security policy and the new emphasis on performing govern-
mental functions as openly as possible suggest that estimates
will increasingly need to be made available to the Congress,
and perhaps even to the public in some form. To do so will
require a good deal of careful work in the area of determining
what must remain classified in order to protect intelligence
sources and methods and to avoid damaging our relations with
friendly governments.
Approved For Release 20021 I*,iPUA- ?P80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
In conclusion: their place in the policy-making process
may remain modest and their ability to provide satisfactory
answers about the shape of the future will be limited, but
for all that, national intelligence estimates will be needed
more than ever. There should be a sustained effort to do
them as well as possible.
107
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Much of the commentary and information contained in this
study was derived from 66 oral interviews with 97 individuals,
many of whom have had extensive experience or direct acquain-
tance with national intelligence estimates. Many, although
not all, of the suggestions offered in the study stem directly
or indirectly from their views of what needs to be done to
improve the product and the process.
Interviewees were contacted by members of the study team
principally because we knew them, or knew of them, and believed
they would have some useful views on the subjects included in
the study. Inevitably, we contacted many more individuals than
were actually interviewed. Some meetings we ardently hoped to
arrange simply proved impossible from the start, or had to be
scrubbed at the last minute, because of the press of other
obligations on some very busy people. In no sense, then,
can this be considered a "scientifically" constructed inter-
view base. We did try very hard, however, to cover the whole
spectrum of officials involved with estimates both on the
consumer and on the producer sides. We also tried to avoid
giving undue weight or attention to the views of any one
element or sector.
It proved impossible to cover the entire range of
issues addressed in this study with any one individual.
Approved For Release 2002/ I1 lhI 1$Tlkk80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
While most of the people we talked with gave generously of
their time, inevitably there was not enough opportunity with
many of them to cover more than several of the many broad
questions we wanted to take up. It did help that three and
at times all four members of the study team attended the
interviews, thereby enabling us to cover more ground and com-
pare impressions afterwards of what was said and meant.
Careful notes were taken.
Consumers
We talked with 46 individuals whom we could classify as
current users, to one degree or another, of estimative prod-
ucts, and we interviewed seven prominent former consumers,
whose reflections on estimates as they viewed them from some
very senior positions promised to provide an additional
dimension to the study.
Of the current consumers, seven were National Security
Council staff members or their deputies. We spoke with 21
consumers in the Department of State, including one Under-
secretary, the Counselor, 11 officers at the Assistant
Secretary or Deputy Assistant Secretary level, six at the
level of country director, deputy country director, or desk
officer, and two officers of the Policy Planning Staff. At
the Pentagon we interviewed 12 consumers, including eight
in the International Security Affairs office ranging from
the Deputy Assistant Secretary level on down, two in the
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Office of the Secretary of Defense, and two general officers
on the staff of the Joint Chiefs. We also talked with four
officials in the economic policy realm--three in the Department
of the Treasury, and the Special Representative for Inter-
national Trade Negotiations. We interviewed one officer of
the Arms Control and Disarmament Agency and the Executive
Secretary of the President's Foreign Intelligence Advisory
Board.
The former consumers whom we interviewed once held the
following posts: Secretary of State, Secretary of Defense
(two individuals), Assistant to the President for National
Security Affairs, Assistant Secretary of Defense for Inter-
national Security Affairs, Director of the Arms Control and
Disarmament Agency, and Executive Secretary of the National
Security Council. All had insightful and valuable comments
and suggestions to offer.
Producers
intelligence producers with current responsibilities relating
to some aspect of the estimative production process. We also
talked with five former members of the Board of National
Estimates who are now retired.
Current producers whom we talked to in considerable
detail included ten National Intelligence Officers or their
deputies. Non-CIA intelligence producers we saw included
Approved For Release 2002/~I~/U4ID8 A-80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
the Director and the Deputy Director (for Estimates) of the
Defense Intelligence Agency, and the Director and Deputy
Director of the Department of State's Bureau of Intelligence
and Research.
We interviewed or held seminars with 26 officers of
CIA's Directorate of Intelligence, including the Deputy
Director for Intelligence, five analysts and managers from
the former Office of Current Intelligence, six from the
former Office of Political Research, six from the Office of
Strategic Research., six from the Office of Economic Research,
and one each from the Office of Scientific Intelligence and
the Office of Weapons Intelligence.
Use of Estimates and Other Writings
From the outset, it proved difficult to develop a workable
plan for using national intelligence estimates themselves in
the study. Because the existing body of estimative literature
includes so many different kinds of products published on such
a wide variety of subjects over the past 27 years, the prob-
lem of choosing a representative sample to dissect and comment
upon would have been formidable. enough by itself. Assuming
that such a sample could have been assembled, the study team
would have been severely limited in its ability to comment
substantively on all but the relatively small number dealing
with. topics on which. team members had some degree of personal
expertise. Exploring in detail the processes by which all
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
but the most recent estimates have been produced would have
been almost as difficult; few. records are available, and
personal recollections of participants usually fade rapidly
and become untrustworthy.
We also derived relatively little help from earlier
studies and commentaries on the estimative record per se.
No comprehensive look at estimates as a whole has ever been
undertaken, and only a handful of post-mortems or validity
studies on series of estimates have been done, although those
that were written were consulted and proved useful. Reading
those studies helped to reaffirm to us the wisdom of avoiding
comment on the accuracy of specific estimates. This is
devilishly hard to do with any real measure of validity.
Many estimative judgments are conditional, i.e., their
accuracy hinges on something else happening that may or
may not occur. Others will prove "wrong" simply because
the United States or some third country will have taken
steps that altered the situation that obtained when the
judgments were made. Of those that can more readily be
identified years later as "right" or "wrong" calls, some
will have been of major and some of minor importance, and
it would be difficult to weigh them accurately now. Finally,
to be able to conclude whether a past estimate was either on
or off the mark, one would have to know his history well, or
do a lot of research to find out what really did happen.
Approved For Release 2002/11/04D CI i bP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
For these reasons, we decided to limit our use of the
texts of estimates to the study of some of those issued over
the past year or two and to the identification of specific
problems or issues concerning them by seeking comments and
feedback from users. In the event, most of the comments we
gleaned were too general or fragmentary to give us more than
a few leads for further investigation.
A fairly extensive although fragmentary body of literature
about estimates and their role in the policy-making process
does exist and was used. Especially helpful were the more
than three dozen articles on various aspects of the estimative
business that have been included in the quarterly publication,
Studies in Intelligence, since 1957. The reports of the Com-
mission on the Organization of the Government for the Conduct
of Foreign Policy (the Murphy Commission) and of the Senate
Select Committee on Intelligence were very useful. Various
internal CIA memoranda, histories, and reports on estimates
also proved helpful, as did selected portions of books,
magazine articles, and other open literature concerning
estimates and their role in the policy-making process.
Most of our data, however, came from the people we interviewed.
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
CONFIDENTIAL
APPENDIX B: SELECTIVE BIBLIOGRAPHY
1. Material in the following unclassified publications that
were among those used by members of the study team appeared
especially relevant to issues discussed in the study.
A. Books
25X1A
Hilsman, Roger, The Politics of Policy Making in
Defense and Foreign Affairs, (Harper & Row,
New York, 1971
To Move a Nation, (Dell, New York,
1974).
Kent, Sherman, Strategic Intelligence for American
World Policy, (Princeton University Press, 1949).
Kirkpatrick, Lyman B., The U.S. Intelligence
Community, (Hill & Wang, New Yor , 1973T.
Moorstein, Richard, and Abramowitz, Morton,
Remaking China Policy, (Harvard University
Press, 1971).
deRivera, Joseph H., The Psychological Dimension
of Foreign Policy, (Merrill Co., Columbus,
Ohio, 1968).
Wohlstetter, Roberta, Pearl Harbor-Warningand
Decision, (Stanford University Press, 1962).
B. Reports
Final Report, Senate Select Committee to Study
Governmental Operations with Respect to
Intelligence Activities, (U.S. Government
Printing Office, Washington, D.C., 1976).
Final Report, Commission on the Organization of
the Government for the Conduct of Foreign
Policy, (U.S. Government Printing Office,
Washington, D.C., 1975), Volume 7.
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-00630A000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
C. Magazine and Journal Articles
Ben-Zvi, Abraham, "Hindsight and Foresight: A
Conceptual Framework for the Analysis of
Surprise Attacks," World Politics.
Cline, Ray S., "Policy Without Intelligence,"
Foreign Policy, Winter 1974-75.
Cooper, Chester L., "The CIA and Decision-Making,"
Foreign Affai.rs, January 1972.
Graham, MG Daniel 0., "Estimating the Threat:: A
Soldier's Job," Arm Magazine, April 1973.
Hughes, Thomas L., "The Power to Speak and the
Power to Listen," in Secrecy and Foreig~n
Policy, Thomas M. Franck E Edward Weisband,
eds., (Oxford University Press, 1974).
Kent, Sherman, "Estimates and Influence," Foreign
Service Journal, April 1969.
Knorr, Klaus, "Failures in NIEs: The Case of the
Cuban Missiles," World Politics, April 1964.
Shlaim, Avi, "Failures in NIEs: The Case of the
Yom Kippur War," World Politics, April 1975.
Wohlstetter, Albert, "Is There a Strategic Arms
Race?" Foreign Policy, Summer 1974.
, "Rivals But No Race," Foreign
Policy, Fall 1974.
, "Optimal Ways to Confuse Ourselves,"
Foreign Policy, Fall 1975.
II. The following classified reports, memoranda, articles, and
studies were found to be especially pertinent to some of
the issues taken up in this study.
A. Articles in Studies in Intelligence (all issues
classified Secret).
25X1A
115
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 25X1 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
25X1A
B. Miscellaneous Official Studies, Memoranda, and Reports
Center for the Study of Intelligence, Intelligence
Monograph, CIA Intelligence Sup oorrt for Foreign
and National Security_Polic Mating, (TR/IM 76-01
January 1976, S).
, Intelligence
Monograph, The Future Market for Finished
Intelligence, (TR/IM 76-02, August 1976, S).
Intelligence Community Staff, An Assessment of National
Foreign Intelligence Support to the U.S. Government,
-(December 1976, S).
, Analysis of Intelligence
on the Middle East War - October 1973, (August
1.974, TS-Sensitive).
Kent, Sherman, The Law and Custom of the National
Intelligence Estimate, (February 1976, S - CIA
Internal Use Only).
Office of National Estimates, Memorandum, 1967's
Estimative Record--Five Years Later, (CIA/ONE
August 1972, TS).
Memorandum for the DCI,
NIEs on the Soviet Role in the Middle East, 1965-
1970, (CIA ONE July 1970, S-Sensitive).
, Study, NIEs on the USSR,
1950-1957, (CIA/ May 1958, TS).
Staff Memorandum, The
Arabs an t e Estimates; 1951-1965, (CIA/ONE
November 1965, TS).
Office of Research and Development, Re ort on Proceedings,
Symposium on Analytical Methodology, (CIA ORD,
Report No. RD[CDAM-1, June 1974, C).
117
CONFIDENTIAL
Approved For Release 2002/11/04: CIA-RDP8O-0063OA000300040001-3
 25X1 Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3
Approved For Release 2002/11/04: CIA-RDP80-0063OA000300040001-3