NOTES ON ESTIMATING

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP93T01132R000100020038-1
• Release Decision: RIPPUB
• Original Classification: C
• Document Page Count: 6
• Document Creation Date: December 27, 2016
• Document Release Date: May 2, 2012
• Sequence Number: 38
• Content Type: MISC

Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
I
The evolution of some techniques
in the national estimating system.
NOTES ON ESTIMATING
Keith Clark
Since NIE 1 appeared in 1950, more than a thousand National
Estimates have been considered and approved by the United States
Intelligence Board or its predecessor, the Intelligence Advisory Com-
mittee. This large number of very solemn documents, the collective
progeny of the intelligence community at large, have been delivered
through the midwifery of the Board of National Estimates and its
Staff. Both the process and the product have undergone certain
changes in the course of seventeen years, and there ought to be some
lessons in a review of this evolution, not only for the midwives in
ONE but for all who participate in the process of conception, gestation
and delivery.
My purpose is to identify, primarily from the ONE viewpoint, some
recurrent dilemmas and common pitfalls in producing estimates, to
note different ways of coping with these, and to suggest some main
sources of strength or weakness, as well as some avoidable wastes of
time and effort. No two estimators would identify all the same
problems as being important or perennial enough to rank as matters
of continuing professional concern, but I offer my observations under
two headings: (1) Style and Scope: the treatise versus the short
answer. (2) Methods and Discipline: predictive estimating and
prophecy.
Having drafted, chaired, or otherwise participated in many of the
National Estimates, I disqualify myself from engaging in much praise
or condemnation, but some subjective judgments seep through. I
hasten to add that the judgments which follow, the arguments which
support them, and the idiosyncracies which pervade them are my
own; they do not necessarily reflect the opinions of any colleagues
on the Board or on the Staff, though I am indebted to members of
both and to other professionals for some of the ideas.
Studies and Short Answers
These tags denote two sets of values, or schools of thought, each
valid by its own lights, which often collide when estimates are written,
Declassified in Part - Sanitized Copy Approved for Release 2012/05/04 : CIA-RDP93T01132R000100020038-1
 Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
W
CONFIDENTIAL
Notes on Estimating
Notes on Estimating
CONFIDENTIAL
debated, and coordinated. It is not a question of mere prose style.
Everyone agrees that for our purposes good writing calls for economy
in words. It is a question of scope and approach. Some look on
estimates as vehicles for educating the reader in all he ought to know
about the problem posed. They reason justly that an informed policy-
maker, like an informed electorate, is a good thing, and the more
informed the better. Acknowledging that NIE's are not encyclopedias
and do have severe limits on length of discussion and depth of detail,
adherents of this approach nonetheless strive to incorporate as much
information as possible into the document, and don't like it when
something they consider information or insight of cardinal importance
is defined by someone else as superfluous detail.
At the other extreme are the short-answer men. They are imbued
with a perfectly correct conviction that most high-level policy-makers
have too much to read as it is, and that if the intelligence community
sins in its publications, it is in the direction of too much rather than
too little. In common mercy, as well as in the interests of getting the
essential message across, they conclude that estimates should be
sheared of all that is not strictly necessary to making the main judg-
ments, and that the latter should be supplied as crisply and quickly
as possible.
It is a rare estimate that does not give rise to some clash of opinion
along these lines, and since it is a very subjective matter, prevailing
doctrine or fashion shifts from time to time and from person to person;
in fact, individuals feel differently on different occasions, depending
on whom they are writing for, their own depth of knowledge and
interest in the subject, their patience or lack of it, and many other
variables. Speaking only of ONE, I once thought it generally correct
to say that the Board favored short answers and the Staff liked informa-
tive detail. This is probably more often true than not, but there are
so many instances of the Board's demanding the addition of informa-
tion and detail to staff drafts that the generalization is not very valid.
The National Estimates show fluctuating trends in this respect over
the years. Insofar as general patterns can be discerned and briefly
described, we leaned in the earlier years toward spareness. This
reflected the strong military influence on early estimative methods, an
influence which made for short answers to short and crisp questions.
It also reflected the kind of problems which preoccupied estimators
in those days almost exclusively-direct Communist threats to the
clear-cut or were made to appear so, and could be sharply defined.
Thus NIE 1, of 3 November 1950, was on "Prospects for Communist
Armed Action in the Philippines During November."
We then entered a kind of baroque period (mid-fifties to early
sixties) in which estimates became more informative, full of subtleties,
refinements, and detail, aimed at describing and assessing foreign
societies and governments in a more complex way. This evolution
was helped along by the participation of more civilians in the process,
with their academic skills and habits of work. It was also partly due
to a growth in the amount of intelligence available (e.g., photography
of the USSR). And it was probably most of all the result of require-
ments for estimates on more complex subjects. For example, the
nationalist revolution in the undeveloped world then in full flower
gave rise to important policy problems for the United States and
consequently to the need for estimates on a subject that was new and
complex. It required conceptualization and even some new vocabu-
lary; short answers to short questions would not do.
Choosing Between Them
In recent years, we have followed an eclectic approach-using both
methods and often mixing them, with the choice being made by the
predilections of those involved after more or less considered judgment
about the , requirements and preferences of the consumer. I shall
not argue for one approach over the other. In the present state of,
the art, and in light of varying consumer needs, we probably do best
to be eclectic. But I offer a few observations about some pitfalls
in the choice.
One observation chiefly concerns "country" or "area" estimates.
These are not done as frequently as they once were, but the art form
is far from dead. What is dead-or ought to be, I think-is the classic
60 or 70 paragraphs that methodically discussed almost every subject
under the sun relating to a country or region in a kind of mechanical
way, under the headings Introduction, Political, Foreign Policy, Eco-
nomic, and Military. Experience has persuaded most of us that this
approach involves much waste motion, and that country or area
estimates can most usefully emphasize a few main points, sometimes
a single main theme with variations. It seldom requires more than
20 paragraphs or so to render these judgments for any country, with
all the supporting detail necessary.
And to do it in shorter scope increases the chances of attaining
several desirable ends: one is that the estimate will be read and
United States Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
 Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
CONFIDENTIAL
Notes on Estimating
remembered by officials at high levels; a second is that the truly im-
portant judgments will shine forth clearly, and not be hidden or
dulled by clouds of detail; a third is that the estimate will not become
obsolete or obsolescent quite so fast when day-to-day developments
put one detail or another out of date; and a fourth, rather bureaucratic
one, is that short papers take less time to do, at least in the stages of
coordination and consideration by the USIB. (It is a true, if lamen-
table, fact that time spent in discussing and coordinating papers often
varies more nearly with the quantity of words to be gone through than
it does with the importance and complexity of the problem at hand;
we sometimes devote so much effort to not being wrong about
secondary and even trivial matters, or to group discussions of literary
idiosyncracie, that we lack the energy and perspective to make sure
that we are right about the big questions.)
The foregoing amounts to a rather more dogmatic argument for
short papers than I really want to make. Let me note two or three
exceptions to the main proposition. One is the kind of estimate
casionally requested (or in some cases annually expected) by high-
level consumers who are already broadly familiar with the problems
about which they ask. Certain levels of brevity and simplification
which might be just right for many kinds of estimates would tell these
particular consumers nothing they don't already know. In these
cases a considerable degree of informative detail becomes mandatory
if the estimate is to have any value. Certain annual Soviet and Chinese
papers fall into this category, since generalized assessments of the
Russian and Chinese military threats are of negligible use to anyone.
Another exception is formed by some special estimates on, say, reactions
to given U.S. courses of action. No one needs to be told that Com-
munist and neutralist reactions to some forward military move by the
United States would be adverse; they need to know how adverse, and
in what ways-particularly the difference between verbal responses
and retaliatory actions on the part of the governments in question.
Sometimes we cannot make these distinctions clearly, but we ought
to try.
Another occasional exception is the "how to think about" estimate policy
most often addressed to some fairly new and unfamiliar forreign problem, or some particular aspect of an area or country which
feels it would be useful to conceptualize in a nonconventional
,irr%ngP tnav be more to structure the problem than to
way.
r--- e p Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
Notes on Estimating
more leisurely, to give more information, detail, flavor, and atmos-
pherics than are otherwise called for. In sum, there are problems
which cannot be treated shortly if the estimate is to do the job it
should. But we can at least try when we start these jobs to be clear
in our own minds what the job is.
Prediction and Prophecy
One of the most persistent half-truths held in the intelligence pro-
fession and among our customers is that estimates are predictions of
things to come, prophecies of the future. This is dogma and it is also
largely true, but when couched in these terms it frequently leads us
down some unfortunate paths and stultifies our thinking. Prediction
is indeed the heart of the matter, but there is a world of difference
between predictive estimating and mere prophecy. Lest I appear to
make a case by pejoratives, let me define my terms.
I use the term predictive estimating to suggest a process which takes
due account of its own limitations and uncertainties. It begins with
awareness of present unknowns, the slippery ground we start on be-
cause of the things we don't know, or can't be sure we know, about
the past or present. It goes on to the future to predict what can be
predicted-by induction from some kinds of evidence, deduction from
other kinds, testing hypotheses against all evidence available, and the
rest of the familiar intellectual disciplines hopefully instilled in us all.
But as it moves along these tried and true paths, predictive estimating
differs from mere prophecy in its continuing awareness of its limita-
tions in the face of the extraordinarily complex array of matters which
will in fact determine future developments.
More specifically, it distinguishes between constants and variables,
and shows awareness of interaction between them; it defines critical
points-crossroads or crunches-and suggests alternative lines of de-
velopment leading from these; it admits ignorance and uncertainty
when it reaches the outer limits of evidence, analysis, and logical
speculation; without yielding to the crudities of "worst case" estimating,
it also avoids the pretentious and useless fallacy of the "single best
guess"; it distinguishes-sometimes explicitly, always implicitly-the
model of a fairly tidy and rational world delineated for purposes of
analysis and comprehensible exposition versus the messier world of
flesh and blood and emotion; it keeps in mind the fact that foreign
governments-even apparently monolithic dictatorships-are as often
as not inwardly subject to conflicting pressures, ambivalences, and
contradictory impulses, even though usage often compels us to talk
 CONFIDENTIAL
Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
Notes on Estimating
Notes on Estimating
as though 'the Soviets,' "Peiping," or "Israel" were each of one mind-
whole, coherent, and consistent-'Plies that the future is already there,
Prophecy, deep within the crystal ball, to be discerned by those who are wise A and lucky enough to do It ctiveaestgreat atingpdces not reject
aided by intuition and hope.
these aids altogether, but it is based essentially on a concept of the
future as too complicated and chancy to permit easy hlmore fro P Able
you are to where you want to be. It is, in short ,
and humbler than prophecy. It is also typically less dramatic, more
cautious and tentative in its conclusions, and perhaps less exciting to tements read. Sometimes it is possible bbut this is legitimate uonly if as laborious
boldly impressive foresight, g
and disciplined intellectual process has been gone through first
All this may sound like pretentious counsels of perfection, and in
any, case inconsistent with earlier remarks on the desirability of short cOns papers. Certainly a published ti of all the seines suciOu , aeblloed ggested out its own scrupulous
would be an infinitely elaborate and tedious document, too much like
a Ph.D. thesis in one of the fields of social science where concentration
on methodology crowds otcontent- But I am talking here as Much
as about the visible product delivered
about an intellectual process
to the printer. We all use various forms of verbal shorthand in getting
our message across; Buththerelis a difference betweentishort Flu d not i
e
be communicated.
getting the message across and short
be indu g d ing only at t the risk
message should be. The latter can
esti-
mates e must quality and, eventually, below the visible substance they
must have a
are to hold together and stand up.
Guessing Games
The record of National Estimates over the years in these respects
is a mixed one. One practice occurs often enough in various guises
' To illustrate an effect of this approach: a number of National Es amens
in recent years have employed the device of presenting the most likely judgmced
on the central question, and then, in immediately following paragraphs ointrod
of going on to
by the sensible admission that this reasoning might be in error,
suggest the implications of alternative hypotheseseven if the odds dont appear
to favor them. I cannot escape the belief that--on close questions of particu-
..aa. an..m-ously to the usefulness of the
to warrant some criticism. It is the temptation or compulsion to
estimate with apparent confidence about any question that anyone in
authority wants to know about. The potent old blandishment, that
if the estimators don't supply answers someone less qualified will, can
sometimes be resisted only by appearing mutinous.
But the plain fact is that estimates on some questions are of neg-
ligible worth, no matter how sophisticated the thinking behind them,
and we ought honestly to say so. We may be paid to estimate, but
we are not paid to do the impossible, and certainly not to pretend
to do the impossible when we can't. A confession of ignorance or
uncertainty may annoy someone who wants practical answers to
practical problems, but in the long run it is better to annoy than to
con him. This is not an argument for refusing to do difficult tasks,
or even to try what may look like impossible ones; it is an argument
for being clear, to ourselves and to our readers, just how safe it is to
skate on the ice in certain areas and just where the ice, for all we
know or might wish otherwise, may be water.
One case in point is the amount of time devoted to predicting the
survivability of governments. Using again the "country paper" as a
whipping horse, these are too often conceived of as vehicles for quoting
odds on whether an incumbent regime will be in place when "the
period of this estimate" draws to a close. The trouble is that when
it is possible to say yes or no with a really high degree of assurance,
the answer is usually so obvious that no literate policy-maker really
needs to be told it; and in cases where the forecast is much more
uncertain-often, for example, in unstable and volatile countries of
the underdeveloped world-no prudent policy-maker is going to place
many chips on that particular prognostication.
I am not arguing for total abolition of this kind of estimate. It
probably has to be made, the odds have to be quoted, the conclusion
may even be informative and helpful at the time it is published. But
as a continuing guide to planning and action in the real world it has
severe limitations, and we ought to avoid exaggerating its importance.
Among other defects, it becomes obsolescent quickly, since in these
matters one wants the latest information, whether it changes a con-
clusion reached earlier or not; even the best estimate as of a given
date cannot allow for all the accidents, whimsicalities, and other
variables likely to affect the outcome in close questions of this sort;
very often what the United States does or does not do will help de-
termine the results (we normally leave this factor aside); and many
Andoeument. Declassified in Part - Sanitized Copy Approved for Release 2012/05/04 : CIA-RDP93T01132R000100020038-1
 Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
CONFIDENTIAL
Notes on Estimating
Notes on Estimating
of these situations are quite literally tossups, touch-and-go matters,
in which rational planning must be kept flexible and contingent, with
shadings rather than sharp choices in between alternatives.
We have too often focused on this kind of question as though
"probably yes" or "probably no" were the single most important answer
we could give, one on which our reputations as estimators will stand
or fall. I suspect that this particular kind of forecast is often read
by our policy-making friend with a healthier skepticism about its real
value than we ourselves show; and then the whole thing is forgotten
unless and until something happens in the benighted country,
case the estimate is dragged from the files and the prediction is either
pointed to with pride or viewed with chagrin by those who made it.
This review of the record, though interesting to professional estimators,
is not very important in a broader sense, and certainly should not be
made the touchstone of estimative reputations or a very serious cri-
terion of quality. Success or failure in this kind of spot forecasting is
too much a matter of luck and chance. It often comes closer to what
I have defined above as useful as athan ponpbldlhelp to planning and
is consequently not very action. We may have to indulge in it, but we should not confuse
ourselves about its usefulness.
Cards on the Table
One way in which estimates have grown more sophisticated deserves
special mention, strong endorsement, and even more attention in the
future: that is the laudable practice of leveling more with the reader
on questions of methodology and our own confidence in certain esti-
mates. I am not talking about the words we use for expressing degrees
of probability, whether we conclude that something is "probable," of t "unlikely," or "almost certain." These terms are essential t and uthe
trade, available to all in a well-defined glossary, accepted
by most writers and readers, and already the subject of several scholarly
articles in this journal. To gain common agreement on the meaning
of these terms has been no easy achievement, but it has now largely
been done.
What I am applauding here is rather the practice of saying more
about sources and methods, what can be expected of the evidence,
and-more importantly-what cannot. To do so is to tread on delicate
ground. There are many who feel that intelligence loses potency if it
hints at the mysteries behind its findings, and the suwbject is to be
strengths and limitations of sources or methods tends to be translated
into favorable or adverse reflections on some particular contributing
agency's present and potential importance. Anyone who has parti-
cipated in an estimate on strategic warning or concealment and de-
ception will recognize the symptoms, but they are not confined to these
subjects. Obviously there are distinct limits on how far one ought to
go in telling all. Security and the "need to know" principle obviously
impose distinct limitations. In many cases the' whole story about
sources and methods would also be tedious to the reader, and it is
often unnecessary to an honest and useful paper. But it is also often
quite relevant to giving the reader a sophisticated understanding of
what he can rightly expect and what he would be foolish to count on.
We were probably pushed or pulled into being more forthcoming
on this score than we might have volunteered on our own. Ten or
fifteen years ago intelligence did go about its business-including
estimating-with a propensity for the mysteries of the priesthood
which has since diminished. The collective "we believe," as it ap-
peared in the earlier estimates, had an aloof and oracular tone which
has undergone subtle changes in recent years. I have a feeling that
the propositions which it introduced were put forward in the fifties
with less fear of contradiction or challenge than in the period since.
Perhaps the chief reason for the change was the new style of foreign
and defense policy-making introduced by the Kennedy administration
and still carried on. Broadly speaking, two things happened simul-
taneously: intelligence was taken more seriously than ever before as
a continuing and responsible contributor to decision making; and
it had to come down from the mountain and engage more vigorously
in asserting and defending its judgments in strenuous debates before
some very tough-minded audiences. The process was marked by
much closer communication between intelligence producers and users,
each became more familiar with the other's needs and assets, and
estimates were geared more closely to practical problems in their
scheduling and subject matter. All very fine, flattering, and generally
beneficial-but it cost something.
The price was that intelligence lost something of its former mystery,
autonomy, and immunity. Oracular assertions were out, argumentation
which marshalled data was in. More and more technical experts lined
the walls at meetings on increasingly complicated questions-and we
would have been lost without them. Formal, published NIE's were
preceded, accompanied, and followed up by a great deal of less formal
particularly Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93TO1132R000100020038-1
 Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1
CONFIDENTIAL
Notes on Estimating
SECRET
25X1
paper and a lot of informal talk. Judgments could no longer be made,
published, and filed away until next year; they came under constant
scrutiny and had constantly to be defended or modified in the light of
an increasing flow of intelligence. Information about U.S. policy plans
was made available to intelligence to a degree previously unheard of,
and estimates took cognizance of this in various ways. In the pre-
vailing atmosphere, a few extreme heretics were heard to challenge
the first premise of all-that policy-making and intelligence were,
or should be, separate and distinguishable functions. The translation
of some former intelligence officers into high policy positions seemed
to add force to the radical new winds of opinion.
I suspect that some of the more drastic efforts to remodel the whole
system in the early sixties will, in time, be seen as excessive reaction
to some previous rigidities and excessive compartmentation. Intelli-
gence and policy-making are likely to remain distinctly separate func-
tions-with accompanying differences in perspective and a certain
amount of intellectual and bureaucratic tension between them, some
of it wasteful, some of it creative. But our particular professional
world will never be quite the same as it was before. Having ex-
perienced the joys and sorrows of a more direct and responsible role,
of seeing the product sold to sophisticated customers in a competitive
market, few members of the profession would willingly return to the
mysteries and immunities of an overcompartmentalized Olympus, even
if they had the option. And they don't.
Aspects of geographic
intelligence in action.
LANDSCAPE ANALYSIS
Arthur R Hall
In the highlands of North Vietnam the road south to Mu Gia Pass
threads its way upstream along a narrow, steep-sided valley. To the
left rise dog-toothed limestone peaks, to the right is a flat-topped
plateau. Dense tropical rain forest covers the entire area, almost
frustrating aerial observation. The road is carved out of the steep
hillside, for in most places there is not enough room for both road
and stream in the constricted bottom of the ravine. At the pass itself
there is (or was) a North Vietnamese army barracks. Beyond the
crest of the pass the road descends into Laos and branches eventually
into several alternate roads that run southward through the Laos
panhandle, where tracks and trails lead back east into Vietnam.
This complex of roads, part of the so-called Ho Chi Minh Trail, has
been a principal supply route for the Communist forces in South
Vietnam. In early 1966 its most vulnerable section was the stretch
of single road through this narrow valley, for at that time there was
no feasible alternative nearby. Bombs dropped accurately in the defile
could create landslides, blocking the road. Bombs had been dumped
on the road network south of the pass but had not impeded the traffic
to any significant extent. In February 1966 a geographic intelligence
officer wrote a report on the vulnerability of the valley road to Mu Gia,
and a month later, during the briefing of a policy officer, he pointed
it out again. Soon thereafter the road was bombed and the Com-
munists were forced to divert considerable manpower to reopen it.
The bombing may or may not have been the result of this particular
intelligence tip, but the sequence does illustrate the work of the
intelligence geographer.
Problems and Products
Geographic intelligence, as practiced in CIA, is concerned with
analyzing the distribution of things on the earth's surface as they
relate to the formulation and execution of U.S. policy. The surface
in question, the landscape, is in reality a zone extending upward from
Declassified in Part - Sanitized Copy Approved for Release 2012/05/04: CIA-RDP93T01132R000100020038-1