SENIOR REVIEW PANEL EXAMINATION OF SEVERAL ESTIMATIVE PRODUCTS FROM THE FIRST HALVES OF 1982 AND 1983

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP98S00099R000400830001-4
• Release Decision: RIPPUB
• Original Classification: T
• Document Page Count: 30
• Document Creation Date: December 27, 2016
• Document Release Date: May 31, 2012
• Sequence Number: 1
• Publication Date: September 10, 1984
• Content Type: MEMO

25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
R
Next 1 Page(s) In Document Denied
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
--"..
Senior Review Panel
THE DIRECTOR OF CENTRAL INTEWGENCE
WASHINGTON, D.C. 20505
NIC-05182-84
10 September 1984
MEMORANDUM FOR: Chairman, National Intelligence Council
SUBJECT:
Senior Review Panel Examination of Several
Estimative Products from the First Halves of
1982 and 1983
Purpose of Study
1. In response to your request, the Senior Review Panel
undertook a sampling of estimative products published by the
National Intelligence Council during the first six months of 1982
and the equivalent period of 1983. You asked that the Panel
address, in particular, two questions:
.1???
How well do the projections stand up in the light of
subsequent events?
How useful, in our view, were the papers to the
policymaking community?
2. The field of inquiry was thus the four NIEs and 17 SNIEs
published in the first half of 1982 and the 12 NIEs and 13 SNIEs
published in the first half of 1983.
Main Findings
3. Annexes to this memorandum contain our specific
comments. Our main summary conclusions are:
a. 1982-1983 estimative production was of extremely
high quality in forecast validity and policymaker utility.
Of the 46 NIEs and SNIEs published in the two semestral
periods, 45 met--or are meeting--acceptably severe standards
of estimative dependability.
TOP RET
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 201_2/05/31 : CIA-RDP98500099R000400830001-4
b. There was only one failed estimate in the two half-
years--SNIE 34/36.2-82, Implications of Iran's Victory Over
Iraq, 8 June 1982--on which we comment further at Tab E.
c. The high estimative performance of 1982 was
maintained in 1983 with an appreciable increase in quantity.
d. For the most part, 1982 and 1983 production would--
in our experience--have met most policymaker needs in
timeliness, policy issue relevance, analytical quality,
readability, and estimative judgment. Actual use made is,
perhaps, a separate question and one beyond the scope of our
study. Overall, as policymaking support, we would regard
two-thirds to three-quarters of the papers as "Superior to
Outstanding," with the percentage rising as one moves from
1982 to 1983.
e. Length of preparation/coordination time bore little,
if any, relation to quality of finished product. Granted
special response factors to high priority estimative
requests, there seemed in these two years an almost inverse
relation: the quicker the response, the better the product.
f. Improvements in quality are still--and probably
always will be--possible. We cite a number of these in the
concluding section to this memorandum.
Section of the Sample
4. As indicated, our summary findings derive from a review
of all the NIEs/SNIEs published in the two periods. We selected,
as requested, a smaller sample for more intensive study on the
basis of the following criteria: (a) the projective time frame
of each had to be sufficiently short-term to permit a valid
comparison with our perceptions of subsequent "ground truth"
twenty-four and twelve months later; (b) the ultimate sample
needed to be broadly representative of the diversity of NIC
output and large enough to be statistically significant, taking
into account the increased production of 1983 over 1982; and
(c) subsequent Community and Agency analyses were available to
test validity half-lives.
5. The sample which emerged comprised the following 12
NIEs/SNIEs. Our individual comments are attached as case studies
at the indicated Tabs:
1 January 1982 - 30 June 1982
Tab A
2
,TOP SECRET
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98S000gqpnnnennwnnt-11 A
25X1
TArk ......
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
?
2
Methodology
6. The estimates themselves were of course our primary
source. Additionally, we requested computer runs of Agency
holdings of finished and unfinished intelligence bearing on
subsequent events and trends. From these we selected for
detailed study what we believed pertinent to our inquiry,
numbering from two to three dozen for some estimates to several
hundred for others. All Panel members reviewed all retrieved
documents. In some cases we found it necessary, as our work
progressed, to request papers published prior to the estimates.
Finally, our judgments necessarily reflect our own recollections
and impressions of the passage of events since the issuance of
3
TOP SECRET
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
5X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
?
the estimates forming the sample. We have tried to discipline
this subjectivity, as indicated, by the production record.
Concluding Observations
7. As indicated above, estimative production in the two
periods reviewed seemed to us of an impressively high quality.
Quantity also increased. With few exceptions the estimates were
timely. They addressed issues of policy importance. They would
have been, again with a few exceptions, of value to
decionmakers. And the majority provided predictions which were
perceptive and which have been validated by subsequent events.
8. In our view, it is also true that length remains a
problem, and that too much of the burden of concision may now be
carried by the Key Judgments. Within generally excellent
estimates, individual findings are at times more discursive and
fuzzy than rigorous thinking would abide. There are persistent
tendencies to hedge bets without giving the reader a feel for the
odds on--often implicit or casually developed--alternative
outcomes. And Community practice regarding dissenting footnotes
on major issues may need a further look. In particular, their
rationale appears to need fuller development in the text--perhaps
with an opportunity for a one-round rejoinder--and additional
highlighting in the Key Judgments. At the end, clarity of aim,
brevity of expression, and a decent respect for man's
imponderables remain the cardinal virtues.
9. The distribution of this report has been limited as
noted.
Attachments:
Tabs A .through L
cc: DCI
DDCI
William Leonhart
4
TOP SECRET
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
SUBJECT: Senior Review Panel Examination of Several Estimative
Products from the First Halves of 1982 and 1983
Distribution
SRP/DCI:tb
Cy
1
- Chairman, NIC (w/atts)
2
- DCI (w/atts)
3
- DDCI (w/atts)
4
- ER (w/atts)
5
- DDI Registry (w/atts)
6
- SRP File (w/atts)
7
- SRP Chrono (w/atts)
8
- SRP
(Amb. Leonhart
w/atts)
9
- SRP
w/atts)
10
- SRP
w/atts)
11
- SRP
w/atts)
12 -
VC/NIC
(Herb Meyer)
(w/atts)
5
TOP SECRET
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
. Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
. Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
R
Next 1 Page(s) In Document Denied
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
. Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
R
Next 3 Page(s) In Document Denied
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
. Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
R
Next 2 Page(s) In Document Denied
Declassified in Part - Sanitized Copy Approved for Release 2012/05/31 : CIA-RDP98500099R000400830001-4
25X1