TRADECRAFT REVIEW CONTINUOUS LEARNING IN THE DI: THE MAY 2004 REVIEW OF ANALYTIC TRADECRAFT FUNDAMENTALS

• Document Type: FOIA
• Collection: FOIA Collection
• Document Number (FOIA) /ESDN (CREST): 01179397
• Release Decision: RIPPUB
• Original Classification: U
• Document Page Count: 16
• Document Creation Date: March 9, 2023
• Document Release Date: September 22, 2021
• Case Number: F-2021-00447
• Publication Date: May 1, 2004

Approved for Release: 2021/09/21 C01179397
Tradecraft Review
Continuous Learning in the DI:
The May 2004 Review of
Analytic Tradecraft Fundamentals
August 2004
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
SCOPE NOTE
During 10-12 May, 2004, I convened the analyst and manager corps of the
Directorate of Intelligence for a mandatory, half-day "stand-down" session on
analytic tradecraft. As indicated in my opening remarks, the state of analysis in the
Directorate is strong but with room for improvement. The goal in asking every officer
to come "off line" for a half-day was therefore two-fold: to underscore the importance
of the Directorate's analytic mission and to provide a collective opportunity to learn
and improve by reviewing both the requirements for sound analytic practices and
lessons learned from poor tradecraft.
In these sessions, jointly organized by the Sherman Kent School and the Office of
Policy Support, presentations by senior DI officers addressed the trademarks of
quality analysis and offered exemplars of strong and weak practices. Many thanks
to those involved for their illuminating and candid presentations. In order to capture
the lessons identified and make them readily accessible as an ongoing reference
for DI officer e the Sherman Kent School
has distilled the key concepts from these presentations for
both hardcopy and web-based dissemination.
No topic could be more appropriate
than this booklet, prepared by a Senior Analytic Service analyst. It is not an
exhaustive rendering of all that was covered during the refresher sessions but
provides a synthesis that covers the essential elements. It will assist analysts in
continuous tradecraft learning and in attaining the highest standards of our
shared profession.
Deputy Director for Intelligence
In
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
MESSAGE FROM THE DDI ON THE MAY 2004 "STAND-DOWN"
In holding these sessions, we are following through on a commitment I made in my
February speech on the State of Analysis in the Directorate to hold a Tradecraft
Refresher Course for all DI analysts and managers. As I said in February,
intelligence analysis is our profession and it is our craft. As practitioners of that
craft, it is up to us�not others�to ensure that we are continuously learning and
improving and that we do not lose sight of the fundamentals of our work.
Our generation of analysts and managers has an especially heavy burden of
responsibility. First, we face an exponentially growing volume of information
that must be read, digested, and analyzed. Second, the time it takes to send a
requirement to the field, mount collection, disseminate and write reports, and deliver
finished intelligence analysis has been radically compressed. And third and most
importantly, we now have unprecedented access to the President and his most
senior national security advisers. This means we must meet a higher standard than
ever before�a standard set first and foremost by our own professional commitment
to excellence, but also one expected and demanded by the Administration, the
Congress, and the American public.
I called for a "stand-down" because, quite simply, it is overdue and it is needed. In
my February speech, I characterized the Directorate's "state of analysis" as strong,
with room for improvement. And that is why these sessions are being held�to
improve. We will not get better without trying to get better�and without focusing on
our mistakes and learning from them.
Our mission and our corporate responsibility must be to give the policymaker
the best-crafted analytical product possible. And, when that product leaves CIA,
it represents and reflects on every one of us. One key message of these "stand-
down" sessions is that no analyst and no analytic manager can be either a
passive recipient of intelligence or a disengaged producer of intelligence. We have
to cultivate a passion for this work, and every one of us must accept personal
accountability for the quality of his or her work. Moreover, this is not something to
be feared; it is what gives us our greatest professional satisfaction�taking pride
in our work.
In these sessions, analysts and managers will hear some of the challenges we face
in our work that go beyond the fundamentals of tradecraft.
We have launched a number of initiatives to ensure that those
obstacles are overcome. But our primary focus now is the importance of continuous
improvement. Our goal is not to produce analysis that our customers agree with or
necessarily even like. Our job is to be objective, cogent, and provide value-added
to the policy debate over important national security issues. When the Directorate
is delivering a difficult or unwelcome message, we need to be sure there is a solid
foundation of credibility with our consumers so they will hear the message. Products
based on strong tradecraft will stand up well to the scrutiny of skeptical readers.
Deputy Director for Intelligence
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
CONTINUOUS LEARNING IN
THE DI: MAY 2004 REVIEW
OF ANALYTIC TRADECRAFT
FUNDAMENTALS
"This is a difficult profession.You are
asked to inform the debate on some
of the country's most important policy
judgments, usually based on limited
and conflicting information. We only get
pieces of the puzzle. It is like trying to
do a 1,000-piece jigsaw puzzle with only
200 pieces. And, as a kicker, you do not
get to see the lid of the box to tell you
what it is supposed to look like."
--DD!, February 2004
"When we make mistakes we need
to learn from them collectively as
a Directorate. We will enhance our
expertise and broaden our point of view
by reaching out to others, employing
contrarian analyses and, perhaps most
importantly, by expanding the diversity of
our workforce."
--DD!, May 2004
A Time for Reflection on the
Quality of Analysis
The foregoing observations highlight
the significant responsibilities and
challenges that the Directorate of
Intelligence faces in the period ahead
for providing insightful analysis to our
nation's most senior leaders. Seldom
has the Directorate had such access to
or such impact on the highest levels of
our government. As the DCI said earlier
this year, we will never be "all right or
all wrong!' When problems occur, it is
important for DI managers and analysts
to reflect on what we have done right
and what we can do better. A period
of self-examination is important now
and can serve as one of the best ways
of constantly working to improve the
"quality of our analysis." Like any catchall
phrase, this concept is shorthand for
a lot of different things. In its broadest
sense, it requires:
� Sophisticated, value-added
analysis on key national security
issues; it is not "gisting" cables
but rather putting an issue into a
bigger context, providing nuance
and details that will drive a
critical foreign policy decision
or discussion.
� World-class expertise on an
issue; it is not just opinion but
insight, informed by a career-
long development of expertise.
� Rigorous re-examination of
analytical judgments and
assumptions; it is not holding
onto an analytical model or line
of reasoning just because it has
proven to be reliable in the past.
Translating these broad objectives
into every day practice means
that DI analysis must contain the
following attributes:
� Precision of language.
� Clear articulation of judgments
and levels of confidence.
U. . it is
important for
Dl managers
and analysts to
reflect on what
we have done
right and what
we can do better.
A period of self-
examination is
important. ."
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
"Continuous
learning will
require new
tradecraft review
processes. .."
"The current
focus on Iraq
should not
mislead analysts
... analytical
controversy
could well
visit others in
the future."
2
� Understanding and explaining
the quality of intelligence
sources and key gaps.
� Examination of alternative
analytical possibilities
or outcomes.
Key Principles to Guide
Dl Analysis
Maintaining high-quality analysis
demands that the DI understand and
put into practice daily some important
principles that should shape everything
we do. Among these are: continuous
learning, personal accountability,
corporate responsibility, analytical
integrity, precision and accuracy, and
insightful analysis.
Continuous Learning. The DDI has
noted that any organization that does
not learn from its mistakes is destined
to repeat them. While the DI has had
numerous successes, it has also made
mistakes and will make mistakes again.
When we do, we will admit them, study
them and learn from them. As a senior
manager remarked, "the East India
Company�a global power in the 18'h
century, exists today but only in the
history books. If we do not constantly
reassess our tradecraft, we too could
go the way of the East India Company."
As practitioners of that craft it is up to
us�not others�to ensure that we are
continuously learning and improving. We
must not lose sight of the fundamentals
of our work�the key elements of our
tradecraft. We will not get better without
trying to get better.
Continuous learning will require new
tradecraft review processes to develop
understanding of where our analysis
can improve and where our analytical
workforce can be assisted. The Iraq
WMD Review Group, established
in 2003, was one mechanism for
examining critical judgments and
incorporating lessons learned into
the Directorate's trainina proarams.
ill allow us to examine
our tradecraft more systematically,
and turn those findings into
new tradecraft training objectives that
will strengthen the Directorate. Already,
the DI has initiated an "Advanced
Analyst Training Program" that will be
offered in Fall 2004 and will draw heavily
from our lessons learned to bolster
best practices.
Personal Accountability. The current
focus on Iraq should not mislead
analysts to think they are not personally
affected by tradecraft mistakes. Analysts
who worked on the Soviet Union or
Central America in the 1980s,
India-Pakistan nuclear issues in the
mid-1990s, and terrorism since 9/11
know that analytical controversy could
well visit others in the future. Rigorous
testing of assumptions and revisiting
of past judgments can prepare each
analyst for the day when his or her issue
becomes the focus of policymakers'
scrutiny and challenge. An analyst
prepared for such controversy is an
analyst who has identified key analytical
assumptions, has a clear understanding
of the strengths and weaknesses of
the reporting, and can articulate how
judgments were reached
A dose of
analytical humility�that is, graciously
accepting the comments and criticisms
from colleagues�is a professional
requirement. Analytical arrogance in
the coordination process has no place
in the DI and is dangerous as it leads
to dismissing valuable information and
insights into our work.
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Senior mentoring is now
a key skill to insure the DI's continuous
learning and ultimate survival as a
world-class analytical service.
The pressure to "get it right" should
not lead analysts to shy away from
reaching firm judgments. Mistakes
will happen; they should teach analysts
how to improve, not how to avoid
tough calls. Analysts must take
responsibility for getting the facts
straight. Relying on other analysts or
reviewers to catch errors is a recipe
for failure and rebuke by senior
policymakers who notice them.
Corporate Responsibility. Our
judgments are not our own but
rather are the Agency's. We have the
obligation to coordinate our products
fully across the DI and increasingly
across the Intelligence Community.
Just as no analyst can assume his
or her issue is safe from political
controversy or analytical challenge,
the entire Directorate must share in
the responsibility for each and every
analyst's judgments. That means an
individual analyst or issue team's work
cannot be done in a vacuum. There
must be dialogue among analysts
and across issue teams to maximize
the talents of our diverse and
sophisticated experts. There must be
involvement of senior managers as well
as the Senior Analytical Service in this
analytical dialogue, to take responsibility
for nurturing newer analysts and
developing their skills as self-critics
and skeptics. In sum, colleagues must
seriously read and comment on each
other's drafts; managers must rigorously
develop and challenge analysts; senior
Agency managers from the DCI, DDCI,
and DDI on down must be prepared
to stand behind our judgments to the
President, the Congress, and the
American people.
Analytical Integrity. The Directorate
exists for one reason�to analyze all-
source information in as even-handed
a fashion as is humanly possible and to
provide policymakers with judgments
not effected by any policy agenda. The
best minds and information in the world
will be useless if DI managers and
analysts are not able to present frank
assessments to powerful leaders who
may not agree with our judgments. It
sounds easy but it is not. Policymakers,
who rightfully have a policy agenda,
can make it hard for DI analysts to bring
them bad news. They can demand that
we re-examine the evidence, delineate
our analytical approach, or look for
"We have the
obligation to
coordinate our
products fully
across the Di . . ."
3
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
"... we must
be prepared
to explain our
reasoning
and challenge
ourselves to
ensure we are
not missing
something."
"Credibility
with senior
policymakers
begins with
getting the
facts right."
". . the
Directorate has
to "ask the right
question" if it is
to produce useful
and insightful
analysis."
4
alternative scenarios. As a Directorate,
we must be prepared to explain our
reasoning and challenge ourselves to
ensure we are not missing something.
However, at the end of the day, we must
be prepared to stick to our judgments
when they are sound or admit to errors
and correct them as appropriate.
Analytical integrity is not a new
challenge but rather an enduring one.
Richard Helms's memoirs recount that
the Special Assistant for Vietnamese
Affairs in the late 1960s had worked
himself into exhaustion while briefing
unpleasant assessments to President
Lyndon Johnson. Vice President
Humphrey later praised this senior
officer for having "served the President
well by holding your ground and telling
us about the situation as you saw it
in Vietnam. It was brutally frank and
forthright analysis:' And today, the
challenge remains the same.
also to be open to the
idea that we could be wrong and will
admit our mistakes." Doing so makes
policymakers respect the analyst and
the institution. It is what demonstrates
the Directorate's integrity and validates
its mission.
Precision and Accuracy. Credibility
with senior policymakers begins with
getting the facts right. There is a need
for absolute accuracy and precision
in everything the Directorate writes
and briefs. Failing to do that not only
leads to incorrect judgments, but also
seriously undercuts our credibility with
our readers. Misquoting
will be noticed and,
if not corrected, foster a belief among
policymakers that they cannot depend
on CIA fact-checking.
Choosing our words carefully so that
the reader grasps the exact meaning
intended is equally important. Analysts
should not confuse elegant prose
for precision and clarity. Caveats or
qualifications lost in the editorial review
process can distort the analyst's
intended message and mislead the
reader. Explaining that some nuance
had been removed by a night editor is
not likely to mollify an annoyed reader
the day after. Analysts must be prepared
to go the extra mile in articulating the
importance of caveats and context to
editors who do not know the issues as
well as the experts, and they must be
willing to stand by their analysis.
Insightful Analysis. The mission of
preparing insightful analysis is what
distinguishes the Directorate from any
other organization to which senior US
policymakers could go for information
and analysis. The New York Times,
The Economist, or Foreign Affairs can
write about whatever they decide will be
controversial, appealing, or profitable.
The CIA, however, is in the business of
telling a select, demanding, and decisive
group of policymakers what it does not
already know, will find useful, and will
find credible. Simply put, the Directorate
has to "ask the right question" if it is to
produce useful and insightful analysis.
Providing obvious conclusions will cause
our work to be discounted and make our
jobs harder the next day. In order to ask
the right intelligence question, analysts
need to know the audience, the policy
agenda, and the state of play within
the interagency process.
� Analysis that is irrelevant
or ignored comes from
targeting the wrong audience
or misunderstanding what
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
the audience's level of
understanding and policy
needs are.
� Insightful or "value-added"
analysis comes from knowing
what the policymaker already
knows, what he cannot yet
know, and what he would find
useful in understanding a
complex story.
Ultimately, the target audience�the
President, an Assistant Secretary, or a
military commander�will be the judge
of whether CIA analysis has been useful
or credible. CIA team chiefs, publication
editors, or IC colleagues will review
this analysis, but they are NOT the
audience. Persuading the policymaker
that you are right is your job, not the
policymaker's responsibility. This
challenge warrants the additional
elaboration that follows.
Crafting Insightful,
Persuasive Analysis
CIA analysts must know what the
audience knows
Understanding
where the policymaker is in the decision-
making process will help determine the
kind of analysis that is needed:
A Policymaker's Top Ten
for Analysts
--Get me information and analysis
on time, so I can make an informed
decision. If it is late, I will decide
without it.
--Tell me something I do not already
know. Know what I know and what
I need.
--Give me fact-based analysis, and let
others write op-ed pieces. You need
to organize facts in a way that reveals
patterns, notes change, and has insight.
--What are all the angles on the issue?
What did I not ask for that I would need
to understand the issue?
--Have you convinced me that you
have examined all the alternatives? You
should know all the reasons for doubting
your conclusions.
--Is your argumentation transparent?
You must tell me what you know, where
your data come from, and whether you
are confident or not. Tell me if you are
changing your line of argument and why.
--What should worry me, but also what
opportunities are there? If you over-
warn, I will ignore you to our peril.
"Persuading the
policymaker that
you are right is
your job . ."
"Understanding
where the
policymaker is
in the decision-
making process
will help. ."
5
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
". . Caveats
are essential
and help
policymakers
gauge how much
trust to put in our
analysis."
". . our analysis
becomes more
insighfful when
It goes beyond
answering the
Immediate
question and
examines related
issues . ."
(b)(2)
(b)(3)
--Keep me honest with any bad news
but you will have to persuade me. If I
misunderstand an issue, figure out a way
to make me listen.
--Check for your own policy bias and
be open to your own fallibility. Do not let
your personal views color your analysis
or blind you to contrary information.
--If you are wrong tell me so. If you do, I
am more likely to respect you and have
confidence in you than if you do not.
Useful information can be conveyed
in many forms. It may be a fact
Dr
a graphic that displays a complex
Drocess on a
single page. Sometimes the most
useful information is not a secret, but
rather an unclassified description
Once the analyst understands the
audience's needs, the challenge is
to present facts and argumentation
in a way that is clear, accurate, and
compelling. Analysis that has no clear
focus, is missing details, and has not
carefully weighed the evidence or
assessed information gaps cannot be
fixed by editors. The goal of sending
forward well-supported argumentation
does not mean, however, overstating a
case or ignoring problems of sourcing or
contradictory reporting.
Caveats are essential and help
policymakers gauge how much trust to
put in our analysis and how much risk
they run in basing policies on it. It is
the analyst's responsibility -n knnw thp
certainty of the information
Finally, our analysis becomes more
insightful when it goes beyond
answering the immediate question and
examines related issues, which the
policymaker must also be aware of in
order to appreciate the key conclusions
of our analysis. That is what will
distinguish insightful analysis from the
obvious. Answering the policymaker's
first question may also suggest other
areas where the analyst should provide
further details or context to understand
the issue
even if it is only a mental
exercise mat captures the broader
context in which the immediate issue sits
and connects it to other issues important
to the target audience.
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
Directorate
must earn its
reputation
everyday. . ."
"We are paid to
make the tough
calls."
These simple, yet important, steps to
improved tradecraft are a beginning
for a revitalized DI commitment to
personal and corporate responsibility
for producing high quality, insightful
analysis. As the Directorate enters
its second half-century in supporting
American foreign and security policies,
we must rededicate ourselves to the
notion that the Directorate must earn
its reputation every day as the world's
best analytical service, and that
requires continuous learning from
past efforts.
Final Thoughts from the DDI
The demands and expectations on us
are enormous, but we must recognize
that they are a result of our success.
What we do matters. There are some
who say we should not try to make the
tough calls. However, we cannot avoid
making a judgment because we are
afraid we will be wrong. We are paid
to make the tough calls. That is what a
professional intelligence analyst does.
Most often, this means relying upon
our expertise, reporting, and tradecraft
to come down on one side or the other
of an argument; at times, however,
this may mean acknowledging that
9
Approved for Release: 2021/09/21 C01179397
Approved for Release: 2021/09/21 C01179397
"... we must
give the
policymaker full
transparency into
our confidence in
the judgments."
10
the reporting could support several
hypotheses. But we must remember that
when we "call it as we see it," we must
give the policymaker full transparency
into our confidence in the judgments. We
must be clear, tell them what we know,
what we do not know, what our judgment
is, and occasionally�when there is
no firm reporting�tell them what our
experience suggests to us. As I said in
February, "if you apply good tradecraft�
and do your job to the fullest�you can
be sure that I will defend your analysis
before any critic."
Approved for Release: 2021/09/21 C01179397