ON ANALYTIC SUCCESS AND FAILURE

• Collection: FOIA Collection
• Collection: Declassified Articles from Studies in Intelligence: The IC’s Journal for the Intelligence Professional
• Document Number (FOIA) /ESDN (CREST): 0000622830
• Document Page Count: 7
• Document Release Date: July 30, 2014
• Case Number: F-2013-02322
• Publication Date: April 1, 1993

Approved for Release: 2014/07/29 000622830TITLE: On Analytic Success and FailureAUTHOR: Martin PetersenVOLUME: 37 ISSUE: Spring YEAR: 1993Approved for Release: 2014/07/29 000622830pproved for Release: 2014/07/29 000622830?STUDIESINTELLIGENCEA collection of articles on the historical, operational, doctrinal, and theoretical aspects of intelligence.0All statements of fact, opinion or analysis expressed in Studies in Intelligence are those ofthe authors. They do not necessarily reflect official positions or views of the CentralIntelligence Agency or any other US Government entity, past or present. Nothing in thecontents should be construed as asserting or implying US Government endorsement of anarticle's factual statements and interpretations.Approved for Release: 2014/07/29 000622830Approved for Release: 2014/07/29 000622830Toward standardsstandardsOn Analytic Success and FailureMartin Petersen"Intelligence failure" are two words that analystsnever want associated with their name. But what isan intelligence failure? According to popular culture,at least as reflected by the press, our job is to knoweverything and to predict the future?and with a highdegree of specificity.' Intelligence, of course, willnever be all-knowing and all-seeing, and finishedintelligence often boils down to informed opinionabout the implications of precious few facts. This begsthe real question: what are reasonable standards?Setting reasonable standards begins with an apprecia-tion of the nature of our work and the field we playon. There are three givens in our world, and our suc-cess will always be hostage to them. First and fore-most, the information available to us is generallylimited, conflicting, and open to a variety of in-terpretations. Time pressures are great, and the issueswe are asked to address are almost always multi-faceted and evolving. The yes or no question is rare;we deal in trends and outlooks. The smoking gun ofan intelligence failure?that piece of reporting thatpoints clearly to the truth?is generally visible onlyin hindsight, and sometimes it is not so clear to us asit is to our critics.'The policy environment is frequently charged, and attimes it can be hostile. Not all agendas are above-board, and not everyone is interested in an objectiveassessment. For some policymakers, the truth?to thedegree it can be established?is as likely to makethem angry as it is to make them free. Thus, intelli-gence and the intelligence analyst may be seen as partof the problem rather than as part of the solution.Finally, a key piece of the puzzle often is what theUS Government is saying and doing in private, andthat can be the hardest piece to obtain. The task iseven more difficult when multiple policy agencies?each with its own agenda?are involved.5Setting a StandardBut what standard should we apply to ourselves? Ifwe cannot predict the future, is it enough just topresent the possibilities? Does a policymaker have toact on a piece of intelligence in order for us to besuccessful? Does the action itself have to be suc-cessful? Can there be an intelligence success and apolicy failure?I believe that in asking the question in this way weimply that intelligence, like a game, has a clear out-come and that we can keep score of wins and losses.The nature of our work indicates otherwise. Rather,we should define our standards in terms of the or-ganizational purpose of the Directorate of Intelli-gence (DI)?support to the policymaker. I believetherefore the difference between success and failureis whether or not we put the policymaker in a posi-tion to make the best-informed decision possibleunder the circumstances: we have to ensure thatpolicymakers are given the dynamics of a situationor issue, the risks to US interests, the options theyand their adversaries have, and the consequences ofvarious choices.To those who might say that this standard is toolimited and too subjective, I can only answer that,however subjective it may be, meeting it is not easy.'Moreover, it does have three concrete elements. We,have done our job well if, and only if:? We have answered the yes or no questions correctlyand have gotten the trend lines right. Yes or noquestions are comparatively rare: Will X invade Y?Who will win the election? More frequently, we areasked to assess the outlook for, or implications of,some development. In such cases, we have to beable to say that we foresaw the trend, even if wewere unable to measure it precisely: growth incountry X's exports over the decade; a decline incountry B's military capabilities.Approved for Release: 2014/07/29 000622830-Secret-Approved for Release: 2014/07/29 000622830? We have illuminated for the policymaker theforces at work and correctly estimated their im-portance, strength, and trends. It is not enough tojust report or even add context. We have to striveto put events in broader perspective and to pro-vide a framework that allows the policymaker toincorporate new information. We also have to beable to say that we correctly saw the threats to USinterests and laid out clearly options and conse-quences. We have to state the degree of confi-dence we have in our judgments and what factorscan change them.? We have marketed our product in a way thatallows it to play a role in the policy process. Thisrequires us to study our customers, to tailor ourproduct so that they can use it easily, and to makethem aware of the full range of our capabilities.?An intelligence success cannot be declared on thebasis of a single piece of finished intelligence.Rather, success on an issue or set of issues is deter-mined over time: did we adjust our analysis as cir-cumstances changed; did we effectively target andtailor our product to the shifting needs of our con-sumers; and were we timely? 5 If success is alwayssomehow conditional, failure can be immediate andabsolute. It may be a single piece that is poorlycrafted or the absence of a piece that should havebeen written. In either case, the stain is indelible,?and no amount of good work before or after com-pletely removes it.Knowing What You BelieveSetting a standard is one thing, but meeting it isanother. I believe the key to getting it right is know-ing what you believe and questioning what you be-lieve. Knowing what you believe involves identify-ing, and separating in your mind, the facts, what isnot known, your assumptions about how thingswork, and where the wheels can come off.As a rule, the facts are not that hard to identify, ifonly because there are generally so few of them.And most analysts are good at identifying what it isthey do not know that is critical to the issue at hand.We are also adept at dissecting the facts for clues tothe unknown. These things are not at the root ofmost flawed analysis.AnalyticThe problem frequently is our assumptions abouthow things work. Any analyst who has spent time onan issue quickly begins to form assumptions aboutits dynamics. Each of us builds out of assumptionsand opinions a subliminal model about some of thehardest things to know: motive, character, culture,who has influence, how decisions get made, and theinterrelationship of events. This barely conscious setof beliefs is the product of everything from earlier,accurate analysis to impressions gleaned from snip-pets we have read. These assumptions and opinionscan be, and often are, contradictory.Knowing what you believe also involves knowingwhere things can go wrong. Avoiding surprise isoften a function of knowing what is out there thatcan radically change our assessment. This is morethan an appreciation for the things we do not know;it is an awareness of the potential for the suddenso-called low-probability, high-impact event. We alsoneed to be attuned to how much our bottom-line judg-ment hinges on a particular variable or assumption.Three Examples(b)(1)(b)(3)(n)6Approved for Release: 2014/07/29 000622830Approved for Release: 2014/07/29 000622830Analytic(b)(1)(b)(3)(n)Living With AssumptionsBecause we cannot do analysis without making someassumptions, we have to reduce our vulnerability byincreasing our awareness of our assumptions. Analystshave to ask themselves what they believe about thepriorities, motives, and interrelations involved in theissue at hand. The goal is to expose contradictorynotions and to identify the most questionable as-sumptions and the most volatile factors in the ana-lytic equation. This process sharpens our thinking,thereby making our analysis more persuasive. And,most important, it reduces the impact of unspoken,half-realized assumptions about new evidence.Thinking through one's assumptions is a tall orderunder the best of circumstances. First, you have toset aside the time to do so. Time is the analyst'smost precious commodity, but minutes invested inthinking about your account on the commute to andfrom the office or while taking a shower are minuteswell spent.Second, you have to limit your thinking to discreteproblems or issues. You are not going to be able tothink through all that you believe about China. You7can, however, think through what you believe aboutChina's relations with Taiwan or the Khmer Rouge'smilitary capabilities or Japan's approach to GATTnegotiations.Four TrapsBy itself, thinking through assumptions will not en-sure accurate analysis. There are many causes offlawed analysis, but the four most common are easyto avoid.Straight Line Projections. One of the easiest thingsto do as analyst is to take the present situation ortrend and project it. The press and TV commentatorsdo this frequently.6 Straight line projections occurwhen analysts fail to identify and think through allthe factors driving a problem. The analyst may be inover his or her head?or he or she may be guilty ofsloppy thinking. In either case, the result is an im-plicit conclusion that what is true today will be truetomorrow.This trap has two variants. One is to focus on a sin-gle variable and neglect others: polls, numericalstrength, and rates of growth. The other is to seek aprecedent in the past and transfer it to the future: thegovernment will survive this crisis because it hassurvived similar crises in the past.Sometimes the present situation does extend into thefuture, trends continue (that is why they are calledtrends), and historical precedents are valid clues tothe future. The point is not that such things are al-ways wrong. Rather, such things should ring a bellthat causes us to examine them closely before put-ting them in print.Seduced by Logic. Much has been written about themirror phenomenon in analysis, the assumption thatothers will behave as we would. We need to remindourselves that risk, goals, honorable outcome, andcost/benefit trade-offs are all culture-bound conceptsthat make actions that seem illogical or counter-productive to us acceptable to others. But I believethis is part of a larger analytic trap?being seducedby logic, our principal tool. We forget sometimes theimportance of emotion and the fact that all leadersmake bad, even stupid, decisions.Approved for Release: 2014/07/29 C00622830-grecTerApproved for Release: 2014/07/29 000622830Sometimes our analytic ability works against us inanother way. Analysts by disposition as well as bytrade, we divine hidden meaning and uncover pat-terns in events, even when they are not there. Givenenough time, a good analyst can fit any two piecesof information together. It is important to rememberthat many things are just what they appear to be, andthat there is such a thing as coincidence. SeeOccam's razor.Leap to Judgment. Because we see making judg-ments as one of the things that distinguishes us fromthe press and academics, we sometimes rush intothem, even when it would be wiser and more helpfulto the policymaker to reserve our opinion on howthings will turn out. Coups attempts, elections, andnegotiations are classic cases for caution, especiallyat the onset. The key variable in any fast-moving,evolving situation is that it is fast moving and evolv-ing, and we generally are trying to play catch-up.We are not forsaking our responsibilities if we with-hold a judgment on the outcome when things are toouncertain and our sources too limited to support aninformed opinion. Indeed, we may mislead our con-sumers if we try to do so in such circumstances. Webest serve the policymakers if we inform them aboutthe factors at work, the signposts to watch for, andthe implications of potential outcomes.The US Factor What Washington does or does notdo is often a key variable. It is essential to cultivatepolicymakers, not only to serve them better but alsoto avoid being surprised by our own government.Our analysis is much more likely to be flawed if wedo not know everything that is causing the other sideto react. We also have to remember that what theothers think Washington will do may be more impor-tant than what Washington actually does.Perceptions, after all, drive one's behavior.The coordination and review process is our bestdefense against these traps, as well as the other thingswe need to keep in mind. When they work as intend-ed, these two processes sharpen judgments and helpus articulate underlying assumptions. They force usto test our views and incorporate the expertise ofothers. Managers and analysts alike have a large stakein making review and coordination as intellectuallyrigorous (as opposed to mind-numbing contentious)as possible.SccrctAnalyticSome RemindersThere is more to successful analysis than avoid-ing analytical traps. We would do well to keepin mind:? How we say things can be as important aswhat we say. No one likes to be told, "I toldyou so," so be mindful of tone. Our access tothe policymakers is at their sufferance, and wewill not be invited back if we insist on puttingour thumb in their eye at every opportunity.? Cynicism is the DI disease. Some policiesthat DIers believe will not work ultimatelydo. We need to remember that the half-emptyglass is also half full and to look at what isgoing right, as well as what is going wrong.? The danger of casual causal judgment.Clauses that start with "if" or "given" aredangerous when carried over from one pieceto the next. They postulate a causal relation-ship that may no longer be that case, and theycan easily become ,unexamined assumptions.Every "if" and "given" should make uspause.? Numbers are meaningless, unless the readerknows how they were derived. Moreover, nomatter how sound our methodology, there is agood chance they are off by some factor.Numbers suggest a precision that intelligencerarely possesses.? Criticism is a given in our business. Indeed,it is necessary for success. We can only hopethat it is honest and informed, which it maybe more times than we are willing to admit.? Our own emotions and biases. Some con-sumers will see our work as a report card ontheir performance. We want to make sure thatthat is not our intent. We also have to takecare that pressure "to get on the team" doesnot lead us to an analytic version of Newton'sthird law, an equal and opposite reaction.8Approved for Release: 2014/07/29 000622830Approved for Release: 2014/07/29 000622830AnalyticA Final WordWe are not primarily economists, historians, or polit-ical scientists. We are intelligence analysts. It is im-portant that each of us makes the time to think aboutour profession?what it is and how to do it better. Ifwe do not, we will never be as good at it as wecould be and as the nation needs us to be.NOTES1. During the Gulf war, some critics took us to taskbecause we could not state exactly how manyScuds Iraq had and where each was. Others havedeclared our work on the Soviet Union, EasternEurope, and any number of other places a failurebecause we did not predict a specific event suffi-ciently in advance or because we hedged ourwarning. Many of these criticisms start with apolicy preference and an assumption that theAgency has one, too.2. In Romeo and Juliet, Shakespeare captured theanalyst's point of view best: "Seal up the mouthof outrage for a while, til we can clear these am-biguities."3. As a quick review of the DI's Product EvaluationStaff's (PES) studies will reveal, a vigorous PESis essential to maintaining standards, howeverthey are defined. It is in the best position tobring the dispassionate, policy-neutral analysisessential to the evaluation of our work. Unlikethe press, PES also brings an understanding ofthe directorate and its mission on the task.9Secia4. I sometimes think that the role of the analyst isakin to getting policymakers to eat their vegeta-bles. We select only the finest vegetables, pre-pare them carefully, and serve them attractively.The problem is that policymakers also have a lotof meat on their plates, and they do not alwayseat their vegetables. Intelligence is good for you,but it is not always appetizing. Thus it is thatyou can have an intelligence success and a policyfailure.5. Personal success is something different. To mymind, it is more than getting it consistently right.As individuals, and as an institution, we oughtto strive to so establish our reputation for objec-tive, insightful analysis that our consumersacknowledge?implicitly if not explicitly?thatthey cannot have an informed opinion unlessthey know what we think. They may ultimatelyreject our views, but they nonetheless feel ob-ligated to check with us, even when they knowthey are not going to like what they will hear.6. Meg Greenfield in the 9 November 1992 issue ofNewsweek mused on why the press was so oftensurprised and wrong about developments in theelections. She argued that the press was swampedwith data but failed to look beyond them. I wouldadd that the press also fell victim to straight lineprojections. The coverage of the primaries, inparticular, is instructive in this regard.This article is classifiedApproved for Release: 2014/07/29 000622830-Sesr-44--