COLLECTION SYSTEMS INTERIM REPORT

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP86B00269R001300050001-1
• Release Decision: RIFPUB
• Original Classification: U
• Document Page Count: 92
• Document Creation Date: December 15, 2016
• Document Release Date: August 15, 2003
• Sequence Number: 1
• Publication Date: August 1, 1983
• Content Type: REPORT

Approved For u-N'-
bIRDIB6BOO269ROOl 300050001-1
rtwMf
R P-4032-83
COPY / OF
INTERIM REPORT
HUGHES
HUGHES AIRCRAFT COMPANY
GROUND SYSTEMS GROUP
FULLERTON, CALIFORNIA
COLLECTION SYSTEMS
INTERIM REPORT
AUGUST 1983
Prepared Under Contract To
THE CENTRAL INTELLIGENCE AGENCY
DEPUTY DIRECTOR FOR INTELLIGENCE
OFFICE OF SCIENTIFIC AND WEAPONS RESEARCH
UNCLASSIFIE
Approved For Release 20 / / : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/Q x,{;_.. IA-RDP86BOO269RO10050901-1
me 30
gency
entraOgence
Office of the Deputy Director for Intelligence
NOTE TO: Director of Central Intelligenc-
FROM Deputy Director for Intelligence
SUBJECT: Hughes Study on Collection Systems
Attached is the study we have been discussing.
I have outlined some key points in the first dozen
pages and call your attention to further highlighting
on pages 17, 33, 59, 71, 78, and 92. The summary
is useless.
OLd t)e4'
One point brought out in my brie in that I
did not find in the report is that in assessing
alternative collection systems you would select
a half dozen or so samples like the surface to
air missile sample discussed in the study to give
you a feel for comparative advantages of systems
in diverse areas. For example, you might want to
look at a particular economic target, two or three
other kinds of military targets and perhaps a
political military target to give you a relative
sense of the advantages and disadvantages of each
of the candid collection systems. You obviously
cannot apply this cumbersome methodology to all
possible targets. Samplings would be required.
I should caution again that Hughes acknowledges
more work remains to be done on this,end that it is
cumbersome and time consuming,and my impression is
that they are continuing this work.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
COLLECTION SYSTEMS
Interim Report
August 1983
Prepared Under Contract To
THE CENTRAL INTELLIGENCE AGENCY
DEPUTY DIRECTOR FOR INTELLIGENCE
OFFICE OF SCIENTIFIC AND WEAPONS RESEARCH
K.J. Baldasari
R.S. McKinnell
Approved by:
Approved by:
Program Manager
K.Jaldasari
Advanced Projects Dept.
W.J. Paik?f, Manager
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
This interim report was prepared by the Hughes Aircraft Company,
Fullerton, California under contract 82-N7880000. The prinidipal`Hughes
analysts for this effort have been Ms. K.J. Baldasari, Mr. R.S. McKinnell,
and Mr. W. Lund.
Approved For Release 2003/09/29 : CIA-RDP86B00269R00130005000.1-1
W
W
no
W
W
W
W
W
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
FOREWORD ................................................................... ii
I. PROBLEM STATEMENT ...................................................... 1
II. METHODOLOGY DEVELOPMENT ...............................................
III. ILLUSTRATIVE EXAMPLES OF METHODOLOGY ................................. 12
A. Information Tree Development ..................................... 15
B. Delineation of Information Tree Weights .......................... 21
C. Collection Concept Definition and Data Development............... 33
D. Collection Concept Performance Analysis .......................... 37
E. Sensitivity Analysis........................................... 39
IV. DEMONSTRATION EFFORT .................................................. 77
A. SAM Problem ...................................................... 78
B. Information Tree Structure ............. 79
..........................
V. St]M1{ARY ................................................................ 92
REFERENCES ................................................................. R-1
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
PROBLEM STATEMENT
HUGHES
? DEVELOP A METHODOLOGY TO PERFORM UTILITY ASSESSMENTS OF
COLLECTION CONCEPTS
- THE METHODOLOGY SHOULD BE CAPABLE OF DEALING WITH
THE ENTIRE CONCEPT LIFE CYCLE
? RELATE THE METHODOLOGY TO SPECIFIC INTELLIGENCE QUESTIONS
The purpose of this effort is to develop a methodology to assess the
utility and capabilities of collection system concepts. Such a methodology is
needed to bring a systematic procedure to a process which has heretofore been'
intuitive and variable.
The methodology developed must be flexible, enabling it to be applied to
the wide range of concerns and problems with which collection systems must
deal. The final task of this project therefore is to show the developed
methodology's flexibility and adaptability by applying the outlined procedures
to a mutually agreed upon problem, assessing the information required to
determine the impact of a new surface-to-air missile (SAM) on the balance of
power.
- 1/2 -
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
METHODOLOGY OBJECTIVES
HUGHES
o DEVELOP A STANDARDIZED PROCEDURE FOR ASSESSING COLLECTION
TECHNIQUES AND STRATEGIES
? STRUCTURALLY INDEPENDENT OF PROBLEM TYPE
? HANDLE MULTI-OBJECTIVE. MULTI-FACTOR DECISIONS
i GENERATE UNIQUE, TRACEABLE PATHS FROM PROBLEM ELEMENTS
THROUGH INFORMATION REQUIREMENTS TO APPLICABLE COLLECTION
TECHNIQUES
e METHODOLOGY SHOULD DEVELOP A MEANS TO FOCUS ON
SIGNIFICANT ISSUES, SCAN THE LESS SIGNIFICANT, ELIMINATE THE
IRRELEVANT
? INCORPORATE MULTI-FACETED TYPES OF DATA AND INFORMATION
? COMBINE QUALITATIVE AND QUANTITATIVE MEASURES OF
EFFECTIVENESS
? INCORPORATE UNCERTAINTY IN PARAMETER QUANTIFICATION
? ALLOW FOR EASY INCORPORATON OF ADDITIONAL INFORMATION
? AGGREGATE EXPERT OPINIONS
J
i
The first step in methodology delineation involved documenting
objectives the methods employed should satisfy. The overriding objective was
to develop an approach which would replace the intuitive methods employed in
the past with techniques which are auditable, generate traceable analysis'
paths and are valid over time.
The need for a standardised procedure is especially acute given the
diverse nature of problems to be addressed and the multifaceted decisions to
be reached. The methodology should highlight the critical decision components
in an easy manner and allow for analysis review and sensitivity testing.
A complicating factor in the problem is the diverse nature of the data
used in collection analysis. The data tends to range from being highly
qualitative, such a the doctrine of the enemy, to the highly quantitative,
such as the physical number of systems to be deployed. Additionally, the
collectibility of required information is not a guaranteed event, but instead
is uncertain and possesses a probability distribution whose form may depend
upon the expert performing the analysis or formulating the question.
Furthermore, the data employed does not remain constant, but instead is
dynamic and depends upon the timing of the analysis and its requirement.
Therefore, to reflect this diversity the methodology should be adaptable to
reapplication and easily incorporate additional information as it becomes
available.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
DECISION ANALYSIS
[HUGHES
------------------
DECISION ANALYSIS COUPLES THE QUANTITATIVE
TECHNIQUES OF SYSTEMS ANALYSIS WITH THE
QUALITATIVE THEORIES OF DECISION THEORY
DECISION ANALYSIS IS SUITED TO PROBLEMS
CHARACTERIZED BY
- UNCERTAINTY
- COMPLEX PREFERENCES
- IMPORTANCE
- UNIQUENESS
- LONG RUN IMPLICATIONS
Decision analysis refers to a body of knowledge and techniques which
combines decision theory and system analysis. Decision theory investigates
the manner in which people assemble information to make decisions and the
actual decision making process. System analysis brings to the effort
techniques for problem decomposition, handling uncertainty in parameters, and
mathematical decision rules.
The combination of these two disciplines is ideally suited to the
collection systems problem since it allows for uncertainty in parameter
evaluation, can handle complex problems through decomposition, and provides a
systematic framework for evaluation. Decision analysis is not appropriate f6r
smaller problems as the effort of analysis may be too large for less
significant issues. Therefore, the techniques are most appropriate when the
issues under study are complex, of relative importance, unique and
identifiable, and the long-run implications are such that a temporarilly
stable method of review is desirable.
10
^
^
^
^
^
w
^
^
^
w
^
^
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
SUCCESSFUL DECISION ANALYSIS
APPLICATIONS
? VOYAGER SPACE PROJECT
? NUCLEAR POWER PLANT SITE SELECTION
? URBAN DEVELOPMENT
? FEDERAL ENERGY POLICY
? CORPORATE INVESTMENT POLICY
? DAM SITE SELECTION
HUGHES
Decision analysis is a proven, developed approach through which
decisions have been made. Decision analysis has been applied on projects
ranging from the highly quantitative Voyager Space Project to the qualitative
concerns of urban development. The broad range of successful applications
illustrates the flexibility of the method as well as its maturity.
Selection decisions have been made successfully using these techniques.
By its very nature, decision analysis is well suited to comparing alternatives
and making incremental assessments.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
DECISION ANALYSIS: THE PRO'S
HUGHES
? THE DECISION MAKER IS ENCOURAGED TO SCRUNTINIZE THE PROBLEM
AS AN ORGANIC WHOLE
? ALLOWS EACH EXPERTTO GIVE TESTIMONY ABOUT A PARTICULAR AREA
OF EXPERTISE IN AN UNAMBIGUOUS, QUANTITATIVE MANNER THAT
CAN BE INTEGRATED INTO THE OVERALL ANALYSIS
? ALLOWS THE DISTINGUISHING OF THE DECISION MAKER'S
PREFERENCES FOR CONSEQUENCES FROM JUDGEMENTS ABOUT
UNCERTAINTIES
? CAN BE USED TO CLEARLY COMMUNICATE THE RATIONALE FOR THE
ADOPTED STRATEGY AND RALLY SUPPORT FOR IT
? CLEARLY POINTS OUT THE FACTORS INCORPORATED INTO THE
ANALYSIS
? ALLOWS FOR THE DECOMPOSITION OF THE PROBLEM INTO ITS BASIC
PARTS SO ISSUES WHERE THERE ARE FUNDAMENTAL DISAGREEMENTS
CAN BE EXPOSED
Decision analysis brings several advantages to a collection system
analysis. The techniques encourage separation of the factors and basis of the
decision from the personal biases and feelings of the decision maker.
Additionally, structure is introduced to the problem which allows for easier
communication of ideas and improved analysis possibilities. The framework is
such that sensitivity analysis is possible. Under the intuitive methods of
the past such analysis of variance was not possible.
Decision analysis makes complex problems easier because decomposition
into smaller problem pieces is possible. Decomposition allows those most
capable of performing a subtask to do so without also requiring performance
outside any areas of expertise. Synergy will result when the pooled opinions
of these experts are generated which will lead to a better basis for reaching
decisions than otherwise would have been available.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
DECISION ANALYSIS: THE CON'S
HUGHES
? DECISION ANALYSIS DOES NOT REPLACE OR LESSEN THE VALUE
OF THE EXPERT'S OPINION
? THE DEVELOPED METHODOLOGY DOES NOT ENSURE ALL
PROBLEM ELEMENTS ARE CONSIDERED
While decision analysis offers many advantages, the methodology is not
without its weaknesses. As with any technique, these limitations need to bee
recognized going into A task so their potential effects can be minimized. The
first limitation of decision analysis is it will still require input from
personnel knowledgeable about the subject matter. The methodology will not
serve as a substitute for knowledge or experience, but the method will take
whatever expertise is available and make optimal use of the information.
The developed methodology also does not ensure all problem elements are
automatically considered. Missing elements will occur since the problem is
structured from the ground up and omissions are possible. These omissions
are, however, easily corrected later in the analysis when they are
discovered. This weakness also points to the need to have a "pool" of experts
work on the problem to assure full coverage of problem issues.
- 7/8 -
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
METHODOLOGY CHOSEN -
DECISION ANALYSIS
OBJECTIVE
? STANDARDIZED
PROCEDURE
? GENERATE TRACEABLE
PATHS
? INCORPORATE MULTI-
FACETED DATA AND
INFORMATION
HUGHES :
DECISION ANALYSIS SOLUTION
? TREE STRUCTURE SUITED TO WIDE VARIETY OF
PROBLEMS WITH MULTIPLE OBJECTIVES
? TREE ANALYSIS HIGHLIGHTS CRITICAL PATHS
AND PARAMETERS OF PROBLEM
? TREE STRUCTURE HANDLES QUALITATIVE AND
QUANTITATIVE DATA
? UNCERTAINTY IS EASILY HANDLED BY
SENSITIVITY ANALYSIS
? EASILY UPDATED WITH NEW INFORMATION
To verify decision analysis is a desirable methodology, a comparison is
needed between the outlined objectives and the advantages of decision
analysis. From the above chart, it is apparent decision analysis satisfies
each of the outlined criteria and is therefore suited to the task outlined.
The remainder of this report addresses the specifics of applying the
methodology to collection system problems. To do this a simple problem of
information collection and evaluation will be used for illustrative purposes.
By working a simple problem in detail, it is felt a more in-depth delineation
of the methodology is obtained and problems which might be expected are-
highlighted and resolved.
The result of using this methodology is the introduction of structure
and a standardized procedure to what has previously been an intuitive,
unstructured analysis. The methodology will not replace or reduce the
importance of expert opinion. The expert's opinion will, however, be more'
visible and available for review and discussion.
- 9 -
Approved For Release 2003/09/29 CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
THREE PHASE APPROACH
-------------------
HUGHES
1. DETERMINATION OF TREE STRUCTURE AND THE
ASSIGNMENT OF BRANCH WEIGHTS
Ii. DEFINITION OF COLLECTION CONCEPT AND THE
DETERMINATION OF COLLECTION UNCERTAINTIES
111. APPLICATION OF OTHER FACTORS AND SENSITIVITY
ANALYSIS
? TIMELINESS
? SURVIVABILITY
r
r
r
^
^
^
^
The basic approach of the defined methodology divides into three
phases. The first phase involves formulating the problem to be addressed in
an information tree structure. This "tree" depicts the relationships and"
hierarchy of the problem from the top of the tree, the question to be
answered, to the bottom of the tree, the collectable data items which combine
to answer the overall issue. The intermediate stages of the tree form the
linkages between collectable data and the overall question. Information at
these levels might include system parameters, doctrine considerations, and
deployment questions.
Once the structure has been outlined, weights are assigned to the
branches to indicate a particular branch's importance in answering the
overriding issue at the next highest level or node. A value between O'and 1
is used to define this importance, 0 indicating no relevance (the branch could
be eliminated from the tree) and 1 indicating this branch is the only relevant
branch (all others emanating from this node could be eliminated).
r
r
r
^
IN
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
With the tree structure complete and the weights assigned, the second
phase begins with specification of the collection concepts along with their
capabilities. Defining the concept involves establishing the probabilities
the concept can collect the data described at the lowest levels of the tree.
The probability assessment will in practice take the form of a distribution
describing the likelihood of successful collection. By combining the concept
definition with the tree structure, a result is obtained which delineates the
value of information available to answer the overriding question of the tree.
Incorporation of outside factors not included in the tree structure or
concept definition is accomplished in phase three. Considerations such as'the
availability or timeliness of a concept, or the availability of data from a
given concept versus another, can be included in the analysis through the use
of a discount factor. By allowing for the time value of information in this
manner, concepts whose timing of availability differ can be compared.
Additionally, concepts whose expected lifetimes differ can be analyzed in a
similar manner. The additional information available because one system will
have a longer lifetime than another can be handled by "compounding" the value
obtained by the tree and concept definition analysis. The result achieved by
completing phase three will be a probability distribution showing the
information availability and its likelihood of collection. Using these
distributions, alternative concepts can be evaluated keeping in mind risk
aversion, minimum requirements or other factors.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
EXAMPLE PROBLEM
HUGHES
JOHN IS CONSIDERING BUYING A HOUSE. HE HAS
DRIVEN PAST ONE ON THE WAY TO WORK THAT LOOKS
LIKE A GOOD POSSIBILITY
HE NOW HAS TO FIND INFORMATION ON HOW WELL THIS
HOUSE FITS HIS NEEDS - BOTH OBJECTIVE AND
SUBJECTIVE
HE WANTS TO KNOW
? WHAT IS THE BEST WAY TO COLLECT THE
INFORMATION HE NEEDS TO MAKE A DECISION
STEP NO. 1: BUILD AN INFORMATION VALUE TREE
STEP NO. 2: ASSESS THE VALUE OF THE INFORMATION
THAT CAN BE GATHERED BY COMPETING INFORMATION
COLLECTION CONCEPTS
STEP NO. 3: EVALUATE OTHER FACTORS IMPACTING
COLLECTION PERFORMANCE
What follows is an in-depth discussion of the methodology which will
describe why it can be such a powerful decision support tool and, more
specifically,
- the techniques necessary to apply it,
- the information it can provide,
- the insight it will give to both analysts and decision makers.
As a supplement to this technical discussion, a simple example will be used to
illustrate the key concepts of the methodology. Please note, however,
- the example is by necessity somewhat over-simplified in order to
demonstrate the concepts; it is not intended to be completely
realistic, and
- so that it can provide insight into how such an analysis is done,
the example will be presented in a very detailed and numerically
intensive manner.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
jFM
s.
a
ME
.00
0
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The example centers around John who is considering buying a house. He
has. driven by one on the way to work that looks like a good possibility and
now must collect information on how well this particular house fits his
needs. Specifically he wants to know:
"What is the single most effective method to use to collect the
information necessary to make this decision."
The basic steps John will need to follow to perform his analysis are:
(1) Build an information value tree to delineate the raw data that
should be collected. as well as the relative value of each piece to
the main question.
(2) Assess collection concept performance by evaluating the ability of
each candidate collection concept to meet each raw data requirement.
(3) Incorporate any other relevant factor and perform sensitivity
analyses.
Note: This example has been specifically designed to illustrate all facets of
the methodology and the reader should not be concerned that the techniques
seem laborious and involved. In actual analyses many of the steps illustrated
will not be necessary because sensitivities of the problem will allow some
simplification.
- 13/14 -
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
- - - - - - - - - - - - -
'HUGHES
INFORMATION VALUE TREE
1 :1 1:
REPRESENTS THE RELATIONSHIP BETWEEN DIFFERENT LEVELS OF
INFORMATION NEED AND THE DATA ELEMENTS THAT SUPPORT THEM
DEVELOPED BY DRIVING DOWN ONE LEVEL OF INFORMATION AT A TIME
FROM THE DECISION MAKER'S QUESTION TO THE AVAILABLE DATA
DEPICTED GRAPHICALLY BY
? NODES - REPRESENTING SOME QUANTUM OF INFORMATION
? BRANCHES - REPRESENTING EITHER
- THE TIE UP TO THE NEXT HIGHER LEVEL OF INFORMATION
REQUIRING THIS PIECE OF INFORMATION
- THE TIES DOWN TO THE LOWER LEVELS OF INFORMATION
ELEMENTS SUPPORTING THIS PIECE OF INFORMATION
ESTABLISHES THE RELATIVE VALUE BETWEEN ALL ELEMENTS OF
INFORMATION RELATIVE TO EITHER
? A PARTICULAR NODE
? A PARTICULAR INFORMATION LEVEL
An information value tree is a construct for relating different levels
of information requirements and the raw data supporting them. It is a
systematic structuring of how these information and data items relate. The
information value tree is essentially a graph with
branches - which represent either data or pieces of information
aggregated from lower levels of information or raw data, and
nodes - which represent the tieing together of the data or
aggregated information to support a specific higher level
information need, that is, a further aggregation of
information.
The purpose of such a structure is to establish the value of each information
need or data item relative to the particular information requirement (node) it
supports as well as to any other information need or data item within the tree.
In structuring the information tree, the nodes employed can either be
"or" or "and" in character. The node type selected will depend upon the
structure and information required by the developer. Whichever structure type
provides the greatest visibility to the developer should be employed. The
only precaution needed for the actual analysis is the numerical combinations
performed at each node must be consistent with the node type. For "and" nodes
all branches into a node should be-summed yielding the nodal result and for
"or" nodes only the branches which are active, actually employed or satisfied,
should be summed.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
At the top of the next page is the tree John developed to evaluate which
collection concept provides the most effective support relative to his main
question: should he consider buying this house. This question is called the
"decision maker question." From the tree it can be seen that for John, the
utility or value of a house is solely a function of
- whether it fulfills certain critical needs he has established,
- the extent to which the house'and yard will need to be cared for, and
- the investment required and the return expected.
These are the "critical issues" that drive the decision maker's question. In
turn, each critical issue is defined by a set of "problem characteristics."
For example,. the investment potential has been broken down into the immediate
affordability, the continuing affordability, and the future worth. The
problem characteristics are themselves derived from "data items" which relate
to pieces of information that are actually collected. Note that the complete
set of data items that support the decision maker's question are on the far
right side and range from the number of bedrooms in the house to the planned
developments in the local area.
In general, the key concepts to consider at each level of the tree are
as follows:
Decision Maker Question - the main question against which the collection
concepts are being evaluated.
Critical Issues - the key issues or subquestions that drive the resolu-
tion of the decision maker question. These are still from a
qualitative vs a quantitative perspective.
System/Problem Characteristics - quantifiable information that defines
the key drivers behind the critical issues.
Data Items - data that is directly collected or directly derived from
collected data.
These levels are guidelines to help the analyst in systematically
constructing the tree structure such that all the relevant issues are
addressed and the key data requirements are delineated. The structuring
process can be done solely by the decision maker, as in this example, or in
concert with a group of experts in the various areas spanned by the tree and
is somewhat of an art. It is critical to be as systematic as possible; all
the relevant information needs to be captured and, conversely, as little as
possible of the extraneous. A key sign that some part of the tree's structure
is essentially complete occurs when review of that section produces
discussions revolving around semantics which generate no new insight or data
The information represented at different levels of the tree ranges from
very qualitative at the decision maker question level (e.g., should John E
consider buying this house) to very quantitative (e.g., the number of bedrooms
in the house). Specifically, quantitative information can be described using
numbers; examples might be the number available, dollars spent, percent lost,
distance traveled, probability of occurrence. Qualitative information, on the
other hand, is sometimes referred to as "intangible"; examples might be margin
for safety, ability to predict, impact on the status quo, level of
performance. One of the strengths of this decision analysis approach is the
ability to tie together such different types of data.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
4
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
JOHN'S TREE
FUNCTIONAL
NEEDS
HUGHES
DISASTER VULNERABILITY
(FIRE. EARTHQUAKE. ETC)
DOWN PAYMENT
IMMEDIATE
AFFORDABILITY CLOSING COSTS
BASIC MONTHLY PAYMENT
DECISION MAKER CRITICAL PROBLEM
QUESTION ISSUES CHARACTERISTICS
LEVELS OF THE TREE STRUCTURE
1. DECISION MAKER QUESTION
? ULTIMATE QUESTION OR PROBLEM TO BE
ADDRESSED BY REMAINDER OF THE TREE
11. CRITICAL ISSUES (KEY SUBQUESTIONS)
? QUALITATIVE QUESTIONS WHOSE SOLUTIONS ARE
NECESSARY TO ANSWER DECISION MAKER
QUESTION
III. SYSTEM/PROBLEM CHARACTERISTICS
? QUANTITATIVE SYSTEM PARAMETERS WHICH
COMBINE TO PROVIDE AN ANSWER-TO CRITICAL
ISSUES
IV. RAW AND DERIVED DATA
? QUANTITATIVE DATA WHICH IS COLLECTED IN ITS
RAW FORM BY COLLECTION SOURCE OR EASILY
DERIVED FROM SUCH DATA
N
7OTa
PROJECTED
EXPECTED APPRECIATION
PLANNED DEVELOPMENTS
------------------
HUGHES
QUALITATIVE
PERSPECTIVE
QUANTITATIVE
PERSPECTIVE
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 As John's tr4PPfRVPO1F4ii$ 9a?IPe3/O% h- C DIiN l iQ0J4QOBgOQQJ 1
of branches; this is not a requirement, however. More complex questions where
more detail is necessary or even just desired may have several tiers of
branches and nodes at each level. Additionally, the number of tiers or
branches at a particular level does not have to be consistent throughout the
tree. For example, instead of just stopping with a branch corresponding to
the relative distance of the house to work, John could have put a node at the
end of that branch and added two new branches that corresponded to the
distance relative to where he works and relative to where his wife works. The
more detailed the branch structure becomes, the more information it captures
and the more insight it provides into the data and issues that drive the
decision maker's question. Care must be taken, however, to avoid excessive
detail; for example, the specific distance to every single school in the local
area is not of any concern to John, so to break the tree down further in this
area is undesirable.
I"
ro
OW
W
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
BRANCHING CRITERIA
-------------------
HUGHES
COMPLETENESS
? THE SET OF BRANCHES SHOULD CHARACTERIZE ALL
THE SUPPORTING INFORMATION NECESSARY TO BE
CONSIDERED AT A GIVEN NODE
SIGNIFIGANCE
? EACH BRANCH SHOULD REPRESENT AN IDENTIFIABLE
AND MEASURABLE ELEMENT OF CONTRIBUTING
INFORMATION
FAMILIARITY
? EACH BRANCH SHOULD BE UNDERSTANDABLE TO THE
EXTENT IT IS POSSIBLE TO ESTABLISH A WEIGHTING
RELATIVE TO THE OTHER BRANCHES AT THE SAME
NODE
INDEPENDENCE
? TWO BRANCHES FROM THE SAME NODE SHOULD NOT
REPRESENT THE IDENTICALLY SAME INFORMATION
_???_?_? ?_?_ REQUIREMENT
BE DEPENDENT ON EACH OTHER
In developing an information value tree, there are some simple
analytical guidelines for delineating the set of branches at a particular
node. Since each node represents the tieing together of all its relevant
data, the set of branches must be "complete", that is there must be a branch
for each contributing piece of information. For example, "immediate
affordability" is completely defined by what the down payment and the closing
costs are. By the same reasoning, a branch should only be constructed if it
represents a "significant" element of information, that is, it has some.
relevance to the information requirement represented by the node. The fact
that the house has a tile roof is not relevant to whether it is big enough.
Next, each branch should be "familiar", that is, it should represent a piece
of information that both the analyst and reviewer can understand and
evaluate. Last, all the branches should be "independent"; no two branches
should depend directly on one another or represent the identically same piece
of information.
- 19/20 -
Approved For Relea%dLQ9/ :nC)A-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
III. B. DELINEATION OF INFORMATION TREE WEIGHTS
DELINEATION OF WEIGHTS
-------------------
HUGHES
JOHN HAS NEVER BOUGHT A HOUSE BEFORE, SO
HE IS NOT SURE WHAT THE RELATIVE
IMPORTANCE OF THE VARIOUS ISSUES ARE
TO GAIN INSIGHT, HE CONSULTS SOME OF HIS FRIENDS
(WHO HE BELIEVES TO BE KNOWLEDGEABLE IN THE AREA --
LOCAL EXPERTS)
MARY: WHO IS IN THE PROCESS OF BUYING HER THIRD
HOME
SUE: WHO HAS EXTENSIVE REAL ESTATE
INVESTMENTS
GARY: WHO HAS NEVER BOUGHT A HOUSE, BUT WHO
JUST FINISHED A SEMINAR FOR FIRST-TIME
HOME BUYERS
Once the structure of the information value tree has been established
(the nodes and the branches have been defined), the next step is to delineate
the relative value weightings. This can be done by the decision maker, in
this case, John, or it can be delegated by the decision maker to a group of
experts. Since John has virtually no experience in the real estate market, he
decided he needed help in assigning the weights to the branches of his tree.
So he selected three experts to advise him.
- Mary is in the process of buying her third home so she's had a fair
amount of experience not only in actually buying a house, but also
in maintaining one. Of all his "experts", she probably represents
the one whose experience John feels is the most consistent with his
perspective.
- Sue is very much involved in the real estate market and has many
investments. She is predominately interested in investment
potential.
Gary has never bought a house but he just finished attending a
seminar designed for first-time home buyers so he has information
about what the "experts" say should be considered when buying a
house.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The weight assigned to each branch reflects its value or contribution to
the information requirement relative to the other branches emanating from the
same node. This weight may range in value between 0 and 1 where
0 - the piece of information has no impact on the information
requirement at all, and
1 - this piece of information is the only contributing information
required.
The weights assigned to all the branches at a particular node must sum to 1,
reflecting the premise that together they represent the complete set of
significant information needed to define the information requirement at that
node.
The actual calculation of the branch weights is a two step process.
First, the relative value of all the branches at a particular node is
established. For example, piece B of information may be twice as important as
piece A and piece C of information may be three times as important as piece
A. This can be analytically described by:
3*B = 6*A = 2*C
The next step is to normalize by forcing the weights to sum to 1 which gives:
A + B + C =A+2*A+3*A= 1
A = 1/6
B = 1/3
C = 1/2
By requiring the branch weights to sum to 1 at every node, a consistent
value structure is forced on the entire tree, enabling the analyst to readily
calculate the relative value of any piece of information or data item within
the tree. Ultimately, this leads to a preference ranking over the set of
collectables (data items) which can then be used to evaluate the set of
potential collection concepts.
0
r
II
W
W
0
0
p
W
IN
W
M
Approved For Release 2003/09129 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
ASSIGNMENT OF RELATIVE VALUES I
HUGHES
PROCESS WHEREBY THE RELATIVE VALUE OF THE
VARIOUS PIECES OF INFORMATION (BRANCHES)
REQUIRED TO SUPPORT A PARTICULAR (HIGHER)
LEVEL OF INFORMATION (NODE) ARE ESTABLISHED
.VALUES (OR WEIGHTS) MAY RANGE BETWEEN 0 AND 1
WHERE
0 - HAS NO IMPACT AT ALL
1 - IS THE ONLY CONTRIBUTING FACTOR
AT A PARTICULAR NODE (QUANTUM OF INFORMATION),
THE WEIGHTS ASSIGNED TO ALL THE CONNECTING
BRANCHES (SUPPORTING INFORMATION) MUST SUM
IDENTICALLY TO 1 TO ALLOW COMPARISON OF
INFORMATION REQUIREMENTS AT NODES OF THE SAME
AND HIGHER LEVELS
CALCULATION OF BRANCH WEIGHTS I
HUGHES
GOALS:
? THE WEIGHTS REFLECT THE RELATIVE VALUE OF THE SUPPORTING
INFORMATION TO DEFINING THIS PIECE OF INFORMATION
? ALL THE WEIGHTS CORRESPONDING TO THE SUPPORTING
INFORMATION MUST SUM IDENTICALLY TO 1
PROCESS:
? ESTABLISH THE RELATIVE VALUE OF THE VARIOUS INFORMATION
REQUIREMENTS, EG
VALUE (B) = 2 ? VALUE (A)
VALUE (C) = 3?VALUE (A)
? NORMALIZE TO FORCE WEIGHT TO SUM TO 1, EG (AS. ABOVE)
A + B + C = 1
A + 2A + 3A = 1
6A = 1
A=1/6,B=1/3,C=1/4
g-23/24-k
Approved For Release z5~~3%0~9/2t1a8)A-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
------------------
SAMPLE OF EXPERT OPINIONS HUGHES
Three very different sets of weights were provided by John's experts as
shown in the trees above. Mary and Sue, for example, have divergent
priorities for the "investment potential" in general and the "future worth" in
particular. Mary is much more concerned about the supportability of the house
and the functionality it provides; Sue, with her background in real estate
investments, concentrates on the investment section of the tree. Gary has yet
another set of weights although his are closer to Sue's.
The next issue for John is how to aggregate these three different
perspectives into a single consensue tree.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The derivation of consensus weights for the decision maker's tree
requires some procedure for aggregating the values for the weights provided by
experts into a single set of estimates. Just summing the values up and taking
the average, although a simple approach, does not take into account any
information the decision maker may have about the differing levels of
expertise the experts may have or any knowledge of their individual biases.
Additionally, such a procedure provides no way to adjust the impact extreme
opinions may have on the aggregation (either to emphasize or to mask them).
The question of how best to create a consensus position from a pool of
expert opinions has received much attention from decision analysis experts.
Many procedures have been proposed varying from the simplistic, such as the
averaging approach suggested above, to the theoretically sophisticated; as
yet, however, there is no general agreement about which approach yields the
most valid results. However, to be useful for the collection systems problem,
an aggregation procedure should meet four requirements:
(1) There should be a way to incorporate any preferences the decision
maker may have between the experts so differing levels of emphasis
can be assigned in proportion to the perception of their expertise.
(2) In order to preserve the relative value structure of the entire
tree, the aggregation procedure must produce consensus weights such
that the sum of the values assigned to the set of branches at any
node is still exactly 1.
(3) The weight structure for any particular expert must be incorporated
in such a way that his or her relative rankings are preserved.
(4) To enable the procedure to be as usable as possible, the
information required to implement it should be readily available.
An example of one of the more analytic approaches that has been
developed is a procedure proposed by Morris. It is a very rigorous Bayesian
approach that uses each expert's past performance to condition, that is
weight, his or her current prediction. Initially, the excellent theoretical
foundation for this procedure, as well as others of a similar nature, seemed
to strongly point to the implementation of such an approach within this
methodology. Upon further investigation, however, the very strength of such
procedures proved to also be their weakness, at least relative to the type of
decision support methodology that is needed to handle the collection system
problem. The application of approaches such as Morris's requires access to a
reasonably large sized data base describing the performance of the expert with
similar problems. Typically, such a data base is not available. As a result,
the approach selected for implementation with this methodology is less
sophisticated but in return the decision maker will have more control of how
the consensus is to be achieved. If over time the data base necessary to
support a method such as Morris' can be accumulated, a more sophistica;ed
approach can be added to the procedure suggested in this report.
IN
^
0
r1
^
11
^
^
11
w
^
r
^
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
AGGREGATION OF EXPERT OPINION
HUGHES
COMBINING THE OPINIONS OF SEVERAL EXPERTS IN A
VALID MANNER ONE NEEDS TO CONSIDER
? DIFFERING LEVELS OF EXPERTISE BETWEEN
EXPERTS
? INDIVIDUAL BIASES
? EFFECTS OF EXTREME OPINIONS
METHODOLOGY CHOSEN MUST ALLOW FOR THE
ADJUSTMENTS NECESSARY AND ALSO BE
SATISFACTORY IN THE
? TEMPORAL
? REASONABLE IN THE AMOUNT OF DATA
REQUIRED
? EASILY TRANSFERABLE AND AT THE SAME TIME
MAINTAIN AS MUCH DATA AS POSSIBLE
MORRIS APPROACH
? APPROACH TO EXPERT RESOLUTION OR
CONSENSUS DISCUSSED IN THE PROPOSAL
? RIGOROUS BAYESIAN APPROACH
J
HUGHES
? BASED ON PRIOR PERFOMANCE OF EXPERTS
? DRAWBACK
- DATABASE ON PRIOR PERFORMANCE NOT
TYPICALLY AVAILABLE
? APPROACH SELECTED
- LESS TECHNICAL
- GIVES THE DECISION MAKER MORE DIRECT
CONTROL OF THE CONSENSUS PROCESS
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
For the collection systems problem, the approach selected to derive the
consensus estimate from a set of expert opinions is based on a weighted
average of the experts' opinions. Given the estimates of N experts for a
value to be assigned to a particular weight,
V(1), V(2), ..., V(N),
as well as a set of preferences (or weights) from the decision maker
associated with each of these experts,
W(l), W(2), ..., W(N),
the consensus value (W*) is the normalized weighted average of the experts'
opinions, that is, the weighted average divided by the sum of the weights,
W = 1 *[ N
W * V ].
W 3_1 j 3
This approach, besides being very straightforward to apply, is appealing in
the way it allows the decision maker to emphasize the estimates of the various
experts in a way that is consistent with his or her evaluation of each of the
experts. Another benefit is that it does not require the extensive historical
database needed to implement an approach like Morris's. In fact, if the
decision maker has no preference between the experts, the weights, Wi, can
be set identically to 1.
John has decided that of his three experts, Mary's perspective is the
one that most closely matches his position. As such, he feels that her
estimates should be weighted about twice as much as either Sue's or Gary's,
whose opinions he feels should be weighted about equally. The resulting
preferences or weights are
Mary. 2
Sue: 1
Gary: 1
Using these values,, John calculates the consensus weights for his information
value tree. As an example, the expert opinions associated with the branch for
"functional needs" are
Mary: 0.35
Sue: 0.15
Gary: 0.25
giving a consensus weight of
W* = (1/4)*[(2)(0.35) + (1)(0.15) + (1)(0.25)]
= 0.275
The complete set of consensus estimates for the weights are shown in thte
adjoining table.
W
ro
0
A
OR
^
IN
In
01
^
A
^
0
F1
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
CONSENSUS METHODOLOGY
HUGHES
GIVEN
? A SET OF WEIGHTS OR RELATIVE VALUES PROVIDED
BY N EXPERTS,
V1,...,VN
? A SET OF DECISION MAKER PREFERENCES OR
WEIGHTS ASSOCIATED WITH THOSE N EXPERTS,
W1,...,WN
THEN
? THE CONSENSUS WEIGHT, W*, IS DEFINED BY
W* = 1 N
Wi j=1WJVJ
i=1
EXAMPLE
? IF V1 = 0.35, V2 = 0.15, V3 = 0.25
W1 =2,W2=1,W3=1
? THEN W* = 1 (2) (0.35) + (1)(0.15) + (1) (0.25) = 0.275
RELATIVE
MARY
SUE
GARY
CONSENSUS
TO
TREE
EXPERT PREFERENCES/WEIGHTS
2
1
1
-
-
FUNCTIONAL NEEDS
0.35
0.15
0.25
0.275
0.275
BIG ENOUGH
0.80
0.30
0.40
0.575
0.158
BEDROOMS
0.50
0.40
0.50
0.475
0.075
SO FOOTAGE
0.50
0.60
0.50
0.525
0.083
LOCATION
0.20
0.70
0.60
0.425
0.117
SCHOOLS
0.40
0.60
0.45
0.463
0.054
WORK
0.40
0.10
0.20
0.275
0.032
SHOPPING
0.20
0.30
0.35
0.262
0.031
SUPPORT REQUIRED
0.50
0.25
0.25
0.375
0.375
MAINTAINABILITY
0.80
0.70
0.50
0.700
0.262
AMT OF YARD WORK
0.40
0.30
0.40
0.375
0.098
AGE OF HOUSE
0.60
0.70
0.60
0.625
0.164
DURABILITY
0.20
0.30
0.50
0.300
0.113
CONDITION OF SIMILAR
0.70
0.70
0.60
0.675
0.076
DISASTER VULNERABILITY
0.30
0.30
0.40
0.325
0.037
INVESTMENT POTENTIAL
0.15
0.60
0.50
0.350
0.350
IMMEDIATE AFFORDABILITY
0.45
0.20
0.40
0.375
0.131
DOWN PAYMENT
0.50
0.50
0.50
0.500
0.065
CLOSING COSTS
0.50
0.50
0.50
0.500
0.066
CONTINUING AFFORDABILITY
0.45
0.20
0.40
0.375
0.131
BASIC MONTHLY
0.40
0.40
0.35
0.388
0.051
INTEREST
0.40
0.40
0.40
0.400
0.052
PROJ. ASSESSMENTS
0.20
020
0.26
0.212
0.028
FUTURE WORTH
0.10
0.60
0.20
0.250
0.088
EXP. APPRECIATION
0.30
0.60
0.50
0.425
0.037
PLANNED DEVELOPMENT
0.70
0.40
0.50
0.575
0.051
: HUGHES
CALCULATION
OF
CONSENSUS
WEIGHTS
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The consensus tree developed by John is shown on the adjoining page at
the left in the top figure. As required, the weights associated with the
branches at each node sum to 1. John has reviewed this tree and is fairly
comfortable with both the structure and the weights.
On the right of the figure is another version of the same consensus tree
which shows the relative value of each piece of information to all other items
at the same level instead of just those items at the same node. The
"maintainability" issue, for example, is evaluated to be three times as
important as the question of the house's "future worth". For this version of
the tree, the values are calculated by multiplying the series of branch
weights from the original tree along the tree from the decision maker question
to that specific branch. In the case of the "maintainability" branch, the
weight is calculated by multiplying, the weight for that branch times the
weight for the "support" branch, that is,
W = (0.700)*(0.376)
= 0.2622
This approach to structuring the information value tree provides a
ranking at each level of all the corresponding information requirements. This
is-particularly useful at the data item level as it produces a prioritization
of the collection requirements. In John's tree, the most important data item
to be collected, the branch with the high relative value is the "age of the
house" (0.164); at the other end of the spectrum, the least important data
item is "possible assessments" (0.028). This modified form of the tree makes
it readily apparent that the chief issue seems to be the support of the house
and, more specifically, the maintenance of the house, whereas the least
concern is given to its future worth.
This completes the first step of the analysis, the development of an
information value tree. The decision maker now has insight into the data
items that would be desirable to collect as well as their relative
importance. This ranking will be useful in the next step, the evaluation of
the various candidate collection concepts.
r
^
^
0
r
p
^
^
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
JOHN'S CONSENSUS TREE
HUGHES
BEDROOMS
BIG ENOUGH
0.475
SD. FOOTAGE
6
I SO. FOOTAGE
0.626
0.083
SCHOOLS
-0.163
SCHOOLS
0
061
LOCATION
WORK
WORK
.
0.126
SHOPPING
0.276
0.032
SHOPPING
0.262
0.031
YARD
MAINTAINABILITY
0.375
YARD
096
0
AGE Of NOME
AGE OF H
ME
.
0.700
0.626
O
0.164
CONDITION
0
CONDITION
DURABILITY
.676
0.076
0.300 DISASTER VUL.
DISASTER VUL.
0.326
0.037
DOWN
DOWN
IMM. FFORD.
0.500
0.0665
0.376 CLOSING
CLOSING
0.500
0.0666
BASIC ~-
0.388
BASIC MONTHLY
0.061
INTEREST VAR.
0.376
0.400
0.062
POSS. ASSESSMENT
POSS. ASSESSMENT
0.028
EXP. APPRECIATION
EXP. APPRECIATION
FUTURE WORTH
0.426
0.037
0.260 PLANNED OEV.
PUNNED DEV.
0.675
0.061
BRANCH VALUES
RELATIVE TO NODES
BRANCH VALUES
RELATIVE TO EACH LEVEL
--------------
PRELIMINARY CONCLUSIONS HUGHES
? CHIEF ISSUE
- MAINTAINABILITY OF THE HOUSE, SPECIFICALLY
AS IT PERTAINS TO ITS AGE
? MOST IMPORTANT DATA ITEM
- AGE OF HOUSE
? LEAST IMPORTANT DATA ITEMS
- POSSIBLE ASSESSMENTS
- LOCATION RELATIVE TO SHOPPING
- LOCATION RELATIVE TO WORK
- EXPECTED APPRECIATION
- DISASTER VULNERABILITY
- 31/32 -
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269RO01300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
INFORMATION COLLECTION
CONCEPTS
HUGHES
NEXT STEP: ASSESS THE PERFORMANCE (I.E., THE VALUE
OF THE INFORMATION COLLECTED) OF THE VARIOUS
INFORMATION COLLECTION CONCEPTS TO BE STUDIED
ACCOMPLISHED BY ASCERTAINING EACH CONCEPT'S
ABILITY TO COLLECT EACH REQUIRED DATA ITEM IN
TERMS OF
? EXPECTED LIKELIHOOD OF COLLECTING THE
REQUIRED DATA
? THE MINIMUM LIKELIHOOD OF COLLECTION
? THE MAXIMUM LIKELIHOOD OF COLLECTION
WHERE THE LIKELIHOOD OF COLLECTION IS A FUNCTION
OF
? THE PROBABILITY OF COLLECTING THE DATA
? THE QUALITY OF THE DATA COLLECTED, I.E., THE
ABILITY TO SUPPORTTHE SUCCEEDING INFORMATION
REQUIREMENTS
The next step of the methodology focuses on the evaluation of the
ability of each candidate concept to meet each raw data collection
requirements. This, combined with the relative weight structure derived in
the previous step, will provide a means for analyzing collection performance.
For each concept, the collection ability relative to each data
requirement will need to be specified not only in terms of its expected or
likely performance, but in order to be able to do confidence assessments,
minimum and maximum performance levels will also need to be delineated. All
performance likelihoods are a combination of both the probability of
collection and the quality of what is collected relative to the data
requirement. For example, relative to information about "planned
developments" in the local area, if one of John's candidate concepts on the
average will have an 80% chance of collecting some data, and if collected, the
data will probably cover 95% of the developments, then a likely performance
level, P, would be
P = .80 * .95
or approximately 75%. Quality thresholds, i.e., stipulation that if the data
is not at least a certain quality it is not any good at all, are handled by
assigning a 0 to the quality factor if it falls below the threshold.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
John has identified four candidate approaches to collect the data he
needs:
(1) Drop by the house in question and talk with the owner.
(2) Call up the city for information.
(3) Arrange with a friend (who is a real estate agent) to get the
necessary data.
(4) Check the newspaper to see if the owner is advertising the house.
There may be other approaches that he might use, but these four are the ones
under consideration. The question then is which one is the most effective
collection concept, which one will collect the information with the highest
overall potential value.
The first step in evaluating any of these collection concepts is to
define the concept's ability to collect each data item in terms of its best or
maximum performance, its worst or minimum performance, and its likely or
expected performance. Data on the performance of the concepts can be provided
by the decision maker or by an expert or experts designated by the decision
maker. In the case of more than one expert, the same consensus techniques
used to delineate the branch weights can be applied.
The table on the adjoining page summarizes the performance data for
John's problem for each of his collection concepts, For example, in the area
of "disaster vulnerability", John's estimate is that the owner's likelihood of
supplying him that information is somewhere between 40% and 60%, the city's
likelihood is between 60% and 80%, and his friend's likelihood is-between 50%
and 70%. He does not expect to find this particular data item in a newspaper
advertisement.
w
r*
W
W
I
-34-
Approved For Release 2003/09/29 CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
POTENTIAL SOURCES FOR
JOHN'S DATA
-------------------
HUGHES
JOHN HAS IDENTIFIED 4 POTENTIAL WAYS TO COLLECT
THE DATA HE NEEDS:
(1) DROP BY THE HOUSE IN QUESTION AND TALK WITH
THE OWNER
(2) CALL UP THE CITY FOR INFORMATION
(3) ARRANGE WITH A FRIEND WHO IS A REAL ESTATE
AGENT TO GET THE NECESSARY DATA
(4) CHECK THE NEWSPAPER TO SEE IF THE OWNER IS
ADVERTISING THIS HOUSE
HE NOW NEEDS TO DEFINE THE PERFORMANCE OF EACH
OF THESE RELATIVE TO EACH DATA REQUIREMENT IN
TERMS OF MINIMUM, EXPECTED, AND MAXIMUM
PERFORMANCE
? CAN BE DONE BY THE DECISION MAKER OR BY USING
EXPERT CONSULTATIONS AND CONSENSUS
TECHNIQUES AS WAS DONE WITH THE WEIGHTS
INFORMATION PERFORMANCE I
FACTORS
TALK WITH
TALK WITH
TALK WITH
FRIEND
CHECK
OWNER
CITY
(AGENT)
NEWSPAPER
MIN
EXP
MAX
MIN
EXP
MAX
MIN
EXP
MAX
MIN
EXP
MAX
NUMBER OF BEDROOMS
1.0
1.0
1.0
0
0
0
0.8
0.9
1.0
0
0
0
SQUARE FOOTAGE
1.0
1.0
1.0
0
0
0
0.6
0.8
1.0
0.5
0
7
0
9
SCHOOLS (LOCATION)
0.5
0.7
0.9
0.8
0.9
1.0
0.5
0.7
0.9
0.2
.
.
3
.
0
4
WORK (LOCATION)
0.1
0.2
0.3
0
0
0
0.2
0.5
0.8
0
.
0
.
0
SHOPPING (LOCATION)
0.2
0.45
0.7
0.2
0.3
0.4
0.4
0.65
0.9
0.1
0.2
0.3
AMOUNT OF YARD WORK
0.7
0.85
1.0
0
0
0
0
0
0
0
0
AGE OF HOME
0.9
0.95
1.0
0.3
0.5
0.7
0.4
0.6
0.8
0.2
0
35
0
0
5
CONDITION OF SIMILAR HOMES
0.5
0.7
0.9
0
0
0
0.3
0.5
0.7
0
.
0
.
0
DISASTER VULNERABILITY
0.4
0.5
0.6
0.6
0.7
0.8
0.5
0.6
0.7
0
0
0
DOWN PAYMENT
0
0
0
0
0
0
0.7
0.8
0.9
0
0
0
CLOSING COSTS
0
0
0
0
0
0
0.4
0.6
0.8
0
0
0
BASIC MONTHLY PAYMENT
0
0
0
0
0
0
0.7
0.8
0.9
0
0
0
PROJECTED INTEREST VARIANCE
0
0
0
0
0
0
0
4
0
55
0
7
0
0
POSSIBLE ASSESSMENTS
0.2
0.4
0.6
0.4
0.55
0.7
.
0.3
.
0.5
.
0.7
0.1
0.15
0
0
2
EXPECTED APPRECIATION
0.2
0.3
0.4
0.1
0.2
0.3
0.5
0.65
0.8
0
0
.
0
PLANNED DEVELOPMENTS
0.4
0.55
0.7
0.7
0.8
0.9
0.6
0.75
0.9
0.1
0.15
0.2
HUGHES
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Once the information has been gathered on the performance range of each
candidate collection concept, the next step is to evaluate their expected
performance. For each information requirement, this is expressed in terms of
an expected information value which is determined by multiplying the expected
performance level by the corresponding relative weight from the modified
information value tree. In the case of the third collection concept (talking
with his friend) and the first collection requirement (the number of bedrooms),
relative value of information = 0.075
expected performance = 0.9
expected information value = (0.9)*(0.075)
= 0.0676
The expected information values are in turn summed over the entire tree to
provide the overall expected information value for the complete tree for each
concept. These can be compared to determine the collection concept with the
highest expected performance.
The adjoining table contains the expected performance figures for each
of the collection concepts in John's problem. The first and the third
concepts (talking with the owner and talking with his friend) have markedly
higher expected performance than the remaining candidates. The margin between
the top two, however, is close enough that it is difficult at this point to
differentiate between them; not enough is known, only the expected performance
has been compared, not the complete performance range. To do this, John will
need to move into the third phase of the evaluation provess, the sensitivity
analysis.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
ro
PR
w
PF
P0
PP.
PW
OR,
 III. D. COLLECTco~ggCFEgrt PR#~RR0,k914-RDP86B00269R001300050001-1
HUGHES
PERFORMANCE ANALYSIS
THE INITIAL CONCEPT ANALYSIS IS BASED ON THE EXPECTED
PERFORMANCE OF EACH CANDIDATE COLLECTION CONCEPT
AND USES
- EXPECTED PERFORMANCE LIKELIHOODS
- CONSENSUS WEIGHTS
BASICALLY INVOLVES WEIGHTING THE EXPECTED
PERFORMANCE LIKELIHOODS WITH THE WEIGHTS (RELATIVE
TO THE DATA ITEM LEVEL) ASSOCIATED WITH THE
INFORMATION REQUIREMENTS
00-
EXPECTED PERFORMANCE `HUGHES
INFORMATION SOURCE
EXP. VALUE
TALK WITH THE OWNER
0.58
TALK WITH THE CITY
0.23
TALK WITH FRIEND (AGENT)
0.60
CHECK NEWSPAPER
0.15
- 37/38 -
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
SENSITIVITY ANALYSIS
HUGHES
? SENSITIVITY ANALYSIS IS THE CRUCIAL STEP OF THE
METHODOLOGY
? PROVIDES DECISION MAKER WITH THE
QUANTITATIVE LIMITS OVER WHICH HIS/HER
QUALITATIVE DECISION WILL BE VALID
? ANALYSIS CAN BE APPLIED SEVERAL PLACES
? OVERALL TREE
? INTRALEVEL ON THE TREE
? BETWEEN CONCEPTS
? INTERNAL TO CONCEPT ANALYSIS
? METHODS
? WORST CASE/BEST CASE
? MONTE CARLO
The sensitivity analysis is perhaps the most critical phase in the
methodology because it provides the decision maker insight into the limits
over which the collection concept selected is the most effective. Finding the
concept with the highest expected value for the information collected is only
the first part of the solution; it is equally important to understand the
range over which it is the most effective solution and how sensitive the value
of the information collected is to variances in each concept's performance
relative to the various information requirements.
Classically, sensitivity analysis refers to procedures in which
parameters are varied over their possible values in order to determine the
degree of variation in the resulting solution. Here sensitivity analysis will
be used in two ways
to evaluate the impact on the overall information value over the
entire range (not, just the expected value) of each collection
concept's performance relative to the various information
requirements, and
to incorporate factors in addition to the value of information that
differentiate the performance of various collection concepts.
Note: The reader is once again warned that the example illustrated here has
been specifically designed to illustrate all aspects of the developed
methodology. In actual practice, not all the steps shown will be
necessary as each problem will possess possibilities for simplification.
Approved For Release 2003/09/29 : CIA-RDP86B00269RO01300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
JOHN'S PERFORMANCE
ANALYSIS
-----------------
HUGHES
TALK WITH
TALK WITH
TALK WITH
FRIEND
CHECK
OWNER
CITY
(AGENT)
NEWSPAPER
MIN
MAX
MIN
MAX
MIN
MAX
MIN
MAX
NUMBER OF BEDROOMS
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
SQUARE FOOTAGE
0.58
0.58
0.23
0.23
0.59
0.62
0.13
0.17
SCHOOLS (LOCATION)
0.57
0.59
0.23
0.23
0.59
0.61
0.14
0.16
WORK (LOCATION)
0.58
0.58
0.23
0.23
0.59
0.61
0.15
0.15
SHOPPING (LOCATION)
0.57
0.59
0.23
0.23
0.60
0.61
0.15
0.15
AMOUNT OF YARD WORK
0.56
0.59
0.23
0.23
0.60
0.60
0.15
0.15
AGE OF HOME
0.57
0.59
0.20
0.26
0.57
0.64
0.12
0.18
CONDITION OF SIMILAR HOMES
0.56
0.59
0.23
0.23
0.59
0.62
0.15
0.15
DISASTER VULNERABILITY
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
DOWN PAYMENT
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
CLOSING COSTS
0.58
0.58
0.23
0.23
0.59
0.62
0.15
0.15
BASIC MONTHLY PAYMENT
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
PROJECTED INTEREST VARIANCE
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
POSSIBLE ASSESSMENTS
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
EXPECTED APPRECIATION
0.58
0.58
0.23
0.23
0.60
0.61
0.15
0.15
PLANNED DEVELOPMENTS
0.57
0.59
0.23
0.23
0.60
0.61
0.15
0.15
J
The first step of a more in-depth analysis of the performance of the
various candidate collection concepts involves investigating the changes in
the overall value of the information collected at the limits of the
performance ranges associated with the various information requirements. The
purpose of such an analysis is to determine
the impact each collection requirement has over its performance
range on the value of information collected, and
the degree to which the various concepts overlap, that is, are-
indistinguishable.
The effect of the performance of each concept relative to each information
requirement is studied by re-calculating the overall value of the information
collected by
fixing the performance of the requirement under consideration first
at its minimum and then its maximum performance level, and
using the expected value performance estimates for all other
information requirements.
This provides the analyst insight into the factors that determine which
concepts are the most effective.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
A
OR
sp
W
W
W
W,
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The results of such a performance analysis are summarized in the
adjoining table. Each entry represents the impact relative to a particular
collection concept of evaluating a specific information requirement at either
the minimum or maximum (versus expected) level of concept performance. There
are a couple of observations of note.
- The impact of some of the information requirements seems to be
independent of whether collection concept performance is fixed at
its minimum, expected, or maximum level of performance. Examples
are:
- number of bedrooms
- location relative to work
- disaster vulnerability
- down payment
- basic monthly payment
- projected interest variance
- possible assessments
- expected appreciation
When this occurs, a more detailed analysis to determine the exact
way in which collection concept performance varies between its
minimum and its maximum levels is unnecessary; it's adequate to fix
its performance at the expected level.
The four candidate concepts have different ranges of information
value:
(1)
Talking with owner
56-59%
(2)
Talking with the city
20-26%
(3)
Talking with friend
57-63%
(4)
Checking newspaper
12-18%
The
range
of values for concept #2 and for concept #4 in no way
overlaps that of any of the other concepts. However, concept #1 and
#3 overlap considerably. Usually talking with the friend seems to
have a higher return than talking with the owner, but not always
(see "age of house" for example).
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
INCORPORATION OF OTHER FACTORS
HUGHES
THE EVALUATION OF COLLECTION CONCEPT PERFORMANCE
MUST INCLUDE TWO OTHER FACTORS
? TIMELINESS
? SURVIVABILITY
TIMELINESS
? THE IMPACT ON THE VALUE OF THE INFORMATION
COLLECTED DUE TO THE DELAYS IN IMPLEMENTING
THE CONCEPT OR DELAYS IN THE CONCEPT OF
PRODUCING THE DATA
SURVIVABILITY
? THE IMPACT ON THE VALUE OF THE INFORMATION
COLLECTED DUE TO THE LIKELIHOOD OF THE
COLLECTION CONCEPT ENDURING OR SURVIVING
BOTH ARE FUNCTIONS OF EITHER
? THE DECISION MAKER'S PERCEPTIONS, OR
? THE CONSENSUS RESULT OF GATHERING THE
OPINIONS OF EXPERTS SELECTED BY THE
DECISION MAKER
The evaluation of collection concept performance must include two other
factors: timeliness and survivability. While the overall value of the
information collected is the key discriminant between concepts, that value can
be impacted by delays in implementing it or by the length of time over which
it continues to provide information. The extent of that value impact is a
function of the performance of the concept as well as the decision maker's
evaluation of its importance.
A key concern in collection system analyses is the incorporation of,
revisit times or capabilities into the analyses. Revisit capabilities are
accounted for in one of two ways in the methodology. The first approach,
which is the most straightforward and easily applied, involves incorporating
revisits into the collection concept performance assessment. With this
approach the concept is viewed over time with the assessment of capability
including several collection time intervals or opportunities. In simple
terms, the question to be answered would be: Given the concept will visit
this often and have these attempts at collection, what is the probability you
will satisfy this particular information need? In this sense the concept
capability includes performance over time and the details of individual visits
or opportunities are suppressed in favor of an overall performance assessment.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
OR
two
IM
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The second alternative which can be applied to revisit questions
involves using discounting methods similar to those discussed under concept
survivability. In this approach each opportunity is analyzed individually and
multiple opportunities over time are combined using the discounting methods
discussed later in this section (see page 46). The application of this method
involves selecting an appropriate discount factor.
The selection of either the overall performance assessment or the
discounted assessment will depend upon analyst confidence and knowledge. The
ultimate goal, however, is an assessment of overall performance which the
analyst feels is believable and accurately reflects systems capabilities.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
N,
---------------
HUGHES
TIMELINESS
? GIVEN
V=VALUE Of INFORMATION NOW
N=AMOUNT OF TIME BEFORE IT CAN BE COLLECTED
Ft(N)=DISCOUNT FACTOR TO BE APPLIED TO
INFORMATION VALUE GIVEN A DELAY OF N
Pt(N)=PROBABILITY OF A DELAY OF N
? THEN
Et (V)=EXPECTED VALUE OF INFORMATION GIVEN
TIMELINESS CONSIDERATIONS
=V>Ft(j)*Pt(j)
j
When there is a gap between the point in time the need for some
information is assessed and when it is actually received, the value of what is
collected may be reduced. To evaluate collection concepts with different
delays, the decision maker must specify to what extent delays of various
durations impact the overall value of the information collected. This impact
is expressed as a discounting factor on the overall value of information.
Given
V = overall value of information relative to some decision
maker question
N = number of intervals that the collection of that
information is delayed
Ft(N) = timeliness discount factor associated with a delay of N
intervals
Vt(N) = discounted value of information given a delay of N
intervals
a V*Ft(N)
Typically, the discount factors, Ft(N),are expressed as elements in a
discount stream, i.e., if each period of delay corresponds to "p" percent
loss, then
Ft(N) [-.L-]
1+p
- 44 -
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
mom
^
^
d
^
^
u
^
A
^
01
P
KTIN
F
^
01
0
01
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
For example, a 5% discount factor for 4 periods would be
F (4) = 1 4
t [ 1+.05 J
= 0.8227
Since the decision maker may not be able to predict a priori the exact
number of intervals that the receipt of information may delayed, there is
usually a need to specify the probability of delays of various duration and to
evaluate the expected discounted value of information.
Given.
Pt(N) = probability of a delay of N intervals
then
Et(N) = expected value of information'discounted for timeliness
M
= E [Vt(j) * Pt(j)]
j=l
M
= V* Z [Ft(j) * Pt(j)I
j=l
Note that the probability distribution does not need to be specified in a
detailed way. It is sufficient to grossly estimate it over a handfull of
points; for example,
Pt(0) = 0.2
Pt(l) = 0.4
Pt(2) = 0.3
Pt(3) = 0.1
- 45 -
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
SURVIVABILITY
HUGHES
? GIVEN
V=VALUE OF INFORMATION NOW
N=AMOUNT OF TIME OVER WHICH IT WILL BE
COLLECTED
Fs(N)=ENHANCEMENT FACTOR TO BE APPLIED
FOR INFORMATION IN THE Nth PERIOD
Ps(N)=PROBABILITY OF AN ADDITIONAL N PERIODS
? THEN
Es(V)=EXPECTED VALUE OF INFORMATION GIVEN
SURVIVABILITY CONSIDERATIONS
k
V* I P (k) [I F (j)]
k S j=0 S
When information is collected over an extended period of time, the value
of what is collected may be enhanced. To evaluate collection concepts with
different survivabilities, the decision maker must specify to what degree
extended collection enhances the overall value of the information collected.
This impact is expressed as a compounding of value over the number of
intervals.
Given
V = overall value of information relative to some decision
maker question
N = number of intervals over which the collection of that
Fs(N)
then
Vs(N)
information continues
= relative value of some information in the Nth interval
enhance value of information over N intervals
N
_ I V*Fs(j)
j=0
N
we
M
w
P1
n
0
In other words, the decision maker must specify the incremental benefit of
each additional interval of collection. If there is none, then Fs(j) is set
^
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
to zero. As with timeliness, however, these compounding factors are typically
expressed as elements in a discount stream, i.e., if each interval of extended
collection corresponds to a "p" percent discount factor, then
Fs (N) = (1+p ] N
Since the decision maker may not be able to predict a priori the exact
number of intervals over which the receipt of information may continue, the
probability of survival over various durations usually must be specified and
the expected enhanced value of information evaluated.
Given
PS(N) = probability of surviving N intervals
then
Es(N) =
expected value of information enhanced for survivability
N [Vs(k) l
* Ps(k)
k=0 JJ
}
=
N
V * I {P (k) * I [F (j)]
r
s
S
k =O j=0 1
Again, the probability distribution does not need to be specified in a
detailed way. It is sufficient to grossly estimate it over a handful of
points. For example,
if additional information is discounted at a rate of 10%, and
Ps(0) =
0.4
Ps(l) =
0.3
Ps(2) =
0.2
Ps(3) =
0.1
then over 3
years
Es (3) = V * E {PS(k) * 2 [(l+.l k1}
k=0 j=0
V ' j (.4*1) + .3 (1+1.1) t '/- ^ k1T1.1
+ .1 * 1 + 1 + 1 + 1
1.1 (1.1)2 (1.1)3 }
(1.1)
- 47 -
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
ADJUSTMENTS TO JOHN'S ANALYSIS
-------------------
HUGHES
-------------------
IN ORDER TO GET THE INFORMATION FROM HIS REAL
ESTATE FRIEND, JOHN WILL HAVE TO WAIT AT LEAST
TWO WEEKS UNTIL HE COMES BACK FROM VACATION
JOHN FEELS THATTHIS INEFFICIENT USE OFTIME WILL
REDUCE THE VALUE OFTHE INFORMATION HE COLLECTED
BY THE FACTORS
1
1.050 =.95
ALL OTHER CONCEPTS HAVE COMPARABLE
TIMELINESS
SURVIVABILITY
THEREFORE THE PERFORMANCE ANALYSIS NEEDS TO
BE RECONSIDERED
In order to get the information from his friend in real estate, John
will have to wait at least two weeks until he comes back from vacation. Over
this period of time, John feels that the value of the information that will be
collected will degrade somewhat due to the fact that the house may go off the
market. Therefore he estimates that a delay of two weeks will discount the
value of what is collected by about 5%, for a discount factor, F, of
F = 1/1.05 = 0.95
This third collection concept is the only one expected to have such a delay;
John expects to be able to implement any of the others immediately.
To adjust John's analysis to reflect this delay in timeliness for the
third candidate collection concept, what is required is the reduction of all
the relative values associated with the information requirements by 0.95 since
by the time the information is collected it will have been discounted in value
go,
OR
P1
ME
PM
Pq
X
0
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
by about 5%. For example, the expected information value the third collection
concept is expected to collect for the first collection requirement, the
number of bedrooms, is recalculated by
relative value of information = (1/1.05) * (0.075) = 0.072
expected performance = 0.9
expected information value = (0.9) * (0.072) = 0.0648
When all of the expected information values have been recalculated, they are
again summed to determine the discounted expected value for the information
collected.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The adjoining tables show the effect that the discounting had on both
the performance factors associated with the third concept and on the
performance analysis as a whole. The expected performance of the third
concept, talking with the friend who is a real estate agent, typically is
reduced between 3 and 4 percent across all of the information requirements.
As for the comparison of the complete set of collection concepts, a check of
the performance ranges shows that while concepts 2 and 4 are still much lower
than 1 and 3, the choice between 1 and 3 is even less clear than before.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
P0,
rF
PR
PW
PP
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
ADJUSTED PERFORMANCE FACTORS
HUGHES
ORIGI
NAL
DISCO
UNTED
MIN.
MAX.
MIN.
MAX.
NUMBER OF BEDROOMS
.60
.61
.57
56
.57
59
SQUARE FOOTAGE
SCHOOLS (LOCATION)
.59
.59
.62
.61
.
.56
.
.59
WORK (LOCATION)
.59
.61
61
.57
57
.58
58
SHOPPING (LOCATION)
.60
.
.
.
AMOUNT OF YARD WORK
.60
.60
.57
.57
AGE OF HOME
.57
.64
.54
.61
CONDITION OF SIMILAR HOMES
.59
.62
.56
.59
DISASTER VULNERABILITY
.60
.61
.57
.57
DOWN PAYMENT
.60
.61
.57
.57
CLOSING COSTS
.59
.62
.56
.59
BASIC MONTHLY PAYMENT
.60 ,
.61
.57
.57
PROJECTED INTEREST VARIANCE
.60
.61
.57
.57
POSSIBLE ASSESSMENTS
.60
.61
.57
.57
EXPECTED APPRECIATION
.60
.61
.57
.57
PLANNED DEVELOPMENTS
.60
.61
11 .57
.58
PERFORMANCE RANGES
---------------
HUGHES :
INFORMATION SOURCE
MIN.
VALUE
MAX
VALUE
TALK WITH OWNER
.56
.59
TALK WITH CITY
.20
.26
TALK WITH FRIEND (AGENT)
.54
.61
CHECK NEWSPAPER
.12
.18
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
STOCHASTIC DOMINANCE
HUGHES
? A CONCEPT IS COMPLETELY DOMINATED IF THERE
IS AT LEAST ONE OTHER CONCEPT HAVING ALL ITS
MINIMUM INFORMATION VALUES GREATER THAN ANY
MAXIMUM INFORMATION VALUE FOR THE
DOMINATED CONCEPT
? SUCH CONCEPTS NEED NO FURTHER
CONSIDERATION
J
In analyzing the competing collection concepts, there occasionally will
be a case where the performance of one of the candidate concepts is so low
relative to other concepts that it no longer needs to be considered. A
concept, A, is said to be stochastically or probabilistically dominated if no
matter what values the collection performance variables take, there is at
least one other concept, B, such that for every information requirement, Rj,
the minimum performance range associated with B exceeds the maximum
performance associated with A. Since the goal is to find the single most
effective concept, no further consideration needs to be given to a dominated
concept.*
As shown in the preceeding table on performance ranges, concepts 2 and 4
are completely dominated by concepts 1 and 3. Therefore John will no longer
consider concepts 2 and 4 in his analysis.
*Except if the ultimate use is in cost-effectiveness trade off. However, this
is not the problem presently being considered.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
W
IN
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The ultimate goal of this methodology should not be the achievement of a
stochastically dominant solution. While this is desirable from the decision
maker's viewpoint, his task is made easier, the non-linear aspects of the
problem does not quarantee the existence of a dominant solution. Therefore,
stochastic dominance is not used as a.goal for solution selection, but instead
serves as a means of problem simplification by allowing for the identification
and elimination of totally unacceptable solutions. If a single, dominant
solution is not found, the decision maker must then view the decision in terms
of required levels of performance, as well as risk aversion or acceptance
preferences. Using these two measures, alternatives which do not dominate
each other can be distinguished and a solution selected.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
PROBABILISTIC ANALYSIS
------------------
HUGHES
? A MONTE CARLO APPOACH IS USED TO DETERMINE
THE COMPLETE.PERFORMANCE SPECTRUM FOR EACH
CONCEPT
? PROCEDURE
- PERFORMANCE DISTRIBUTION RELATIVE TO EACH
DATA REQUIREMENT ARE DEVELOPED FOR EACH
NON-DOMINATED CONCEPT
- GIVEN THE RELATIVE VALUE OF EACH DATA
REQUIREMENT, THE PROBABILITY OF THE
INFORMATION VALUE OBTAINED FOR EACH
CONCEPT IS DETERMINED BY A MONTE CARLO
ANALYSIS USING THOSE PERFORMANCE
DISTRIBUTION
- ALLOWS CONFIDENCE ASSESSMENT
IN
IN
a
w
w
R
rq
To further compare the remaining concepts, it is necessary to get more
insight into how the value of the information they collect varies over the
performance ranges relative to each information collection requirement. This
is done by developing distributions that describe the probability of achieving
various values of information. These will allow the analyst to do confidence
assessments.
The first step in performing a probabilistic analysis is to roughly
determine the performance distribution relative to each information
requirement for each candidate collection concept still under consideration.
The specification of these distributions can be a simple and as gross as
merely specifying a handful of points (for example, the minimum performance
level, the 10th percentile, 30th percentile, the 50th percentile (expected
performance), 70th percentile, 90th percentile, and the maximum performance
level). In John's example, all the performance distributions were assumed to
be uniform, that is, given
a minimum performance level
b maximum performance level
w
M
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
then if P(v) is the probability of collecting information with at least a
value of v,
P(v) = (v-a)/(b-a), a s v s b
P(v) = 0, otherwise
and if p(v) is the probability that the information collected had exactly a
value of
v,
p(v) =
p(v) =
1/(b-a), a s v s b
0, otherwise
Another distribution which also could have been selected is the normal
distribution. The key, however, is that it is not necessary to specify a
"classic" distribution if there is not one that represents the collection
concept's performance relative to a particular information requirement; a
simple estimate based on expert judgement is sufficient.
It is important to note that while these performance distributions can
be developed for every information requirement, those requirements which the
performance analysis has shown to have virtually no impact on the overall
value can simply be left at their expected value. This saves the analyst the
extra work of defining distributions for those requirements.
Once all the necessary performance distributions have been defined, they
are used with Monte Carlo techniques to derive the probabilistic performance
of the value of the information collected for each remaining candidate
collection concept. The basic approach is to select samples from the
performance distributions, weight them with the relative value of the
associated information requirement, adjusted to reflect any timeliness or
survivability considerations, and add them all together to get a sample of the
distribution for the value of the information collected. More formally,
given
Xj(i), the ith sample from the performance of jth information
requirement,
Vj, the relative value of the jth information requirement,
n, the number of information requirements
Yi, the ith sample from the information value distribution
then
n
Yi = E Vj * Xj(i)
j=1
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The adjoining graphs illustrate the results of the probabilistic
analysis for John's problem. The top figure is a graph of the probability of
achieving a guaranteed value of information. Note that although the curves
are close, there is a difference which influences which one should be
selected. If John is risk-adverse, he should talk with the owner because
there is less chance of getting the lowest possible values; if however John is
risk-immune, he should instead talk with his friend because there is a chance
he will collect a higher value.
The differences between the two concepts are even more clearly
highlighted in the lower figure. Talking with the owner brings more
consistent results, there is less variance. Talking with a friend could just
as easily give you a lower information value as a higher relative to that
which can typically be achieved by the other concept.
0-
w
ter
W
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
LIKELIHOOD OF COLLECTION
PROBABILITY OF 0.6
AT LEAST
THIS VALUE
0.4
TALK WITH OWNER
TALK WITH FRIEND
-------------------
HUGHES :
0.50 0.55 0.60 0.65
RELATIVE VALUE OF INFORMATION
PERFORMANCE COMPARISON
TALK WITH OWNER
TALK WITH FRIEND - -
HUGHES
------------------
RELATIVE VALUE OF INFORMATION
- 57/58
(Reverse Blank)
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
DECISION MAKER REVIEW
HUGHES
AT ANY POINT IN THE ANALYSIS, THE ENTIRE PROCESS
ACCOMPLISHED TO THAT POINT IS COMPLETELY
ACCESSIBLE TO
? REVIEW
? QUESTIONING
? RE-ASSESSMENT
ALL ASSUMPTIONS ARE OUT IN THE OPEN AND THE
ANALYSIS PATHS ARE READILY TRACEABLE
One of the strengths of the methodology is the generation of traceable
paths through the analysis. Since every assumption is documented and the-
analytic process is straightforward and completely visible, the study is
accessible at any point for review. This means the decision maker does not
need to rely solely on an intuitive understanding of the problem which is
especially important when the results are to be communicated and perhaps
defended to others. The methodology itself provides a structure for
discussing the analysis and the results and, if necessary, for defining at
what point re-assessment may be necessary. Additionally, if re-evaluation is
required, any new assumptions can be readily substituted for old with
retention of much of the remaining structure.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
To purchase a home, John is going to need financial assistance from his
parents; so when he finished his collection concept analysis, he took it to
his parents for review. Overall, they understood and agreed with what John
had done. They did, however, feel that if this house was to be in a sense an
investment for them as well, the weights on the information value tree should
reflect more of an emphasis on the investment issues. Their recommendation
was that the consensus tree should be recalculated, but this time with the
following weights
- 3 for Sue's values
- 2 for Mary's values
1 for Gary's values
The revised calculation of the consensus weights is shown in the adjoining
table. The basic formula used was.
W*(i) = ith consensus weight
= (1/6) * [2 * (Mary's ith weight) +
3 * (Sue's ith weight) +
1 * (Gary's ith weight) ]
rr
0
0
0
Q
IN
in
p
a
01
11
w
- 60 -
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
REVIEW OF JOHN'S ANALYSIS I
HUGHES
AFTER JOHN COMPLETED HIS ANALYSIS HE SHOWED IT
TO HIS PARENTS SINCE HE NEEDS THEIR HELP TO
FINANCE SUCH A PURCHASE
IN GENERAL THEY AGREED WITH WHAT HE HAD DONE.
THEY FELT HOWEVER THAT THE WEIGHTS ON JOHN'S
TREE SHOULD BE ADJUSTED TO REFLECT MORE OF THEIR
EMPHASIS ON THE FUTURE WORTH OF THIS INVESTMENT
THEIR SUGGESTION WAS THATTHEIR JOINTCONSENSUS
TREE SHOULD BE CALCULATED WITH
3 FOR SUE'S WEIGHTS
2 FOR MARY'S WEIGHTS
1 FOR GARY'S WEIGHT
RELATIVE
MARY
SUE
GARY
CONSENSUS
TO
TREE
EXPERT PREFERENCES/WEIGHTS
2
3
1
FUNCTIONAL NEEDS
0.35
0.15
0.25
0.233
0.233
BIG ENOUGH
0.80
0.30
0.40
0.483
0.113
BEDROOMS
0.50
0.40
0.50
0.450
0.051
SO FOOTAGE
0.50
0.60
0.50
0.550
0.062
LOCATION
0.20
0.70
0.60
0.517
0.120
SCHOOLS
0.40
0.60
0.45
0.508
0.061
WORK
0.40
0.10
0.20
0.217
0.026
SHOPPING
0.20
0.30
0.35
0.275
0.033
SUPPORT REQUIRED
0.50
0.25
0.25
0.333
0.333
MAINTAINABILITY
0.80
0.70
0.50
0.700
0.233
AMT OF YARD WORK
0.40
0.30
0.40
0.350
0.082
AGE OF HOUSE
0.60
0.70
0.60
0.650
0.151
DURABILITY
0.20
0.30
0.50
0.300
0.100
CONDITION OF, SIMILAR
0.70
0.70
0.60
0.683
0.069
DISASTER VULNERABILITY
0.30
0.30
0.40
0.317
0.031
INVESTMENT POTENTIAL
0.15
0.60
0.50
0.434
0.434
IMMEDIATE AFFORDABILITY
0.45
0.20
0.40
0.317
0.138
DOWN PAYMENT
0.50
0.50
0.50
0.500
0.069
CLOSING COSTS
0.50
0.50
0.50
0.500
0.069
CONTINUING AFFORDABILITY
0.45
0.20
0.40
0.317
0.138
BASIC MONTHLY
0.40
0.40
0.35
0.392
0.054
INTEREST
0.40
0.40
0.40
0.400
0.065
PROJ. ASSESSMENTS
0.20
0.20
0.25
0.208
0.029
FUTURE WORTH
0.10
0.60
0.20
0.366
0.158
EXP. APPRECIATION
0.30
0.60
0.50
0.483
0.076
PLANNED DEVELOPMENT
0.70
0.40
0.50
0.517
0.082
HUGHES
REVISED
CALC U LATI,O N
OF
CONSENSUS
WEIGHTS
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
------------------
COMPARISON OF DATA PRIORITIES [1HUGHESJ
AGE OF HOUSE
AGE OF HOUSE
AMOUNT OF YARD WORK
PLANNED DEVELOPMENTS
SQUARE FOOTAGE
AMOUNT OF YARD WORK
CONDITION OF SIMILAR HOUSES
EXPECTED APPRECIATION
NUMBER OF BEDROOMS
DOWN PAYMENT
DOWN PAYMENT
CLOSING COSTS
CLOSING COSTS
CONDITION OF SIMILAR HOUSES
LOCATION RELATIVE TO SCHOOLS
SQUARE FOOTAGE
INTEREST VARIANCE
LOCATION RELATIVE TO SCHOOLS
BASIC MONTHLY PAYMENT
INTEREST VARIANCE
PLANNED DEVELOPMENTS
BASIC MONTHLY PAYMENT
DISASTER VULNERABILITY
NUMBER OF BEDROOMS
EXPECTED APPRECIATION
LOCATION RELATIVE TO SHOPPING
LOCATION RELATIVE TO WORK
DISASTER VULNERABILITY
LOCATION RELATIVE TO SHOPPING
POSSIBLE ASSESSMENTS
POSSIBLE ASSESSMENTS
LOCATION RELATIVE TO WORK
The revised consensus trees, both the basic (with branch weights
relative to the preceeding node) and the modified (with branch weights
relative to the level), are presented on the next page in comparison to the
original consensus trees. Note how the emphasis has moved towards the
investment issues in general and the future worth in particular.
In the above table are ranked lists of collection requirements, one
reflecting the original analysis, one reflecting this revised analysis. Some
of the collection requirements have been significantly re-ordered, especially
- square footage
- number of bedrooms
- planned developments
- expected appreciation
IN
11
11
^
^
^
^
P
n
P
F
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
01 -IN
-------------------
REVISED CONSENSUS TREE HUGHES
------------------
BEDROOMS
0
7
SO. FOOTAGE
4
5
0.460
0525
0.660
SCHOOLS
0.163
0.608
WORX
0.276
0.217
SHOPPING
0.262
0.275
YARD
0.375
0.360
wGE OF HOME
0.826
0.660
CONDITION
0.626
0.683
DISASTER VUL.
0
326
.
0.317
DOWN
0.600
0.600
CLOSING
0.600
0.600
BASIC MONTHLY
0.388
0.392
INTEREST VAR.
1
0.100
0400
POSS. ASSESSMENT
212
0
POSS. ASSESSMENT
.
0.208
EXP APPRECIATION
0.425
PLANNED DEV.
0.575
BRANCH VALUES RELATIVE TO
EACH NODE
--------------------
HUG
REVISED CONSENSUS TREE HES
BEDROOMS
0
075
0
061
BIG ENOUGH
F
.
.
SO.
OOTAGE
0168
0.083
0.062
SCHOOLS
0.061
0.061
LOCATION
WORK
0
032
117
0
.
0.026
.
SHOPPING
0.031
0.033
YARD
MAINTAINABILITY
0.098
YARD
0.082
MAINTAINABILITY
AGE OF HOME 0.262
161
0
AGE OF HOM
0.233 0
161
.
.
CONDITION
DURA8ILITY
0.076
CONDITION
0.069
DISASTER VUL.
0.113
0.037
DISASTER VUL.
0.031
DOWN
IMM. AFFORO.
0.0666
0.069
CLOSING
0.131
0.0656
0.069
BASIC MONTHLY
0.061
0.064
CONT. AFFORD I
INTEREST VAR.
052
0
0
066
131
0
.
.
.
POSS. ASSESSMENT
0.026
0.029
EXP. APPRECIATION
0
076
FUTURE WORTH
0.037
.
0 088 PUNNED OEV.
0.061
0.082
BRANCH VALUES RELATIVE TO
EACH LEVEL
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Having completed the revised consensus tree, John can now procede to the
assessment of the potential collection concepts. Since his parents had no
disagreement with the information performance factors that were developed in
the original analysis, the revised performance analysis follows much as
before: relative to each collection concept, the revised relative values of
information for each information requirement are multiplied by the expected
performance of the concept and the resulting expected information values are
summed to determine the overall expected information value collected. The
results of this analysis are shown in the top figure of the next page. Once
again it appears talking with the city or checking the newspaper fall short;
this time, however, the margin between talking with his real estate friend and
talking with the owner is greater. So the preliminary assessment is the third
collection concept appears to be the most effective.
To investigate this preference more thoroughly, John again considered
the impact on the overall value of the information collected relative to each
collection concept by evaluating each information requirement at its minimum
and maximum levels of performance. The results, shown in the figure at the
bottom of the next page, indicate that talking with the real estate friend
appears to stochastically dominate all the other candidate collection
concepts, however the issue of timeliness remains to be accounted for.
01
r
^
Ii
0
0
F.
^
^
PI
PI
^
AW~M
-64-
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
EXPECTED PERFORMANCE
----- ---------
HUGHES
INFORMATION SOURCE
EXP. VALUE
TALK WITH THE OWNER
0.53
TALK WITH THE CITY
0.26
TALK WITH FRIEND (AGENT)
0.61
CHECK NEWSPAPER
0.14
REVISED PERFORMANCE
ANALYSIS
HUGHES
TALK WITH
TALK WITH
TALK WITH
FRIEND
CHECK
OWNER
CITY
(AGENT)
NEWSPAPER
MIN
MAX
MIN
MAX
MIN
MAX
MIN
MAX
NUMBER OF BEDROOMS
0.53
0.53
0.26
0.26
0.61
0.62
0.14
0.14
SQUARE FOOTAGE
0.53
0.53
0.26
0.26
0.60
0.62
0.13
0.15
SCHOOLS (LOCATION)
0.52
0.54
0.25
0.27
0.60
0.62
0.13
0.14
WORK (LOCATION)
0.53
0.53
0.26
0.26
0.60
0.62
0.14
0.14
SHOPPING (LOCATION)
0.52
0.54
0.26
0.26
0.60
0.62
0.13
0.14
AMOUNT OF YARD WORK
0.52
0.54
0.26
0.26
0.61
0.61
0.14
0.14
AGE OF HOME
0.52
0.54
0.23
0.29
0.58
0.64
0.11
0.16
CONDITION OF SIMILAR HOMES
0.52
0.55
0.26
0.26
0.60
0.63
0.14
0.14
DISASTER VULNERABILITY
0.53
0.54
0.26
0.26
0.61
0.62
0.14
0.14
DOWN PAYMENT
0.53
0.53
0.26
0.26
0.61
0.62
0.14
0.14
CLOSING COSTS
0.53
0.53
0.26
0.26
0.60
0.63
0.14
0.14
BASIC MONTHLY PAYMENT
0.53
0.53
0.26
0.26
0.61
0.62
0.14
0.14
PROJECTED INTEREST VARIANCE
0.53
0.53
0.26
0.26
0.60
0.62
0.14
0.14
POSSIBLE ASSESSMENTS
0.53
0.54
0.26
0.26
0.61
0.62
0.14
0.14
EXPECTED APPRECIATION
0.52
0.54
0.25
0.27
0.60
0.62
0.14
0.14
PLANNED DEVELOPMENTS
0.52
0.54
0.25
0.27
0.60
0.62
0.13
0.14
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 200.3/09/29 : CIA-RDP86B00269R001300050001-1
As in the original analysis, John must adjust the assessment of the
third collection concept, talking with the real estate friend, to reflect the
two week delay in collecting any information. The same estimate is used for
the discount rate, 5%, which is a factor of (1/1.05) or approximately 0.95.
This translates into a reduction of about 3 to 4% in the performance factors
for this collection concept as shown in the figure at the top of the next page.
A review of the ranges for the performance factors associated with each
of the collection concepts, adjusted for timeliness considerations (see bottom
figure on next page), indicates that while the concepts of talking with the
city and checking in the newspaper are indeed stochastically dominated, there
is a slight'overlap for the other two. So before John completes his analysis,
he will do a probabilistic analysis on these two remaining concepts.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
REVISED PERFORMANCE
ADJUSTMENTS
HUGHES
ORIGINAL
DISCOUNTED
MIN.
MAX.
MIN.
MAX.
NUMBER OF BEDROOMS
.61
.62
.58
59
SQUARE FOOTAGE
.60
.62
.57
.
59
SCHOOLS (LOCATION)
.60
.62
.57
.
59
WORK (LOCATION)
.60
.62
.57
.
.59
SHOPPING (LOCATION)
.60
.62
.57
.59
AMOUNT OF YARD WORK
.61
.61
.58
.58
AGE OF HOME
.58
.64
.55
61
CONDITION OF SIMILAR HOMES
.60
.63
.57
.
59
DISASTER VULNERABILITY
.61
.62
.58
.
.58
DOWN PAYMENT
.61
.62
.57
59
CLOSING COSTS
.60
.63
.57
.
59
BASIC MONTHLY PAYMENT
.61
.62
.58
.
.59
PROJECTED INTEREST VARIANCE
,
.60
.62
.57
59
POSSIBLE ASSESSMENTS
.61
.62
.58
.
59
EXPECTED APPRECIATION
.60
.62
.57
.
59
PLANNED DEVELOPMENTS
.60
.62
.57
.
.59
REVISED PERFORMANCE RANGES
HUGHES
INFORMATION SOURCE
MIN.
VALUE
MAX.
VALUE
TALK WITH OWNER
.52
.55
TALK WITH CITY
.23
.29
TALK WITH FRIEND (AGENT)
.55
.61
CHECK NEWSPAPER
.11
.16
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
The revised results of the probabilistic analysis are shown in the
figures on the adjoining page. This time there is no question of which
concept has the most effective performance. As the top figure shows, talking
with the real estate friend consistently yields a likelihood for collecting a
higher overall value of information. It is important to note, as the lower
figure shows, this is completely independent of the range of performance of
the two concepts, which was not changed; the third concept still has a greater
possible variance in its performance. Because the relative weights of the
information collected have changed, however, the overall expected performance
of the two concepts has diverged, leaving the third concept the clear choice.
91
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 4
w
4
4
4
^
4
4
^
R
4
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
REVISED LIKELIHOOD OF COLLECTION I
PROBABILITY OF
AT LEAST 0.6
THIS VALUE
REVISED PERFORMANCE COMPARISON
0.45 0.50 0.55 0.60
RELATIVE VALUE OF INFORMATION
0.45
TALK WITH OWNER
TALK WITH FRIEND - -
RELATIVE VALUE OF INFORMATION
HUGHES
TALK WITH OWNER
TALK WITH FRIEND
-------------------
HUGHES
- 69/70 -
Reverse B1 1~
Approved For Release 2 03/09/29 : C~i-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
CONTINUING APPLICABILITY
HUGHES
SIGNIFICANT AMOUNTS OF DATA ARE COMPILED TO
SUPPORT THE METHODOLOGY, INCLUDING
? KEY PROBLEM ISSUES
? CONCEPT PERFORMANCE ESTIMATES
? TIMELINESS AND SURVIVABILITY ADJUSTMENT
FACTORS
LARGE PORTIONS OF THAT DATA MAY BE REUSABLE
? RELATED PROBLEMS MAY USE SOME - EITHER
DIRECTLY OR AS A GUIDELINE
? FOLLOW-ON WORK FOR PROBLEM UNDER STUDY MAY
USE SUBSTANTIAL PORTIONS
As is readily apparent, the application of this methodology to
collection systems problems is not a trivial exercise. Typical of all
decision analysis approaches, it requires the compilation of a significant
amount of data, including
- key issues to be investigated, including the information required to
support the process and each piece's relative weight
- evaluation of experts
- collection concept performance -- not just at the likely or expected
level but also minimum and maximum levels and, in some cases, an
approximation of the entire performance distribution
quantification of impacts of discounting effects due to timeliness
considerations and compounding effects due to survivability
considerations
As such, this approach should be reserved for large, resource-critical
decisions.
However, the usefulness of much of this data does not end with the
study. Related problems may be able to use portions of the performance
estimates or large subsections of the tree either intact or as a template or
prototype. Similarly, there may be follow-on applications that reconsider
some of the basic elements and leave the rest essentially the same. Examples
might be the evaluation of an additional concept or the availability of a new
expert or the analysis of the same problem but from a different point in time.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Instead of initiating an information collection effort at the time the
original study was done, John just did some preliminary investigation and
delayed collecting everything else. A month later he decided to go ahead and
seriously analyze the house in question. But the evaluation he would have
done before was not completely correct any longer as he had already collected
a certain amount of information (such as the number of bedrooms, the square
footage, the relative distances, etc.). Therefore, collecting it again had
significantly reduced importance. So he went back to his experts for updated
weights. New consensus weights were calculated using the same evaluation of
the experts as was done for the revised weights
- 3"for Sue's value
- 2 for Mary's value
- 1 for Gary's value
The updated weights supplied by the experts as well as the new consensus
weights are shown in the top figure on the next page. These weights produce a
new ordering of information. Again, a signficant re-ranking has taken place,
especially for
- age of house
- amount of yard work
- square footage
- number of bedrooms
- planned developments
- expected appreciation
- disaster vulnerability
- possible assessments
u
on
M1
Approved For Release 2003/099 :_CIA-RQP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
UPDATED CALCULATION OF
CONSENSUS WEIGHTS
-------------------
HUGHES
RELATIVE
TO
MARY
SUE
GARY
CONSENSUS
TREE
EXPERT PREFERENCES/WEIGHTS
2
3
1
-
FUNCTIONAL NEEDS
BIG ENOUGH
0.15
0.05
0.10
M092
0.092
BEDROOMS
0.20
0.10
0.10
0.133
0.012
SO. FOOTAGE
0.20
0
80
0.10
0.10
0.133
0.002
LOCATION
.
0.90
0.90
0.876
0.010
SCHOOLS
0.80
0.90
0.90
0.867
0.080
WORK
0.40
0.60
0.45
0.508
0.041
SHOPPING
0.40
0.10
0.20
0.217
0.017
0.20
0.30
0.35
0.275
0.022
SUPPORT REQUIRED
0.35
0.15
0.20
0
225
0
225
MAINTAINABILITY
0.40
0.30
0.20
.
0
317
.
0
071
AMT OF YARD WORK
0.60
0.60
0.60
.
0
600
.
0
043
AGE OF HOUSE
DURABILITY
0.40
0.40
0.90
.
0,400
.
0.028
CONDITION OF SIMILAR
0.60
0
50
0.70
40
0
0.80
0
35
0.683
0.154
DISASTER VULNERABILITY
.
0
50
.
.
0.425
0.065
.
0.60
0.65
0.575
0.089
INVESTMENT POTENTIAL
0.50
0.80
0.70
0
683
0
683
IMMEDIATE AFFORDABILITY
0.45
0.20
0.40
.
0.317
.
216
0
DOWN PAYMENT
0.50
0.50
0.50
0
500
.
0
108
CLOSING COSTS
CONTINUING AFFORDABILITY
0.50
0.50
0.50
.
0.500
.
0.108
BASIC MONTHLY
0.45
0.35
0.20
0.30
0.40
0
30
0.317
0
317
0.216
INTEREST
PROJ ASSESSMENTS
0.35
0.30
.
0.35
.
0.325
0.069
0.070
FUTURE GROWTH
0.30
0
10
0.40
0.35
0.358
0.079
EXP APPRECIATION
.
0
30
0.60
0
60
0.20
0
50
0.366
0.251
PLANNED DEVELOPMENT
.
0.70
.
0.40
.
0
50
0.493
0
517
0.121
.
.
0.130
- - --' ... _
-
UPDATED COMPARISON OF
DATA
HUGH
ES
PRIORITIES =-----------------
INFORMATION REQUIREMENT
ORIGINAL
REVISED
UPDATED
AGE OF HOUSE
AMOUNT OF YARD WORK
SQUARE FOOTAGE
CONDITION OF SIMILAR HOUSES
1
2
3
4
1
3
8
7
12
10
15
9
NUMBER OF BEDROOMS
DOWN PAYMENT
CLOSING COSTS
LOCATION RELATIVE TO SCHOOLS
5
6
7
8
12
5
6
9
16
3
4
11
INTEREST VARIANCE
BASIC MONTHLY PAYMENT
PLANNED DEVELOPMENTS
DISASTER VULNERABILITY
9
10
11
12
10
11
2
14
7
8
1
5
EXPECTED APPRECIATION
LOCATION RELATIVE TO WORK
LOCATION RELATIVE TO SHOPPING
POSSIBLE ASSESSMENTS
13
14
15
16
4
16
13
15
2
14
13
6
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
With the updated consensus tree, John procedes to the collection concept
evaluation phase. Assuming the same set of collection concepts with the same
performance characteristics, an analysis is done of the updated performance,
generating overall expected values for the information collected for each
concept. The results are presented at the top of the next page. This time
talking with the friend appears to be significantly more effective than any
other concept.
To verify that talking with the friend is the concept to be chosen, John
considered for a third time the impact on overall value of information
collected relative to each collection concept from evaluating each information
requirement'at its minimum and maximum levels of peformance. The results,
shown in the figure at the bottom of the next page, demonstrate unequivically
that the third concept stochastically dominates the entire set. Therefore,
there is no need for further analysis.
----Approved For--Re1ease--220830- : -GM RDP86BOO269ROO1300 0001-i '' - -
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
UPDATED EXPECTED PERFORMANCE
HUGHES
INFORMATION SOURCE
EXP. VALUE
TALK WITH THE OWNER
0.35
TALK WITH THE CITY
0.29
TALK WITH FRIEND (AGENT)
0.62
CHECK NEWSPAPER
0.07
UPDATED PERFORMANCE ANALYSIS I
:HUGHES.:
TALK WITH
TALK WITH
TALK WITH
FRIEND
CHECK
OWNER
CITY
(AGENT)
NEWSPAPER
MIN
MAX
MIN
MAX
MIN
MAX
MIN
MAX
NUMBER OF BEDROOMS
0.35
0.35
0.29
0.29
0.62
0.62
0.07
0.07
SQUARE FOOTAGE
0.35
0.35
0.29
0.29
0.62
0.63
0.06
0.07
SCHOOLS (LOCATION)
0.34
0.35
0.29
0.30
0.62
0.63
0.06
0.07
WORK (LOCATION)
0.34
0.35
0.29
0.29
0.62
0.63
0.07
0.07
SHOPPING (LOCATION)
0.34
0.35
0.29
0.29
0.62
0.63
0.06
0.07
AMOUNT OF YARD WORK
0.34
0.35
0.29
0.29
0.62
0.62
0.07
0.07
AGE OF HOME
0.35
0.35
0.29
0.30
0.62
0.63
0.06
0.07
CONDITION OF SIMILAR HOMES
0.33
0.36
0.29
0.29
0.61
0.64
0.07
0.07
DISASTER VULNERABILITY
0.34
0.35
0.28
0.30
0.61
0.64
0.07
0.07
DOWN PAYMENT
0.35
0.35
0.29
0.29
0.61
0.64
0.07
0.07
CLOSING COSTS
0.35
0.35
0.29
0.29
0.60
0.65
0.07
0.07
BASIC MONTHLY PAYMENT
0.35
0.35
0.29
0.29
0.62
0.63
0.07
0.07
PROJECTED INTEREST VARIANCE
0.35
0.35
0.29
0.29
0.61
0.64
0.07
0.07
POSSIBLE ASSESSMENTS
0.33
0.36'
0.28
0.30
0.61
0.64
0.06
0.07
EXPECTED APPRECIATION
0.33
0.36
0.28
0.30
0.61
0.64
0.07
0.07
PLANNED DEVELOPMENTS
0.33
0.37
0.28
0.30
0.61
0.64
0.06
0.07
- 75/76 -
Approved For Release 2 Ri YY/ P6R1DP86B00269R001300050001-1
 . Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
IV. DEMONSTRATION EFFORT
DEMONSTRATION EFFORT
-- ----------------
HUGHES
-------------------
PROBLEM SELECTION CRITERIA
? CONSENSUS PROBLEM
? IMPORTANCE OF PROBLEM MUTUALLY AGREED
UPON
? MUST DEMONSTRATE ADAPTABILITY OF
METHODOLOGY
? POSSESS A DEGREE OF COMPLEXITY TO EXERCISE
ALL FACETS OF METHODOLOGY
? MUST ALLOW FOR APPLICATION OF SENSITIVITY
ANALYSIS
For the methodology developed to be of use to the analyst, it must be
easily applied to the problems with which he/she is concerned. The preceding
"toy" example is useful for illustration and education, but a demonstration
effort on a more relevant problem to the collection concept analyst is
needed. Care should be taken in choosing the topic of this effort so that
maximum benefit may be achieved.
First of all, the problem should be of enough importance to warrant the
effort undertaken and be of relevance to the community. The problem should
have sufficient complexity to fully utilize and illustrate the documented
methodology. Finally, the problem should be such that sensitivity analysis
can be meaningfully employed and be used to show the conclusions and results
such an effort makes possible.
In summary, the demonstration effort should show the power and
usefulness of the methodology as well as its flexibility and adaptability.
The remainder of this report documents what has been completed to date
relative to this effort.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
IV. A. SAM PROBLEM
SAM PROBLEM ; HUGHES
-- -- -- ---- --
WHAT EFFECT WILL THE INTRODUCTION OF
A SURFACE TO AIR MISSILE HAVE ON
THE BALANCE OF POWER?
The final stage of this analysis involves applying the developed
methodology to a problem of interest to the intelligence community. By
documenting this process, an example is available to the-analyst to use-as a
guideline when performing additional analyses relating to collection concept
concerns.
The problem chosen for illustration involves the information required to
answer the question: What effect will the introduction of a surface to air
missile (SAM) have on the balance of power? The point to keep in mind here is
the frame of reference for the analysis. The methodology does not attempt to
answer the.question regarding the SAM and its effects, but instead the
methodology determines the information required to answer the question. Given
these requirements, the ability of a particular collection concept to collect
the desired information can be evaluated and the rest of the sensitivity
analysis completed.
As of_the_wr1 tang of this interim report, the SAM problem has been
completed through the structuring of the information tree the structure is
illustrated on the following pages. The remainder of the analysis will be
documented in the final report of this study.
rM
go
M
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
TOP LEVEL TREE r HUGHES
CRITICAL ISSUES
UNMANNED
COUNTERS AGAINST N~
THIS SYSTEM
INCREASES
IN SYSTEM
VULNERABILITY
SIGNATURE MODIFICATION
CRUISE MISSLE
NEN TRAJECTORY CAFABILITY
RECONNAISSANC II NFR1
The top level tree for the SAM problem is shown above. The level of
information is seen to get more specific with descent through the tree. For
example, the first division is between direct threats posed by the system and
modifications that could be undertaken to regain the balance of power. The
tree expands in detail until the raw or collectable data level is reached.
The tree structure in its entirety is illustrated on the following pages and
is indexed using Roman numerals. The numerals refer to the figure number on
which the continuation of the tree can be found. For example, the expansion
of the homeland deployment tree can be found on Figure VIII.
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
I. CRUISE MISSILE
INTEGRATED
SYSTEM W/
HUGHES
RADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
XIX
RADAR CAPABILITY XVI11
C3 CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
XIX
II. UNMANNED RECONNAISSANCE 1
------------------
HUGHES
RADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
MISSILE CAPABILITY XX
?C3 CAPABILITY XIX
RADAR CAPABILITY Xv
?MISSILE CAPABILITY XX
-RADAR CAPABILITY
?C3 CAPABILITY
RADAR CAPABILITY (ED)
C3 CAPABILITY XIX
MISSILE CAPABILITY XX
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
III. BOMBER
INTEGRATED
SYSTEM W/
E
-------------------
HUGHES
------------
RADAR CAPABILITY
C3 CAPABILITY
ERADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
E
RADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
ERADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
XVIII
XIX
XIX
XX
XVI11
HUGHES
IV. MANNED RECONNAISSANCE
INTEGRATED
SYSTEM W/
RADAR CAPABILITY
C3 CAPABILITY
EMISSILE CAPABILITY
E ADAR CAPABILITY
3 CAPABILITY
ISSILE CAPABILITY
E
C3 CAPABILITY
ERADAR CAPABILITY
C3 CAPABILITY
MISSILE CAPABILITY
xVlu
XIX
X V 111
XIX
XVIII
XIX
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
V. HARD COUNTERS -ARM HUGHES
L VULNERABILITY TO
ACQUISITION
RADAR
VULNERABILITY -
C DEPLOYMENT
ANTENNA
INTRINSIC
SIGNATURE
ECM CAPABILITY
DEPLOYMENT
ANTENNA
VULNERABILITY TO INTRINSIC
SIGNATURE
ACQUISITION
ECM CAPABILITY
Vi. SOFT COUNTER - ECM HUGHES
? JAMMING
-SPOOFING
AREA OF INFLUENCE
CCM TECHNIQUES
J
?DEGRADED MODES
-SUSCEPTIBILITY TO INTERCEPT
VULNERABILITY TO JAMMING
VULNERABILITY TO SPOOFING
MAXIMUM TRACK RATE
GIMBAL LIMITS
ERROR MEASUREMENT
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29: CIA-RDP86B00269R001300050001-1
VII. SOFT COUNTER - TACTICS
------------------
HUGHES
FORMATION MODIFICATIONS
TRAJECTORY MODIFICATIONS
LOCK ON RANGE
LOW ALTITUDE CAPABILITY
RECEDING TARGET CAPABILITY
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
VIII. HOMELAND DEPLOYMENT HUGHES
HOMELAND
DEPLOYMENT
IX. WARSAW PACT DEPLOYMENT
E
HUGHES
TRAINING
PHYSICAL DEPLOYMENT
EQUIPMENT
BARRIER DEFENSE PHYSICAL DEPLOYMENT
MANUFACTURING RATE
E: C MANUFACTURING
E CREW AVAILABILITY
TRAINING
1
TRAINING
PHYSICAL DEPLOYMENT
EEGUIPMENT
L
rM TJIIAL UtrLUYMtNT
EQUIPMENT
OPERATIONAL
UNDER CONSTRUCTION
E MANUFACTURING RATE
DOCTRINE
CREW CREW AVAILABILITY
` TRAINING
-EQUIPMENT
r
r
im
^
r
^
P
111
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
X. OTHER DEPLOYMENT
------------------
HUGHES
-TRAINING
PHYSICAL DEPLOYMENT
-EQUIPMENT
fPHYSICAL DEPLOYMENT
EQUIPMENT
OPERATIONAL
UNDER CONSTRUCTION
E MANUFACTURING RATE
DOCTRINE
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
XI. CRUISE MISSILE
SIGNATURE
MODIFICATION
NEW TRAJECTORY
CAPABILITY
INTEGRATED SYSTEM
W/
INTEGRATED SYSTEM
W/
-------------------
HUGHES ,
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
XII. UNMANNED RECONNAISSANCE
SIGNATURE
MODIFICATION
UNMANNED
RECONNAISSANCE
HUGHES
SOLE SYSTEM RADAR CAPABILITY
MISSILE CAPABILITY XX
INTEGRATED SYSTEM
W/
NEW TRAJECTORY
CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
=MISSILE CAPABILITY
XX
RADAR CAPABILITY XVIII
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
SIGNATURE
MODIFICATION
NEW TRAJECTORY
CAPABILITY
XIII. BOMBER
SOLE SYSTEM RADAR CAPABILITY
MISSILE CAPABILITY
INTEGRATED SYSTEM
W/
INTEGRATED SYSTEM
W/
XX
RADAR CAPABILITY QED
MISSILE CAPABILITY XX
RADAR CAPABILITY XVI11
MISSILE CAPABILITY
XX
RADAR CAPABILITY
XVIII
MISSILE CAPABILITY XX
RADAR CAPABILITY XVIII
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
-----------------
HUGHES
RADAR CAPABILITY XVIII
MISSILE CAPABILITY XX
XIV. MANNED RECONNAISSANCE I
HUGHES
MANNED
RECONNAISSANCE
INTEGRATED SYSTEM
W/
INTEGRATED SYSTEM
W/
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY
MISSILE CAPABILITY
=RADAR CAPABILITY
MISSILE CAPABILITY
RADAR CAPABILITY XVI11
MISSILE CAPABILITY XX
RADAR CAPABILITY XVI11
=MISSILE CAPABILITY
XX
RADAR CAPABILITY
MISSILE C
XVIII
XX
XVIII
APABILITY XX
RADAR CAPABILITY XVIII
MISSILE CAPABILITY XX
SIGNATURE
MODIFICATION
NEW TRAJECTORY
CAPABILITY
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
XV. HARD COUNTERS - ARM
INCREASED ACQUISITION
CAPABILITY
SEARCH/HARDNESS DEPLOYMENT
ANTENNA STRUCTURE
ENGAGEMENT/HARDNESSr DEPLOYMENT
ANTENNA STRUCTURE
SEARCH INTRINSIC SIGNATURE
ECM CAPABILITY
ENGAGEMENT -INTRINSIC SIGNATURE
ECM CAPABILITY
HUGHES
0- S
XVI. HARD COUNTERS - NON- :HUGHES:
RADIATING HOMING MISSILE ---------
HARDNESS
C ACQUISITION VULNERABILITY
HARDNESS
ACQUISITION VULAERABILITY
A
- 88 -
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
XVII. SOFT COUNTERS - ECM
------------------
HUGHES
JAMMING
SPOOFING
AREA OF INFLUENCE
CCM TECHNIQUES
?DEGRADED MODES
?SUSCEPTIBILITY TO INTERCEPT
VULNERABILITY TO JAMMING
VULNERABILITY TO SPOOFING
MAXIMUM TRACK RATE
GIMBAL LIMITS
ERROR MEASUREMENT
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
XVIII. RADAR
PERFORMANCE)Y (SOLE I
XXI
XX I
LOW ALTITUDE CAPABILITY
DETECTION RANGE
REACTION TIME
NUMBER OF TARGETS
LOW ALTITUDE CAPABILITY
DETECTION RANGE
REACTION TIME
NUMBER OF TARGETS
SEARCH/REACQUIRE MODE
XIX. C3 CAPABILITY (SOLE
------------------
HUGHES
r PERSONALITIES
DOCTRINE TRAINING
PERSONALITIES
SUBORDINATION EQUIPMENT
TRAINING
F PERSONALITIES
AREA OF INTEREST DEPLOYMENT
r PERSONALITIES
AREA OF INFLUENCE EQUIPMENT
rEQU1PMENT
DEGREE OF AUTOMATION TRAINING
EQUIPMENT
LIMITS AND CAPABILITIES MAINTENANCE
rTRAINING
ACCURACIES EQUIPMENT
LINE OF SIGHT
MEDIUM LAND LINE
TROPHOSPHERIC
TRANSMISSION RATE/CAPACITY
f-POWER
~--RANGE SENSOR SENSITIVITY
(-CRYPTOGRAPHY
SUSCEPTIBILITY.
MEDIUM
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
to
N
PM
P
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
XX. MISSILE CAPABILITY
HUGHES
LOCK-ON RANGE
F PERFORMANCE --f CLUTTER TARGET DISCRIMINATION
CLUTTER REJECTION
4SEEKER
LMECHANIZATION AUTOPILOT
CONTROL SYSTEM
PERFORMANCE -CFLYOUT CAPABILITY
MANEUVERABILITY
rMISSILE AERODYNAMICS
LMECHANIZATION -}-PROPULSION
L MASS PROPERTIES
FUZE
MECHANIZATION --C
WARHEAD
XXI. SEARCH/ENGAGEMENT RADAR
CLUTTER
REJECTION
OPERATING f
ENVIRONMENT TOPOGRAPHY
PRACTICES
L FREQUENCY
POWER
LOSSES
TRANSMITTER
PROPERTIES I-- GAIN
AMBIGUITY
RESOLUTION
LOSSES
NOISE FIGURE
SCAN TIME
WAVEFORM
SEARCH/REACQUIRE MODE
(ENGAGEMENT APPLICATIONS ONLY) SCAN TIME
HUGHES
------------------
WAVEFORM
PROCESSING
STABILITY
J
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
SUMMARY
HUGHES
--------------------
THE METHODOLOGY DEVELOPED AND DOCUMENTED HAS
THE FOLLOWING STRENGTHS
? ADAPTABLE
? TRACEABLE
? USES MULTIPLE SOURCES AND TYPES OF DATA
? REFLECTS DECISION MAKERS POINT OF VIEW FOR AUDIT
AND REVIEW
? ALLOWS FOR EXPERT OPINION AGGREGATION
? SENSITIVITY ANALYSIS
The major product of this interim report is the methodology which has
been developed to perform collection concept analyses. The methodology has
been shown to be adaptable to a wide range of problems, is easily used toThe
track the analysis path, and is able to make use of many types
method handles both quantitative and quantitative data as well as allowing
gsfor
several experts opinions to be aggregated to form a consensus opinion.
importantly, between choicesfas sensitivity analysis
of confidence- levels
critical review
in the conclusions reached.
47
w
P
r
p
M
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
REFERENCES
Arbel, Ami and Tony, Richard M.
"On the Generation of Alternatives in
Decision Analysis Problems,"
Journal of Operations Research Society,
Vol. 33, No. 4, pp. 377-387
Bordley, Robert Francis
Studies in Mathematical Group Decision Theory
Doctural Thesis, Univ.of Calif., Berkeley,
1979
(8014621)
Bordley, Robert F.
The Combination of Forecasts: A Bayesian Approach,
Journal of the Operations Research Society,
Vol. 33, No. 2, pp. 171-174
Brown, Bernice & Helmer, Olaf
Improving the Reliability of Estimates
Obtained from a Consensus of Experts,
The RAND Corporation
Brown, R.V. and Lindley, D.V.
Improving Judgement by Reconciling Incoherence,
Theory and Decision, 14 (1982), 113-132
Cook, W. Howard
Decision Analysis for Product Development
IEEE Transactions on Systems Science &
Cybernetics, Vol. SSC-4, No. 3,
September 1968, pp. 342-354
Freeling, Anthony N.S.
Reconciliation of Multiple Probability
Assessments
Organizational Behavior & Human Performance, 28,
(1981), 395-414
Garvey, Thomas D., Lowrance, John D., &
Fischler, Martin A.
An Interference Technique for Integrating
Knowledge from Disparate Sources
SRI International, Menlo Park, California
Howard, Ronald A.
Bayesian Decision Models for System Engineering
IEEE Transactions on Systems Science & Cybernetics
Vol. SSC-1, No. 1, November 1965, pp. 36-40
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Howard, Ronald A.
Decision Analysis: Applied Tecision Theory
Proceeding of the Fourth International
Conference on Operational Research
New York: Wiley-Interscience, 1966,
pp. 55-71
Howard, Ronald A.
Information Value Theory
IEEE Transactions on Systems Science and
Cybernetics, Vol. SSC-2, No. 1
August 1966, pp. 23-26
Howard, Ronald A.
The Foundations of Decision Analysis
IEEE Transactions on Systems Science &
Cybernetics, Vol. SSC-4, No. 3,
September 1968, pp. 211-219
Howard, Ronald A.
Decision Analysis in Systems Engineering
from Systems Concepts: Lectures onContemporary
Approaches to Systems
Ralph F. Miled, ed.
New York: John Wiley & Sons,. 1973-
Hwang, Ching-Lai & Yoon, Kwanysun
Multiple Attribute Decision Making,
Methods & Applications
Springer-Verlez, Berlin, Heilberg, New York
1981
Jamison, Dean
Cojoint Measurement of Time Performance &
Utility
Memorandum RM-6029-PR
June 1969
The Rand Corporation
Keeler, Emmett & Zeckhauser, Richard
Another Type of Risk Aversion
Memorandum RM-5996-PR
May 1969
The Rand Corporation
Matheson, James E. & Roths, William J.
Decision Analysis of Spare Projects: Voyager
Mars (paper)
presented at "Saturn/Apollo and Beyond", America
Astronautical Society, 11 June 1967
w
P
P
P
P
0
A
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Matheson, James E.
Decision Analysis Practice: Examples & Insights
Proceedings of the Fifth International
Conference on Operational Research, Venice,
1969, London: Tavistuck Publications,
1970, pp. 677-691
Moore, John R. and Baker, Norman R.
An Analytic Approach to Scoring Model Design -
Application to Research & Development
Project Selection
IEEE Transactions on Engineering Management,
Vol. EM-16, No. 3, August 1969
Morris, Peter A.
Decision Analysis Expert Use
Management Science, May 1974, pp. 1233-1241
Morris, Peter A.
Combining Expert Judgement: A Bayesian
Approach
Management Science, March 1977, pp. 679-693,
North, D. Werner
A Tutorial Introduction to Decision Theory.
IEEE Transactions on Systems Science &
Cybernetics, Vol. SSC-4, No. 3,
September, 1968
Olender, Henry A.
A Method for the Allocation of Exploratory
Development Resources in.Logistics
SRI International
December 1978
Oppenheim, A.N.
Questionnaire Design & Attitude Measurement
Basic Books Inc/New York
Raiffa, Howard
Decision Analysis: Introductory Lectures on
Choices Under Uncertainty
Addison-Wesley Publishing Company,
Reading, Mass., 1978
Roberts, Fred S.
On Transitive Indifference
Memorandum RM-5782-PR
September 1969
The Rand Corporation
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
Rummel, J. Francis and Ballaine, Wesley C.
Research Methodology in Business
Harper & Row, New York
1963
Selvidge, Judith
Assigning Probabilities to Rare Events
presented to Fourth Research Conference on
Subjective Probability, Utility & Decision
Making, Ruine,' September 3-6, 1973
Spetzler, Carl S.
The Development of a Corporate Risk Policy
for Capital Investment Decisions
IEEE Transactions on System Science & Cybernetics
Vol. SSC-4, No. 3, September 1968,
pp. 279-300
Spetzler, Carl S. and Carl-Axel S. Staelvon Holstein
Probability Encoding in Decision Analysis
presented at ORSA-TIMS-AIEE 1972
Joint National Meeting, Atlantic City,
New Jersey, November 1972
Tversky, Amos and Kahnemon
Judgement Under Uncertainty: Heuristics and
Biases
Science, September 1974, pp. 1124-1131
r
r
N1
IN
IN
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
 UNCLASSIFIED
Approved For Release 2003/09/29 : CIA-RDP86B00269R001300050001-1
UNCLASSIFIED