NOAH'S ARK EXPERIMENT

• Document Type: CREST
• Collection: NGA Records (Formerly NIMA)
• Document Number (FOIA) /ESDN (CREST): CIA-RDP79B00873A000800010055-6
• Release Decision: RIPPUB
• Original Classification: K
• Document Page Count: 12
• Document Creation Date: December 28, 2016
• Document Release Date: October 18, 2012
• Sequence Number: 55
• Publication Date: February 1, 1967
• Content Type: MISC

Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
JS-516 PROGRAM DESIGN
NOAH'S ARK EXPERIMENT
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
JS-516 PROGRAM DESIGN
NOAH'S ARK EXPERIMENT
by
STAT
1 February 1967
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
In looking forward to the design of future photographic interpretation viewing
stations for black and white imagery, the utilization of new and superior light sources
is being considered. Since it is desirable to limit research to those light sources that
are most efficient and are, at the same time, preferred by those performing the inter-
pretation task, it is necessary to determine which sources are really preferred. Thus,
prior to dictating a specific research direction, it is first necessary to select a repre-
sentative sample of light sources that may be utilized in photo interpretation under-
takings and have these sources evaluated by photo interpreters.
The substantive or test hypothesis of this program is that there exists some optimum
source of illumination for photographic interpretation tasks. Whether this optimum
exists in terms of maximum attainable luminance level, a particular spectral distri-
bution, or some other variable is unknown and will not be analytically determined in
the program presented herein.
The operational equipment constructed to test the hypothesis has been termed
Noah's Ark. This piece of equipment allows the presentation of several light sources,
two at a time, side by side, on separate ten-inch-square viewing screens. Thus, the
following operational constraints are placed upon the equipment.
1. A finite number (4 or 5) of specific light sources may be utilized.
2. No single light source may be presented on both screens as a test for posi-
tional bias.
3. A single viewing arrangement and condition are utilized.
It may be pointed out that this combination of operational equipment and test hypoth-
esis may generate results that could become trivial. If, for example, a given source
were found to be optimal; and, based on this result, further research generated a new
family of lamps; it may be possible that these new lamps do not have the optimal
qualities of the original, since only the general optimal property will be known, i. e. ,
the known property will be that the original source is optimal compared to N other
sources.
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79BOO873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
The following program comprises two separate experiments which are conducted
simultaneously. The first experiment utilizes a quantitative response measure of photo-
interpreter performance, the second is a choice-discrimination experiment. The
experimental procedure and data analysis will be described in the following pages.
QUALITATIVE EXPERIMENT
Presentation Procedure. The general operational hypothesis, upon which the response
measure of this experiment is based, is that a decrease in photointerpreter decision
time indicates an increase in photointerpreter performance. No weighting or distinction
is made to denote whether the decision is correct or incorrect. (Weighting requires
the presence of ground truth data.) The response measure is recorded under two dis-
tinct experimental conditions. Basically, both of these designs are identical, except
that the task, the photographs, and the photograph presentation function differ. These
experiments are designed to obtain a maximum of information concerning the allocation
of variability in the response variable.
As presently designed, each of the two parts of this experiment requires that the
experimental equipment termed Noah's Ark be supplied with the following light sources.
a. One tungsten source.
b. One mercury arc.
c. One xenon are.
d. One electroluminescent panel.
e. One halogen cycle lamp.
Twenty experienced photointerpreters are required for each part of the experi-
ment. The same twenty subjects are to be utilized for both parts of this experiment,
so that the times of performance of different tasks by each individual may be compared.
The utilization of experienced personnel is emphasized, since newly trained photo-
interpreters do not have enough experience to form realistic equipment preferences
or make discriminations.
Forty photographs will be required for each of the two parts of the experiment.
2
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
These photographs should be in concomitance with the dictated task. For example,
the first task of the experiment may be to answer the question:
"Are there ICBM's (of specified type) in the missile silos?".
Obviously the imagery must contain missile silos; however, missiles may or may not
be contained therein. The location of the target area (missile silos) should be marked
on the photographs as a rapid scan task, and search time is not a part of the defined
response variable. Decision time is the response variable.
The second task should be equally realistic and must be so stated that the
response variable may be termed decision time. Thus, it is required that decisions
be made on 80 different photographic areas. It is suggested that every effort be made
to obtain 80 different photographs, each group of forty being presented under a specific
task dictation.
Turning to the section of this book titled "Presentation and Response Data Sheets",
the reader will find on the first page the light source presentation sequence. Observe
that this sequence is for only one task and that the sequence of presentation of
the lamp pair is randomly ordered. This presentation sequence is derived from the
next page in that section, termed "Stimulus Presentation Function Tabulation, Task 1,
subject 1-20". This random ordering is altered for the second task, as can be seen
on the two pages following the above. In this manner the effects of the lamp presen-
tation sequence may be randomly distributed and will appear in the error term of the
data analysis. In addition, since the random ordering changes for each task, no lamp
sequential patterning can be established across the two tasks for any individual
photographic interpreter.
The photographs in each set of forty, one set for each task, will be numbered 1
through 40. Due to the construction of Noah's Ark, two photographs are presented
at a time. This presentation function is randomly ordered in terms of lamps, the
specific photographic pairs presented, and subjects. This may be noted from the
Stimulus Presentation Function Tabulation sheet in the second section of this book.
For example, the first presentation to the first subject will be photograph No. 5 over
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
the halogen cycle lamp on the left and photograph No. 32 illuminated by the xenon arc
lamp on the right. The dictated decision task will be performed on both photographs,
the decision time for each task being recorded on the appropriate Response Tabulation
Sheet (these will be found immediately following the "0" tab in this program book).
The second presentation to the first subject will be photograph No. 13 over the tungsten
light source on the left and photograph No. 25 on the right, illuminated by the halogen
cycle lamp. For the second subject, the lamp presentation sequence remains the same,
but the photograph presentation sequence changes. After completion of task No. 1,
the same photo presentation sequence is utilized with the second set of photographs,
the second task and the second lamp presentation sequence.
Utilizing the second set of forty photographs with the second task, the first presen-
tation to the first subject will be photograph No. 5 over the tungsten light source on
the left and photograph No. 32 over the xenon arc on the right. The decision time will
be recorded in the appropriate Response Tabulation Sheet.
It may be noted that the five lamps are presented in a fixed-crossed design, with
the main diagonal representing a null condition due to the nature of the construction of
Noah's Ark. No lamps can be presented on both viewing screens. With a design of
this type, effects due to left or right discrimination on the part of the subject may be
detected, as will be shown under "Data Analysis".
Data Analysis. The section of this program book immediately following the "0" tab
contains the response tabulation sheets. Upon these sheets are recorded the decision
times obtained for the presentation set. It may be noted that the blocks are notated to
indicate the photograph number presented in each trial. These sheets and the data
analysis are broken into two parts (analyses for task No. 1 and task No. 2), both parts
utilizing the same data analysis procedure. To the right and to the bottom of the
response matrix are sets of blocks in which row and column sums may be entered. It
may be noted that the row sums for tungsten will be a summation of the data below the
diagonals in the first row. Likewise, the sum for the tungsten source in a left presen-
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
tation is the first column sum of all data above the diagonals.
row sum (tungsten )
k = left = 1
column sum (tungsten)
(EIC2J)
k = right = 2
In general, these sums will be termed row and column tabular summations and will
be represented by
4
ETc...
c=r
where the dots represent the treatment subscripts for those treatment levels being
held as parameters in the summation process.
Turning to the section herein titled "Data Analysis; Quantitative Measure", the
reader will observe that the first page therein lists the factors, the general mathema-
tical model and the EMS (error mean square) table for the entire experiment. The
factors are listed as to whether they are random, fixed, or nested within another
factor. Imagery is a random factor, denoted as P with the levels i = 1, 2, ..., 40.
The subject's individual characteristics constitute a random factor denoted by S with
the levels j = 1, 2, ..., 20. There are only two positions in which the light source
may be placed, on the left or on the right, hence the two levels of the position factor
R are fixed: k = 1, 2. There are five light sources being examined. These light
sources are nested within the presentation position. In other words, a light source
cannot be presented without being either on the left or the right, hence, the light source
effect must be treated as a nested factor in the analysis with the levels l = 1, 2, ... , 5.
Because of the nature of the design of this experiment there can be no replication.
Once a photograph has been presented, learning has taken place; hence interaction
effects cannot be determined and are confounded with error. Thus, the general mathe-
matical model becomes
Tijkl~n = ? + Pi + Sj + Rk + Ll(k) + Emn(ijkl)
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
where en (ijkl) contains all second and third order interactions.
The data analysis of each task is broken into two steps or classifications. This
is necessary because of the presence of the random factors. The error terms for
each of the classifications will not be the same, as can be seen by the following mathe-
matical models for each classification. For the first classification
Tij m= Ii + Pi + Si + 6m(ij)
thus the error term contains all the confounded effects of positions, light sources and
their interactions. The second classification yields a mathematical model
T.jklm = It + Sj + Rk + L, (k) + Em0jkl)
the error term containing the effects due to photographs. The modified EMS table
then becomes
Source
EMS
Pi
200 up' +
0,E2
Sj
400 as2 +
QE2
Rk
4000 aR 2 +
QE2
LI(k)
800 aL,2 +
cr2
2
O l2
However, the variance test sequence is altered, since the a,2 term is actually differ-
ent for the two methods of analysis. It is interesting to note that the effect due to sub-
jects is measured or determined twice, once in each method or analysis classification.
The data analysis starts with a retabulation of the data by summing over positional
and light source variables. The resulting table is a 40 x 20 matrix, the columns repre-
senting subjects and the rows photographs. Each entry in the matrix represents a
decision time. Sums are computed for each row and each column, the column sums
representing subject effects across photographs and the row sums representing photo-
graph effects across subjects.
E
j=1
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
40
column sum = ri.
i=1
The sums of squares for the factors are computed as follows:
"SStotal
Ssphotographs
4,0 20 40 20 2
E Er 2 E Eras
i=1 j=1 i=1 j=1
1 800
4.0 20 40 20
E E (r.3)2
E ri1
E
i=1 j=1 i i=1 j=1
Sssubjects
SSerror
20 ?0 40 20 2
(rI)2
j=1 1=1 i=1=1I j=1
= SSStotal Ssphotographs SSsubjects
Standard ANOVA analysis techniques then follow as in the EMS assessment.
The second analysis procedure requires a reclassification of the data into a 20 x
5 x 2 matrix j x k x l , in which the elements are the tabular summations
4
0=1
It must be remembered that l is a nested factor; therefore, the sum of squares
correction factor will be calculated and applied for the left presentation position and
the right presentation position. The sum of squares calculated for the light sources
when presented in the left position (k = 1) is
SSk =1
EErcj.I
and for the right position (k = 2)
SSk =2
t(420)2
l=1 c=1 j=1 l
ri 5E 20 2
l=1 c=1 j=1 l
5 k 20 2
E E 11 rcj.l
k=2 1=1 c=1 j=1
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
In this way the nesting effects have been removed from the light source effects, and
the resulting sum of squares due to light sources is:
SSi?mps = Sk=1 + SSk=2
The total sum of squares for the second analysis is equal to that of the first analysis.
The subject effects are computed on the tabulated data as
"SSsubjects
20 k 2 5 2 (20425)2
T EE j j=1 c=1 k=1 l=1
and finally the positional effects are computed as
SSp os ition
E(EEEr. I E ttrcjkt
h:=1 c=1 j=1 1=1 k k=1 c=1 j=1 1=1
ANOVA analysis techniques are then utilized by the standard procedure. The result
of this analysis is the elucidation of the factors that are contributing significantly to the
variance in the observed decision time. It is expected that subjects and photographs
will exhibit a definite influence on the observed decision time. If the light sources are
not significant in altering the observed decision time, then no further analysis of this
data need be performed. If the converse is true, the data will be further analyzed to
determine which of the light sources yields the most significant decrease in decision
time from the overall mean. In accordance with the operational hypothesis, the lamp
that allows the most significant decrease in decision time is the optimum. There are
two risks inherent in making the above statement; one is that a lamp that is found to
be optimum may be altered so as to be non-optimum through further research; the second
is that a non-optimum lamp may become optimum through further research. These
statements, respectively, represent the beta and alpha risks taken in the data analysis.
Presentation Procedure. Prior to describing the experimental design, a definition of
terminology is necessary. In psychological terms, an act is an event that is ascribed
to some change within the body of the subject. An outcome is the payoff or reinforce-
ment that is applied to or is given to a subject in accordance with a given response.
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
This may be simple feedback to the subject as to the correctness of his act or response.
Discrimination is a feature of a situation that exists before the subject acts and is
independent of the outcome. Preference, on the other hand, is a feature of the out-
come of acts; and, as such, it is dependent upon the outcome. A subject exercises
preference when he selects his acts in accordance with the outcome a given act will
produce' ? 2
In this experiment there is no outcome function, hence, a preference experiment
cannot be utilized. What can be designed and utilized is a discrimination experiment.
The discrimination experiment can be run at the same time as the quantitative
experiment. After the decision time response measures have been obtained from a
presentation pair of photographs, the subject may be asked which of the two light
sources was of greater aid in the performance of his task. His response selection
would then be left, right, both, or none and would constitute the act following discrim-
ination in response to the request to act. Because of the nature of the quantitative experi-
mental design, the photointerpreter will be forced to utilize each light source eight
times in each task; four times on the left and four times on the right.
The request to discriminate and the response blank are found under the section
heading "Presentation and Response Data Sheets". This request is worded as follows.
"Two transparencies will be presented to you, one over each of two light sources.
You are to perform the dictated task upon each photograph. After completion of the
task on both photographs, indicate which of the two light sources resulted in the opti-
mum performance of your task by checking the appropriate blank. You may use any
personal subjective criteria for "optimum" that you feel will aid the evaluation of these
light sources as sources of illumination for future viewing stations.
"Indicate your response by checking either the left or the right blank according to
your estimate of optimum. If neither light source meets your criteria then check
neither blank. If both sources are considered equally good check both blanks. Any
remarks you may have concerning the color, brightness or any other characteristics
may be entered under REMARKS. "
The responses are separated in accordance with the task performed. All discri-
mination responses will be summed for each light source. The proportion, P , of
responses for a light source, Ns , to the number of possible responses for that light
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
 Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6
source, N , is computed: P = Ns!N . This computation is performed for each light
source when presented on the left and on the right. Also, a pooled (positional pooling)
value is computed by taking the total number of favorable responses over the total
number possible for each light source in both positions. The analysis tabulation may
be found under the section heading "Data Analysis; Choice Experiment". The compari-
son of the response ratios may be performed between the two tasks and may be evalu-
ated in terms of the findings of the quantitative experiment.
The entire program allows the comparison of quantitative measures of subject
performance and the analysis of a completely subjective discrimination task. In addi-
tion these comparisons may be made across two tasks; and the effects that the particular
photograph, an individual subject, and the relative position of the light source have
upon the photointerpretation task will be elucidated.
BIBLIOGRAPHY
1. Bush, R. R., E. Galanter, and R. D. Luce. "Characterization and Classification
of Choice Experiments", Handbook of Mathematical Psychology , Vol. 1, Chap. 2,
New York, John Wiley and Sons, Inc. , (1963), pp 77-102.
2. Irwin, F. W. "An Analysis of the Concept of Discrimination and Preference",
American Journal of Psychology , Vol. 71 (1958), pp 152-163.
Declassified in Part - Sanitized Copy Approved for Release 2012/10/18: CIA-RDP79B00873A000800010055-6