LETTER (Sanitized)

• Document Type: CREST
• Collection: NGA Records (Formerly NIMA)
• Document Number (FOIA) /ESDN (CREST): CIA-RDP78B04770A000700040008-4
• Release Decision: RIPPUB
• Original Classification: K
• Document Page Count: 6
• Document Creation Date: December 28, 2016
• Document Release Date: October 28, 2004
• Sequence Number: 8
• Publication Date: August 13, 1963
• Content Type: LETTER

Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
Here are a few comments and suggestion* regarding
study of the correlation between subjective judgements
and objective measures of photographic image qu?lity. I auk
very much In agreement with the ultivete goals Of the work-
Is doing, so please consider gay comments and sug-
gentions in light of my sympathetic attitude toward his =work.
In other words, if I sound excessively critical, ettribvte
the sound to my Interest and desire to encourage and help
him--assuming that I can help.
1. The magnitude of any obtained correletis n will
limited not only by the discriminability of th
stimuli, i.e., the subjects' skill in d.i.atinguishing
among them, but also by the reliability of the
subjects' judgments. If, for example, the test-
retest reliability, i.e., the correlation between
successive sets of judgments made by the some
subjects with a month or so intervening between
judgmental ssesaicns, turns out to be zero, no
statistically significant correlation between them
and objective, physical reassures can be expected.
If the test-retest reliability of the judgments as
indicated by the correlation between suceeessivo
sets of judgments turns out to be .5, the maxisun
correlation that can be obtained between the
judgments and the objective measures of image
quality is about .7, assuming the objective measures
are perfectly reliable.
The point of this discussion is that an effort
should be made to determine the teat-retest
reliability of the Judgments of quality. The same
subjects should perform the judgmental task twice
with a month or so between judgmental sessions so
Declass Review by NGA.
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
 Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
usst 13, 1
n 2
that they will not recall their first judgment:
of specific atiusuli. The obtained reliability
coefficient will indicate the extent to which the
correlation between subjective and objective seasures
is attenuated by a lack of reliability. Further, it
will Indicate the worth of subjective judgments as
criteria of image quality. An I recall, ?rani, you
too wanted to see a reliability study done.
u. My second comment concerns the psychophysical
procedure and the data analysis. The two cannot be
considered entirely independently, for the psycho-
physical procedure used will limit the number of
appropriate analysis techniques available. (I don't
know how intends to analyze the data or
what kind of correlation he plane to compute, no to
some extent I'm shooting in the dark here.)
As the procedure is currently designed, the subjects
can use almost any number of judgment al c:e>tegi iris
in assessing photo image quality. And because of
individual differences in d1arriminability and
attitudes; toward the task, you will find some
subjects using 4 or 5 categories and others uGing 9
or 10 or possibly more categories. Consequently
the resulting data cannot be combined too conveniently
to arrive at a single regression equation and corre-
lation coefficient based or the judgments of all
subjects. Instead, as I see it, one equation and
coefficient will have to be computed for each subject
or for small groups of subjects that, fortuitously,
ended up using the same number of judgment categories.
It would be more convenient from a st*tisstica point
of view if all subjects wore required to sort the
photos in a specific number of categories, say 7 or
9, ranging from "beet" through "a v S r a3 e" or "median''
to "poorest" inage quality. I am not aware of the
number of physical steps used in generating
timuluss materials (or the approximate number of
just noticeable differences"--find's--from ono end
of the physical scale to the other, which could only
be determined empirically) an t can't guess tea well
about the number of judgmental categories the subjects
might be able to use reliably. But, in any came, If
a specific number of judgment categories were used,
the data from all subjects could be plotted as
follomss
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
 Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
August 13, 1983
Page 3
ategorical
Judgments
of Image
Physical Measure
(in units of blur or graininess.)
The total number of entries in the table would be the
product of the number of subjects and the number of
stimuli judged. From the table a coefficient of
contingency could be computed. Or if the assumpti
could be met (or reasonably ignored) a product-moment
correlation could be computed.
(As I mentioned before, I don't know how
intends to analyze the data. F1e may be aware of
some techniques that I'm not familiar with, or I may
not understand his research goals. In either cease,
these comments might be entirely irra.evan:t. )
From a psychophysical point of view, the most sensi-
tive judgmental technique in a study such as this
is the method of pair comparisons. The subjeets
simply compare two stimuli at one time and respond
by indicating which is poorer and which is better
quality. Every stimulus is compared at least once
with every other stimulus, and that's the limitation
of the method. With a stimuli, there are An-1) s 2
possible pairs, which cen result in a burdensome
judgmental task.
I don't know precisely how the stimulus mntsr ; la
were prepared. L don't know whether blur and rraini-
nose were varied independently or simultaneously. If
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
 Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
.ugus 7, IVF3
Page 4
they were varied independently, there might be, for
example, 7 levels (sagnitudes) of blur at the same
level of graininess. There would then also be 7
levels of graininess at the same level of blur,
yielding a minimum of 14 stimuli. On the oth'r
hand, if they were varied simultaneously, all levels
of blur would occur at each level of graininess, and
vice versa. With 7 levels of each variable, nizaul-
ta neously varying them would produce a minimum of
49 s t i m u l i.
This involved, and probably unclear, discusssi?f has
relevance. I think, for the manner in which the
judgments of quality are obtained. If blur and
graininess were varied independently, shouldn't they
be treated independently in obtaining the judgiesstss
/ of quality? They are probably not equally related
to judgments of quality; they are probably not
equally difficult to discriminate; and, thus, judg-
ments of quality as a function of blur, on the one
hand, and graininess, on the other, are probaaty not
equally reliable.
If blur and graininess were varied simultaneously in
preparing the stimulus materials, an experimental
design should be selected which would permit the
experimenter to determine the relative contributions
of each physical variable to judgments of quality.
I would suggest an analysis of variance design, which
implies the necessity of numerically sealed judgments
of image quality rather than simple, ranked judgments,
and further implies the necessity of developing the
scale. Cr, I would suggest as non-parametric analysis
of variance such as that described by ssarnor and
kc(ill..* The latter design is not as powerful
statistically but in a convenient method of handling
categorical, rather than scalar, date.
I realize, Frank, that this terminology May be foreign
to you, but I find I can't express myself without
usin it, which is unfortunate. Rut,, hopefully, You
and can cut through all
of the statistical jargon. y only excuse for the
jargon is the fact that experimental psychologists
*Garner, W.R.
W.J.The relation between information
t>sycchometrtha, 195d, 1, 219-226.
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
 Approved For Re
August 13,
Page S
are continually faced with the omnipresent and
profound fact of individual differences and must
approach and design experiments using statistical
models.
4. Because of the large number of stimuli being judged,
I don't see any way of experimentally controlling the
effects cue to the order in which they are presented.
Significant effects due to order of stimulus presen-
tation have been demonstrated repeatedly in psycho
physical stork. The only thing I can suggest is that
about throe anchor stimuli, one at the middle and
adjacent to either and of the judgmental scale
be presented first and then the judgments of them
not be used in the analysis. The remaining sTir?uli
would be shuffled and presented in a different randos
order to each subject. In other words, the effects
due to order of atinulus presentation would b+-
randomized across all subjects.
5. 1 would consider whether or not the subjects should
be told In the instructions that the photos differed
in blur and graininess, when they are being asked to
make a one-dimensional judgment of quality.
I an very much Interested in, an
for, the study is doing. I
to discuss it personally with him, for
have communicated too much here.
sympathy with the need
d like an opportunity
wou
n
.ar that I may not
I realize that his work is a first step, but I thi:
an important step toward a goal I feel is very such wor
seeking: a specification of the relations among physical.
measures of photo image quality, subjective measures of image
quality,, and objective measures of c?I performance. I foel that
such a specification would be exceptionally valuable In all
phases of our works materials, collection, processing, and
exploitation.
sal point of cualificaticu. I showed these comments
and suggestions to a fellow in our office here who has
specialized in psychophysical research. His reaction was that
I have over-simplified the problem. F+owevor, In general, he
agrees with me, though he feels what I have said will Bake
difficult reading. My apologies.
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
 Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4
August 13,
Incidentally, Frank, I know I said I would write to ;you
a weak ago. Well, I simply have not had an opportunity to
write. Work at the office end at hone piles up when I'm in.
Pn.ahinvton. This time it piled up at a prodigious ratan.
Tie-t refrardt,
Approved For Release 2004/11/30 : CIA-RDP78BO477OA000700040008-4