Approved For Release 21QA?3ltltb~llc"- P&RA100788RO01500170028-9
Memo
To : P. McNelis
Date :
12-Oct-1988
From ? E. C. May G
l
ti
A
li
' C
Location :
Cc:
G-201
SG1J
y
na
cs
omments on
s
Subject : Response to McNe
My response to your comments (18 August 1988) with regard to the then proposed
addendum to the Enhanced Human Performance Investigations FY 1988 Technical Mid-year
report (now considered the final report to Objective E, Task 4) will be confined to your specific
comments 1-13 (see Attachment).
1. The reference to the FY 1986 is accurate. The work described on page 23
of the 87 Mid-year Technical Report is a description of work performed
under Object E, Task 2-Search in FY 1986. (Note page B-9 of the FY
1987 SOW, Objective F Task 10, "Building upon the results of the FY 1986
program explore the ' limits of quality of remote viewing of analytical
information.") During FY 1986 there was no specific tasking for the
analytics program, but I felt that, if successful, it would be applied, in the
binary search arena, and therefore was a justifiable portion of that task.
Since "analytics" was not part of the formal tasking, the reporting of it
appears in the Mid-year Technical Final report for FY 1987 ahead of the
required time of the end of 3rd quarter, FY 1987 (see SOW).
2. The techniques referred to in (1) and (2) are not necessarily the same. The
description is confusing as written, so I have clarified that point in the report.
3&4. As stated, the objectives were confusing. I have reworded them to more
accurately reflect the SOW and the intent of the project. In my opinion only
a part of science is of the form of hypothesis formulation, test, and
reformulation. The creative part of science manifests itself in determining
the appropriate hypotheses. As part of the FY 1986, SRI initiated this
,concept as part of the search task because we felt it was a good area for
'exploration, given the literature and given the application potential, and that
it met the spirit of Phase I of the contract-knowledge building. In FY 1986,
we did not send a protocol to the SOC in that to do so within a completely
unknown area that is being explored would be inefficient. The protocols
change from moment to moment in that type of environment, and to wait
until an SOC response is inappropriate in this case I will present this whole
effort as a specific agenda item during this year's SOC meeting. The
protocols that were used in the formal studies as well as in the exploratory
studies will be reviewed, and the SOC will determine if the results and
conclusion presented in the final report for Objective E, Task 4 are justified.
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
5. Yes, this bullet is accurate. I have double checked both our formal reports
to you with the "raw" data log and find the following. For the three years,
there has been a total of 2224 binary trials (529-FY 1986, 1168-FY 1987,
and 527-FY 1988). During that time there were only a total of 382
non-binary trials (200-FY 1986, 0-FY 1987, and 182-FY 1988).
6. The differential comparison point has been added to the report in the
appropriate place.
7. A minor error was made in the FY 87 Mid-Year. The p-value quoted there
(0.024) used a normal approximation to the binomial. The value in for final
report is computed from the exact binomial and is 0.033. The difference is
of no significance. Thanks for catching this technical error.
The ideal of sensitivity to putative decline effect was intended to mean that
the very high baseline (or as rewritten-fiducial point) is not as sensitive to
increases in hitting rate as it is to decreases, given that a decline effect might
be true. In any case, this section has been completely rewritten.
8. As it turned out, you did have valuable input into the final test-auto-typing
of Ready. This section has been rewritten to reflect the final report nature
of the document, and all the final trials were completed withing the agreed
upon deadline.
9. The wording has been changed to "The hitting rate for these 18 trials was
78%, p S 1.69 x 10-2, and the effect size was 0.50."
10. In the final report, all binary trials are displayed, so the confusion in the
earlier report is no longer present.
11. The section regarding specific and different internal strategies has been
expanded for the final report.
12. What was meant was that since no decline effect was observed, some other
hypothesis must account for the data. All the conclusions have been
rewritten to reflect the complete data set.
13. Along with yours, Jim's comments have been addressed in the final report.
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
PROJECT 1291
CLIENT REPORT APPROVAL FORM
Transcript of McNelis' Comments Dated August 18, 1988
(Re. Addendum to FY 1988 Mid-Year Technical Report)
There are a number of serious concerns I have regarding this document and the actual
conduct of this experiment which, until clarified and corrected, preclude my giving approval.
My primary concern is with the lack' of a rigorous design for the work you are doing with
this subject. In FY 86 he did 50 formal trials on a binary task; follow-up in FY 87 was frustrated
when, after a large number of pilot efforts, the subject declined formal testing (it is noted that the
results of the pilot tests were not encouraging). In FY 88 the subject continued to do vaguely
defined tasks which, in my opinion, will make it impossible to attain the primary objective of
improving single-bit hitting rates in binary tasks since (a) base line data are not being collected
(or, if FY 86 data is being relied on, it is questionable whether 50 trials is sufficient) and (b) we
will not have the time to run an adequate number of formal trials in FY 88 to demonstrate
improvement.
Specific comments follow:
1. Page 3, first paragraph - reference to FY 86 work in Objective E, Task 2 is
not accurate. Mention is made to the FY 86 effort for the first time in the FY
87 Mid-Year Technical Report (page 23) under Objective F, Task 10.
2. Page 3, Section A, (1) and (2) - are the "techniques" referred to in (1) the
same as those in (2) or are they external versus internal. Please clarify and
specify.
3. Section A, general - multiple objectives are confusing n there is no way one
could design an experiment at this point regarding the two secondary
''objectives since (a) the techniques are not defined, and (b) formal base line
data has not been gathered. These secondary objectives should have been
eliminated or more scientifically addressed.
4. The SOC should have reviewed the "protocol" for this "experiment."
5. Page 4, Section C, first bullet - implies a binary tasks is the usual protocol, is
this accurate since much/most of the data being collected appears to involve
multiple choices.
6. Page 4, Section C, last paragraph - alludes to "formal tests." If single bit
hitting rates in a binary task are to show improvement, they have to
demonstrate improvement compared to like base line data and/or include
sufficient trials over time to demonstrate this. The clarification of this is
critical.
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9
7. Page 5, Section D - indicates a p value of S 0.033, while the FY 87
Mid-Year Tech Report states the p value for the same data"is < 0.024. Also,
this paragraph indicates that the FY 86 data was "sensitive to a putative
decline effect" - what does this phrase mean nom, where is the data to support
8. Page 5, Section E, second paragraph - states "SRI and sponsor's COTR will
determine structure of formal test." This "arrangement" is not in the contract
and is first mentioned by you in this document' presented to me in
mid-August, 1988. I will be happy to input, but see my comment number 4
above. Also, it may well prove to be too late to do a formal test (see my
introductory comments and concerns).
9. Page 7, Section F, second sentence - unclear. Perhaps this should read "The
task was to determine whether... etc."
10. Page 7, Section F, last sentence - when you say the "remaining 63 trials" are
you referring to 63 of the 81 original binary trials? If the first 18 were thrown
out, then it would seem that your total binary trials are just 63 (especially since
the first 18 did not fit your "usual protocol" regarding type of targets).
11. Page 8, first paragraph, first sentence - what are the "specific and different
internal strategies" mentioned. This is critical information and, if only Ingo
knows the answer (assuming he knows), it's not meaningful to us.
12. Page 8, first paragraph, last sentence - what are the "other two hypotheses"
you are referring to; where are they so stated?
13. I have also attached the comments made by regarding this SG1J
document. they are also critical of this work and indicate his contention that
the work is seriously flawed.
Approved For Release 2001/03/07 : CIA-RDP96-00788R001500170028-9