ENHANCED HUMAN PERFORMANCE

• Document Type: CREST
• Collection: STARGATE
• Document Number (FOIA) /ESDN (CREST): CIA-RDP96-00789R002200400001-3
• Release Decision: RIFPUB
• Original Classification: K
• Document Page Count: 51
• Document Creation Date: November 4, 2016
• Document Release Date: October 14, 1998
• Sequence Number: 1
• Content Type: REPORT

CIVI'7NIVl~GI! nurvw~v ~
ENHANCED HUMAN
` INVESTIGATIONS (U)
\
~ v
^
u
..
-.__._..
_.
I
/
~
.
1~
NOT RELEASABLE TO
FOREIGN NATIONALS
Approved ~~~;f~~s~7~8~/?B~T~C~A~~P96-007898002200400001-3
roved For Release 2000/08/08 :CIA-HtDP96-007898002200400001-3
FJna1 Report . _ _
Covering the Period November 1983 to October 1985
PERFORMANCE
 Approved For Releas 08/08 :CIA-RDP96-007898002200400001-3
a-
I. (U) Objective ~'---_
(U) The objective of this program was to provide an overview of the eurre?~ _~
psychoenergetics research and, based upon this assessment, to recommend avenues of approach
for future investigations.
II. (U) Background
(U) Psychoenergetic research can be divided into two major areas of interest:
(1) Informational Processes
(2) Causal Processes.
Each of these areas can be subdivided further into training, screening, and fundamentals such as
various type of functional correlates (e.g., psychological, physiological, and physical).
During FY 1985, SRI International completed a retrospective
analysis of a substantial body o.f open and classified literature in order
to assess existence issues, research questions and potential applications
of the previously reported activity in these areas. Subsequently, part of
this analysis produced two reports that outlined an improved remote
viewing analysis technique and provided ameta-analysis of the random
number generator literature. (These two reports are included as Appendix
A and B, respectively.) what follows are the recommendations, for a
three-phase multi-year research effort.
III. (U) Recommendation
A. (U) Phase I-Knowledge Building
(U) Phase I is considered to be a knowledge building effort. During this phase, SRI
recommends that some form of technical oversight be included in order to provide guidelines on
research protocols, to assess the credibility of the research, and to provide insight into new
directions for future research. This phase should be as wide in scope as resources allow. More
focused research should be delayed until a knowledge base is established. Table 1 shows the
specific areas that are recommended for consideration as research items for Phase I.
Approved For Release 2000/Q~3/08 : CIA-RDP~6-007898002200400001-3
 Approved For Release 2
00/08/08 :CIA-RDP96-007898002200400001-3
Table 1
(U) PHASE 1 RECOMMEI\'DED RESEARCH AREAS
Tapic
Description
Informational Processes
Analysis
A quantitative remote viewing (RV) analysis technique.
Training
Novice and advanced RV training methodologies.
Screening
Techniques to identify goad remote viewers.
Physical Correlates
A search for RV correlates to the physical environment.
Personality Correlates
A search for personality traits in good remote viewers.
Physiological Correlates
A search for physiological correlates to RV.
Medical Correlates
Monitor medical conditions of all viewers.
Feedback
Determine the role of feedback in RV experiments.
Spatial Search
Determine if items can be located in space.
Temporal Search
Determine if events can located in time.
Causal Processes
?Micro-remote Action
Remote action (RA) on random number generators.
Intuitive Data Sorting
Test the Intuitive Data Sorting Model.
Macro-remote Action
Test a variety of physical systems as RA targets.
Correlates
As above, determine correlates to RA.
General
Information Services
Develop auser-accessib]e library system.
(L) While some of the items shown in Table 1 can be considered beyond existence
issues and thus should be considered during Phase II, the predominant effort is toward
knowledge building.
B. (U) Phase II-Development
(U) During Phase II, research areas from the Phase I effort that yielded
incontrovertible evidence for their existence, will be expanded. With the assistance of a
technical oversight committee, hypotheses will formulated and tested.
,Those areas under Phase I that showed the most promise,
will be expanded toward a potential application area. For example, if a
physiological measure could be found that correlated strongly with
excellent remote viewing, then that measure could be used to improve
S applications.
Approved For Release 0/08/08: P96-007898002200400001-3
 Approved For Release 2000/ 8/08 :CIA-RD 96-007898002200400001-3
C. (U) Phase III-Applications
~)Nhile continuing Phases I and II on specific items of
interest, Phase III will be devoted toward applicationsi
This activity should include at least two parts:
(1) Applications research--Formulate and test hypotheses
that are specific with regard to potential
applications.
(2) Application testing--Under actual
conditions, conduct psychoenergetic activity to assess
field utility.
IV. Financial Report
During FY 1985 a total of ffi1,240 K was allocated to contract
_~for the psychoenergetic investigation and review. All
moneys were expended in accomplishing the stated objective.
Approved For Release 2 00/08/08 : CIA-R~P96-007898002200400001-3
 Approved For Release 2~~1'~8~~~~1~00789R002200400001-3
APPENDIX A
A FIGURE OF MERIT ANALYSIS FOR FREE-REPONSE
(This Appendix Is Unclassified)
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
FIGURE OF MERIT ANALYSIS FOR FREE-RESPONSE
MATERIAL
by
E. C. May
B. S. Humphrey
C. Mathews
SRI Internatio~tal, Menlo Park, CA
t
ABSTRACT: A simplified automated procedure is suggested for the analysis of
free-response material. As in earlier similar procedures, the target and response
materials are coded as yeslno answers to a set of questions (descriptors). By
definition, this coding defines the complete target and response information. The
accuracy of the response is defined as the percent of the target material that is
correctly described (i.e., number of correct response bits divided by the number of
target bits = 1). The reliability of the response is defined as tke percent of the
response that is correct (i.e. the number of correct response bits divided by the total
number of response bits = 1). The figure of merit is the product of the accuracy and
reliability. The advantages and weaknesses of the figure of merit are discussed with
examples.
With~he increased use of computers in parapsychology laboratories, it has become possible
to consider more complex methods of analysis to provide deeper insight into the mechanisms of
the phenomena. The Engineering Anomalies Research Laboratory, Princeton University,
provided a major advancement in the analysis of free-response material (Jahn, Dunne and Jahn,
1980).
THE PRINCETON EVALUATION PROCEDURE (PEP) - A BRIEF REVIEW
In general, the Princeton Evaluation Procedure (PEP) is based on comparing a priori,
quantitatively-defined target information with similarly quantitatively-defined response
information. So defined, the PEP applies various methods of mathematical comparisons to arrive
at a meaningful assessment score for remote viewing responses.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release 2~~8wSr.+p~~p0789R002200400001-3
Target Information
The definition of a particular target site (usually outdoor sites in and around Princeton,
New Jersey) is contained in the yes/no answers to a set of questions called descriptors. These
descriptors are designed in such a way as to characterize the typical Princeton target. Each
descriptor bit is weighted by its a priori probability of occurrence in a large target pool. lay
definition, the only target information that is to be considered for analysis, is that which is
contained completely in the yes/no answers to the descriptor questions (with their associated set of
descriptor weights) for the site in question. For example, one descriptor from the Princeton list,
"Are any animals, birds, fish, major insects, or figures of these significant in the scene?" defines
the animal content of the site. The question would be answered "yes" for a zoo and a pet store
target, but "no" in all probability for a typical campus building target. Similarly, a het of yes/no
responses (30 for the PEP) constitutes the target information.
Response Definition
The descriptor list for the target sites is used as a definition of the response as well. For
a given remote viewing session, the remote viewer (or an analyst who is blind to the target site)
attempts to answer the 30 questions on the basis of that single response only. In the example
above, it would be necessary for a viewer (or analyst) to decide whether or,not a particular verbal
passage or a quick sketch could be interpreted as depicting animals. For some responses this
might be an easy task, e.g. "I get a picture of a cow." Most responses, however, are somewhat
ambiguous and require a judgment, e.g. "I see a farm." Nonetheless, the yes/no answers to the
30 questions constitute the only response information that are used in the analysis.
Analysis
For a given response/target combination; the information is contained exclusively in the
yes/no answers to the descriptors. Two binary numbers (30 bits long each for PEP) are
constructed, one for the target and one for the response descriptor questions, respectively. A
"yes" answer is considered a binary "1," while a "no" answer is considered a binary "0." The
resulting two, 30-bit binary numbers can then be compared by a variety of mathematical
techniques involving use of the weighting factors, to form a score for that specific remote viewing
session. For a series of sessions, a quantitative assessment is made by comparing a given response
(matched to its corresponding target site) against the scores that are computed by matching the
response to all other targets used in the series. This procedure has the added advantage of a
built-in, within-group control. In other words, this assessment determines the uniqueness of the
target/response match as compared with all other possible matches for the series.
Advantages of the PEP
There are a number of obvious and proven advantages (Dunne, Jahn, and Nelson
1983) of the Princeton Evaluation Procedure:
UNCLASSIFIED T
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release 20pQ,~$~~~~p,RQ,P'~6~g0789R002200400001-3
~ Automation -Rapid and accurate analysis of a large number of
free-response sessions can be accomplished with ease.
? Archives -With the aid of computer database management, large
numbers of free-response sessions can be organized and maintained in a
usable manner.
~ Control -The cross-target scoring procedure provides a powerful
built-in within-group control.
~ Use -PEP is widely distributed and provides a commonalty of analysis
procedure across laboratories. !
Disadvantages of the PEP
There are actually very few disadvantages to PEP. A common problem that has been
observed before (Dunne, Jahn, 1982) arises in the "granularity" of the descriptor list. With any
finite list of binary-type descriptors, it is always possible that a response will appear to be correct
with "analogue" analysis procedures but will be evaluated as incorrect wjth the "digital" approach.
Another disadvantage of PEP (also noted above, op cit) is that any given descriptor list is likely to
be applicable only to a given target -pool type (i.e., Princeton area natural sites, National
Geographic magazine photographs, etc.). Lastly, one of PEP's strong points--namely, the
cross-match, built-in, within-group control--is also potentially one of its weaknesses.
Since nearly all of the various PEP scoring algorithms involve bit-by-bit weighting,
which us based upon relative probability of occurrences, a given response/target score depends not
only upon the correctness of the response, but also upon the nature of the remaining targets in the
pool. Thus, a score for a given session depends upon the quality of response and the target pool.
The followiing hypothetical example illustrates this dependency: a given target has 10 of 30 bits
present; furthermore, a few bits (e.g. 3) are particularly rare when compared to the remaining bits
(i.e. they possess comparatively large weighting factors). Let us assume that two different viewers
provide responses to this target and that each asserts 8 descriptors in the response, 6 of which are
correct.. If the first viewer's response contains only one of the rare bits, while the other viewer's
response contains all three, the second viewer's score will be considerably larger as a consequence
of the weighting factors.
Such a scoring discrepancy forces us to define what the purpose of the remote viewing
session is. If the goal is to demonstrate the existence of psi phenomena, then the PEP is a
perfectly adequate system of analysis, and it exhibits all of the advantages described above. If the
goal, however, is to demonstrate correlation effects (e.g., correlation of free-response material
with personality, physiology, environment, etc.), then the scoring difficulties described above
confound the correlation measurement.
To summarize, a target pool dependent scoring procedure provides an important
measure of a viewer's ability to discriminate from among a number of possible targets. (The
A-3
Approved For Release 20~6/68T0$-: CTAS~~~3~00789R002200400001-3
 UNCLASS~~d~~
Approved For Release 2000/08/08 :CIA- 007898002200400001-3
second viewer in the example above, for instance, would receive a higher score because his/her
response is more unique to the target pool.) The target pool dependent scoring algorithm is less
applicable, however, as an independent absolute measure of target contact--a necessary
condition for correlation studies.
If we remove the within-group control to eliminate a aowce of variance for a correlation
measwement that is potentially unrelated to psi ability, we are obligated to provide some other
form of control to demonstrate a deviation from mean chance expectation.
FIGURE OF MERIT ANALYSIS
The Figure of Merit analysis (FMA} was developed to address the problems associated with
correlation studies and to provide a novel form of control. a
Target Information
As in the PEP, the Figure of Merit- analysis quantifies the target material into binary
numbers corresponding to yes/no answers to a set of descriptors. Our descriptor list was
developed on the basis of the target material (National Geographic magazine photographs), and
on the basis of responses that might be expected a priori for ow novice remote viewers. Table 1
shows the 20 descriptors that were used for the photon production experiment (Hubbard, May,
and 1'uthoff, 1985). The questions are strongly oriented toward outdoor gestalts, typical of
National Geographic magazine material. The horizontal lines separating the descriptors into
groups of three are provided. as an aid for translating binary numbers (derived from the yes/no
answers to the questions) into an octal shorthand natation.
A self-consistency check is performed on each coded target, and a set of logically
consistent rules must be developed for a given descriptor list. One such example for the list
' (shown in Table 1) involves bits 13 and 14. While it is possible to have eland/water interface that
is not a river, canal, or channel, the reverse (i.e. to have a river, canal, or channel without having
a land/water interface) is not possible by derntion. Thus, if a target analyst asserted bit 14
without asserting bit 13, we could cansIder this an error in coding and assert bit 13. It is beyond
the scope of this paper to provide all the logical consistency Hiles, but most of them are obvious
from Table 1. Naturally. these rules must be defined in advance of any experimentation.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release 2~~~1~1~00789R002200400001-3
Table 1
Bit
No.
Descriptor
1
Is any significant part of the scene hectic, chaotic, congested, or cluttered?
2
Does a single major object or structure dominate the scene?
3
!s the central focus ar predominant?ambience of the scene primarily natural
rather than artificial or manmade?
a
4
Do the effects of the weather appear to be a significant part of the scene?
(e.g., as in the presence of snow or ice, evidence of erosion, etc.)
5
Is the scene predominantly colorful, characterized by a profusion of color,.
by a strikingly contrasting combination of colors, or by outstanding, brightly-
colored objects (e.g., flowers, stained-glass windows, -etc.--not normally
blue sky, green grass, or usual building color)?
6
Is a mountain, hill, or cliff, or a range of mountains, hills, of cliffs a significant
feature of the scene? ti
7
fs a volcano a significant part of the scene?
8
Are buildings or other manmade structures a significant part of the scene?
9
Is a city a significant part of the scene?
10
Is a town, village, or isolated settlement or outpost a significant feature of the
scene?
11
Are ruins a significant part of the scene?
12
is ,~ large expanse of water--s ecifically an ocean, sea, gulf, lake, or bay-a
i
fl
f
h
grn
s
cant aspect o
t
e scene?
13
!s a land/water interface a significant part of the scene?
14
Is a river, canal, or channel a significant part of the scene?
15
!s a waterfall a significant part of the scene?
16
Is a part or harbor a significant part of the scene?
17
Is an island a significant part of the scene?
18
Is a swamp, jungle, marsh, or verdant or heavy foliage a significant part of
the scene?
19
is a flat aspect to the landscape a significant part of the scene?
20
is a desert a significant part of the scene, or is the scene predominately dry
to the point of being arid?
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
Response Definition
The descriptor list shown in Table 1 is applied in exactly the same way in order to
define each remote viewing response. In the SRI program, remote viewers do not fill in the
descriptor list; rather, this task is performed by an analyst who is blind to the target. However, a
set of a priori defined guidelines must be established in order to aid the analyst in consistently
interpreting the responses.
Analysis
The target-pool independent scoring algorithm makes an assessment of the accuracy
and reliability of a single response when matched only against the target materiai+ used in the
session. As described above, the target and response materials are defined as the yes/no answers
to a descriptor list (Table i). Once the session material is coded into binary, we define session
reliability and accuracy as follows:
number of correct response bits
Accuracy =
number of target bits = 1
Reliability =
number of correct response bits
number of response bits = 1
In other words, the accuracy is the fraction of the target material that is correctly perceived, and
the reliability is the fraction of the response that is correct.
Neither of these measures, by themselves, is sufficient for a meaningful assessment. For
example, in the hypothetical situation in which the viewer simply reads the Encyclopedia
Britannaca as his/her response, it is certain that the accuracy would be 1.0 simply because all
possible target descriptors would have been mentioned. This would not be compelling evidence of
psi. Similarly, in a response consisting of one correct word, the reliability would be 1.0, with little
evidence of psi as well. We define the figure of merit (FM) as:
Figure of Merit =Accuracy . X Reliability
The figure of merit, which ranges between zero and one, provides an accurate assessment of the
response. In the example above where the Encyclopedia Britannaca is the response, the FM will
be low. Although the accuracy is one, the fraction of the response that is correct (i.e. the
reliability) will be very small. Likewise, in the example of a single correct word as a response, the
reliability is one, but the accuracy is low.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~l~$j~~1~~~00789R002200400001-3
A figure of merit can be calculated for each- session. For a series of sessions, the FM
may be used to assess a viewer's progress on either a session-by-session or
descriptar-by-descriptor basis or both.
ABSOLUTE FIGURE OF MERIT - A METHOD OF CONTROL
We have obtained an estimate of the meaning of FM on an absolute basis. Criven the
hypothetical situation in which ten viewers contribute 50 sessions each to a remote viewing series,
a figure of merit can be calculated by the above technique for each session. If we add the number
of responses for all viewers for each of the descriptor bits, we can obtain an estimate as to
"response/analysis" bias that may have occurred during the series. For example, if bit number 1
were asserted 40 times in S00 sessions, we can assume on the average for this series (accounting
for all known and unknown conditions) that the probability that bit 1 will be asserted in a given
response is 40/500 or 0.08. By repeating this calculation for each of the descriptor bits, we can
determine the probability of occurrence for all bits under exactly the same conditions that were
used in the series. Since this procedure displays all response/analysis biases that may have
developed during the series, we are able to use this information to constn~ct computer-generated
"random" responses, with a total absence of psi functioning, that are subject to exactly the same
biases that were observed in the series. Therefore, we are able to simulate the ideal control
condition, which addresses an important question that is frequently asked by our critics: namely,
how would an average viewer respond to a no-target session (i.e. the "monkey on a typewriter"
scenario)? A simple bit-by-bit random generation of a response is completely inadequate
because it does not account for the response biases observed during the series. The method for
producing "random" sessions that do account for the biases is described below.
A ra~idom number generator is used to create pseudo-responses that are assumed to be
devoid of psi functioning. Each bit in a given pseudo-response is generated from the empirical
"bias" described above. Once the complete response is generated, the same logical consistency
rules (described above) are applied to finalize the pseudo-response. By this technique, a large set
of pseudo-responses containing no psi information can be generated. To use these
pseudo--responses, we must select, on a random basis, targets from the same set that were used
during the series from which the biases were observed. A complete pseudo-session consists of a
single pseudo-response and a single randomly selected target. The standard figure of merit
analysis is applied to all of the pseudo-sessions in order to calculate figures of merit that have, by
definition, no psi content. The resulting FMs are fit with a gaussian distribution to provide an
estimate of the mean and standard deviation FM for random data.
Figure 1 shows the results of one such fit for a total of 300 pseudo-sessions, using the
remote viewings from aphoton-production experiment (Hubbard, May, and Puthoff, 1985) as
the bias data. From the chi-square, we note that a gaussian is a correct function to use for the fit.
Since the gaussian is truncated at zero figure of merit, we must modify the usual z-score
techniques to provide p-values for the individual session figure of merits. By definition, the
T
UNCLASSI-FLED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~~0'd~~`~1Ai ~~J6-007898002200400001-3
probability of observing a figwe of merit, fo, or greater is the area under the FM-gaussian for f ~
fo divided by the total area under the FM-gaussian. An exact p-value is calculated as follows:
Define the minimum value of a Z-like statistic as
mia Q ~
where ? and v are the mean and standard deviation of the best-fit gaussian respectively
~ -
(? = 0.132 and Q = 0.163 in the example). Define a second Z-like statistic asp
Zo =
fo - ?
where fo is the observed figure of merit. Let min and Po be the p-values calculated in the
usual way assuming minand Zo were valid z-scores. Then, the correct~p-value is given by
p-value S
Po
Pmin
Utts and May (1985) have provided an exact method for combining p-values to enable an overall
series evaluation. For mean p-values calculated for a series greater than .1, and the number of
sessions greiater than 6, a close approximation for the combined Z-score is given by (Edgington,
1972)
Zcombined - (0.50 - p ) X _ -- - _ ,
where p is the average p-value for N sessions.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~OO~~~b'"!~+A~RI!~~-007898002200400001-3
50
n
.3 .4 .5 .6 .7
Figure of Merit
.9 1.0
Figure 1 BEST-FIT GAUSSIAN TO CONTROL FMs
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release 2D~00/0810'S :'CIR-~~P9S 007898002200400001-3
We are proposing atarget-pool independent method (figure of merit analysis) for scoring
free-response material. The FMA provides a; number of advantages over previous methods.
? Figures of merit can be used in correlation studies.
? FMA provides a novel technique for free-response controls.
? Target pool independent exact p-values can be computed for each
free-response session. ~
? Since the FM is computed by simple counting, the computer coding
burden is sharply reduced.
Because of the lack of descriptor bit independence (and thus a need for logically consistent
rules) the effective number of descriptor bits iS reduced. We are presently investigating a way to
utilize a hierarchical descriptor list: that is, $ach level of the hierarchy consists of a variable
number of Independent descriptors. Finally,: the ideal descriptor list would include arbitrary
weighting factors for the level of hierarchy as well as for the individual descriptors within the level.
REFERENCES
Dunne, B.J., Jahn, R.G., and Nelson, R.D., "Precognitive Remote Perception," Engineering
Anomalies Research Laboratory, School; of Engineering/Applied Science, Princeton
University, Princeton NJ, Technical Note PEAR $3003 (August 1983)
Hubbard, ~a.S., May, E.C., and Puthoff, H.>;., "Possible Production of Photons During a
Remote Viewing Task: Preliminary Resiilts," Proceedings of the 28th Convention of the
Parapsychological Association, (Kadin, I~.I, ed.) Tufts University, Medford, MA,
(August 1985)
Jahn, R.G., Dunne, B.J., and Jahn, E.G., "Analytical Judging Procedure for Remote
Perception Experiments," The Journal of Parapsychology, Vol. 44, No. 3, pp. 207-231
(September 1980).
Utts, J.M., and May, E.C., "An Exact Method for Combining P-values," Proceedings of the
28th Convention of the Parapsychological Association, (Kadin, D.I, ed.) Tufts
University, Medford, MA, (August 1985)
UN~LAS5IFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~~/~~-00789R002200400001-3
APPENDIX B
PSI EXPERIMENTS WITH RANDOM NUMBER GENERATORS;
META-ANALYSIS PART 1
('This Appendix pis Unclassified)
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~OO~~~S`~~~R~~~J~6-007898002200400001-3
Psi Experiments with Random Number Generators:
Meta-Analysis Part 1
Dean I. Kadin
Edwin C. May
Martha J. Thomson
SRI International
Menlo Perk, California
ABSTRACT: Ameta-analysis of 332 psi experiments involving binary random number
generators is described. The combined binomial probability for data reported in 56
references published from 1969-1984 is p ~ 10-''3. A "filedrawer" analysis reveals
. that over 4500 additional, nonsignificant, unpublished or unretrieved studies would be
required to bring the overall result down to a nonsignificant level. Using a novel
approach, we estimate the actual size of the "filedrawer" to be 95 studies. Adding the
equivalent of 95 nonsignificant studies to the existing data results in p ~ 10-8, while a
meta-analysis of 98 reported control studies results In p ~+ .78. An analysis of
variance indicates that experimenters' mean z scores are significantly different from
each other. We discuss an approach and propose criteria for performing a
quality-weighted analysis on the existing data. We conclude that the prima facie
evidence supports the notion that observers' intentions can affect the statistical
properties of truly random number generators.
INTRODUCTION
This is Part 1 of a two part meta-analysis of psi experiments involving truly random number
generators (RNG) published from 1969-1984. This part describes the results of a "first-pass"
analysis, in .which the published data was taken at face value. Part 2 will report on a
quality-weighted analysis in which the results of each experiment (in terms of z score) will be
evaluated on each of a dozen criteria to produce an adjusted z score reflecting that experiment's
overall quality.
Background: On the scent of a trail
When Albert Einstein was asked about his way of thinking, he reportedly replied, "All I have is
the stubbornness of a mule; no, that's not quite all, I also have a nose" (Bower, 1985, p.330).
What he meant was that he was not only extraordinarily obstinate in tracking down solutions to
problemso he was also able to sniff out when he was on the right track. The centennial anniversary
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1 ~ w' /~~ ~~~
Approved For Release 2 bhlb 007898002200400001-3
of the American Society for Psychical Research, celebrated this year (1485), clearly demonstrates
that parapsychologists have displayed Einstein's stubbornness over the years. One question we
might ask after 100 years, however, is whether the parapsychological nose has been sniffing along a
clearly defined trail, and if so, is the trail likely to grow more fragrant or more noxious as we
progress?
There is evidence that the nose has not been shirking its duty. This can be seen in the single
most predictable feature found in the parapsychological literature, that is, the perennial call for a
replicable experiment. The ideal experiment is supposed to produce a significant result regardless
of the phase of the moon, the price of pork bellies, and the experimenter's shoe size. This Quest
for replicable experiments is by no mean ~ unique to parapsychology, however. Social and
behavioral scientists in general have been acutely aware of the slow progress in the "spfter" sciences
as compared to the natural sciences such as physics, chemistry, and biology. In experimental
psychology, for example, Epstein (1980) has stated,
Psychological research is rapidly approaching a crisis as the result of extremely
inefficient procedures for establishing replicable generalizations. The traditional
solution of attempting to obtain a high degree of control in the laboratory is often
ineffective because much human behavior is so sensitive to incidental sources of
stimulation that adequate control cannot be achieved.... Not only ire experimental
findings often difficult to replicate when there are the slightest alterations in
conditions, but even attempts at exact replication frequently fail. (p. 790)
Many observers of parapsychology (both within and outside the field) claim that the repeatable
parapsychological experiment does not exist. For example, Beloff (1977) has written, "There is
still no repeatable [psi] experiment on the basis of which any competent investigator can verify a
given phenomenon for himself" (p.759). Critics of the field have pointed to the lack of replicability
as perhaps the single most serious problem in parapsychology (e.g. Kurtz, 1981, p.12). In
response, proponents often point to significant psi studies involving ESP card-guessing (Honorton,
1975), ganzfeld stimulation (Honorton, 1978), remote perception (Dunne, Jahn, and Nelson,
1983), and RNGs (May, Hubbard and Humphrey, 1980) to indicate that there are same significant
replications.
The problem is chat from different perspectives the proponents and critics are both right. There
are indeed many psi experiments that have been repeated, but whether they are considered robust,
successful replications is the crux of the debate. One of the primary reasons for this debate, in our
opinion, is because the traditional approach of assessing the results of a set of related studies is by
descriptive literature review. Within parapsychology there are .many excellent examples of such
reviews (e.g. Carpenter, 1977; Palmer, 1982; Rush, 1982; Schmeidler, 1984; Stanford, 1977;
Stanford, 1984). Unfortunately, what one has typically learned after studying such a review is a
bodge-podge of variables, conditions, and p-values. Rarely is one left with a quantitative statement
of the degree of significance obtained in the studies as a whole.
Addressing this issue empirically, Cooper and Rosenthal (1980) demonstrated that when
knowledgeable individuals are instructed to make judgments about the overall significance of a set
of studies based on their reading of a comprehensive, descriptive literature review, it is possible for
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release I~SIb,~~I~-II~~-00789R0~~0~~~`~OP'~'~31
them to draw conclusions that are completely the opposite of the results obtained "when the same
studies are summarized by more explicit, quantitative methods.
Given the difficulties in assessing evidence from existing psi studies, is the replication trail likely
to be heading -- to reinvoke our metaphor - towards a flowering meadow or something decidedly
Iess pleasant? In general, we believe that the prospects are aromatic. In the last few years,
quantitative techniques of combining and comparing research results in systematic ways have been
developed -- called meta-analysis (Rosenthal, 1984) -- that show great promise in demonstrating
that some areas of social science have been progressing much better than previously thought. In
parapsychology, initial meta-analyses applied to ganzfeld research (Honorton, 1985), hypnotic
induction (Schechter, 1984), RNG studies (M~y, Hubbard and Humphrey, 1980; Nelson, Dunne
and Jahn, 1984; Tart, 1983), and remote viewing (Dunne, Jahn, and Nelson, 1983)ahave shown
that the overall evidence for these psi phenomena is actually quite strong.
Because meta-analysis involves the aggregation of results of numerous studies, several criticisms
of this technique have been raised (Rosenthal, 1984, p.124-132). Perhaps the three categories of
criticism most pertinant to review of parapsychological data are the fallowing: First, authors may
tend to report only the studies with significant results and leave the nonsignificant studies
unpublished (called the filedrawer problem). Second, the meta-analysis combines poorer quality
studies with better studies. And third, meta-analysis may be comparing "apples and oranges" by
combining different experiments studying different variables.
The first two problems may inflate the estimate of an overall effect; the third criticism may make
the overall summary difficult or impossible to interpret. In the present meta-analysis, however, we
actually are interested in whether these psi experiments have borne fruit, not whether they have
borne specific flavors of apples or oranges. In other words, we are not concerned with whether
hypnotic induction, say, has an effect on RNG outputs, but whether there is evidence for any psi
effect on RNG outputs. Thus, in this investigation we have concentrated on the filedrawer issue (in
this report) and the quality of studies (to be described in Part 2 of this study) .
OVERVIEW OF A TYPICAL RNG EXPERIMENT
The typical psi experiment with RNGs involves three main components: An observer (e.g. a
human, goldfish, cat or dog), a truly random number generator based on radioactive decay or
electronic noise, and an experimental task linking the observer with the device, such as a video
game, a set of instructions, a need to keep a heat lamp on or avoid a shock, and so on. The aim of
these experiments is to show that the instructions (when humans are involved) or the induced need
(when animals or plants are involved) are associated in some way -but not necessarily causally --
to the statistical output of the RNG.
For example, say an RNG was designed to produce 100 random bits at the press of a button. An
individual in this experiment might see a digital display of the number of 1's (called hits} produced
immediately after he or she pressed a button. The instructions in the experiment would typically be
to get as many hits as possible for each button press. The results of many presses, or trials, would
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 1lieta-Analysis Part 1 ~w, ~~~~~~~~~-
Approved For Release 6~ 007898002200400001-3
then be evaluated statistically, where under the null hypothesis an average of 50 hits would be
expected by chance. If the average number of hits over thousands of repetitions were say, 52, this
deviation from chance would be interpreted as evidence of a psi effect (provided that the
probability of observing this deviation was less than 1 in 20).
PROCEDURE
Because we were ultimately interested in testing among several different models of mechanisms
possibly aperating in these RNG experiments, in Part 1 of this meta-analysis (this paper) we
surveyed the parapsychological literature with two goals in mind: First, we wanted to see whether
the aggregated result of the RNG experiments showed evidence for an anamalous~effect. And
second, we needed the details of these experiments for use in evaluating a model of the underlying
mechanism. [Our modeling effort is discussed in May, Radin, Hubbard, Humphrey and Utts
(1985).]
Source of references
We searched through the five major English language parapsychological journals over the years
1969 to 1984. We also included the (refereed) Proceedings of Presented Papers for the Annual
Parapsychological Association Conventions (1971 and 1984), and a report published by the
Princeton Engineering Anomalies Research Laboratory (Nelson, Dunne and Jahn, 1984). The
literature search was started in the year 1969 because that was the year Helmut Schmidt (1969)
published the seminal RNG study that has since spawned many replications.
Defining "an experiment"
One of the difficulties faced in reviewing the articles for this meta-analysis was to decide what
constituted an experiment. In most papers, authors analyze their data repeatedly in various ways,
sometimes as a priori analyses, sometimes as post hoc afterthoughts. Even in cases of planned
analyses, there are many ways of interpreting which of several conditions is the "real" experiment.
How we decide what is an experiment is important to the meta-analysis for two main reasons:
First, the meta-analytic statistical power depends on the number of experiments we find; and
second, the z scores are different depending on haw we break down the reported results.
To illustrate the difficulty of deciding what an experiment is, consider this example. Say an
author uses three different groups of 10 percipients each (e.g. meditators, truck drivers and
athletes) and subjects each group to two different conditions (e.g. mental imagery vs. muscular
tension) in a study on psi-conducive states. The results can be broken into one big, combined
experiment, six experiments (3 groups x 2 conditions), two experiments (2 conditions), three
]. These are the Journal of Parapsychology, European Journal of Parapsychology, Rtstarch fn Parapsychology,
Journat of tht Society for Psychical Research, Journal of the American Society for Psychical Research.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Ana] sis Part 1
Approved For Release ~~'BJ~~i~~00789R00220040~001-3
experiments (3 groups), 30 experiments (subject by subject analysis), and so on. How do we
decide what to use?
We resolved this issue for this first-pass analysis in the following way: For cases where there
were multiple hypotheses under test and multiple analyses of the data, we chose as the experimental
unit the largest possible accumulation of data compatible with a single "direction of effort" assigned
to the subjects. A clearly defined direction of effort meant that the experimental protocol required
either more 1's or more 0's from the 1tNG to successfully complete the assigned task, regardless of
whether or not the subjects actually knew their task in detail.
Say, for example, a hypothesis predicted that group A would score higher than group B, and it
was stated that "higher" meant more 1 bits. }'t'hen we would take this study as two experiments:
Group A's and group B's scores. In this particular case, since group A was pre~cted to score
higher than group B, if in fact the difference between z(A) and z($) were significant, then both z
scores would be taken as positive, regardless of the reported z's. Thus if z(A) = 1.5, z(B) _ -1.0,
then the z-score difference between them would be significant one-tailed with za;cr = 1.77. If the
number of trials run in each case were 10000, then the number of hits assigned per experiment
would be hits(A)= 5075 and hits(B) = SO50, which are both positive deviations; similarly the z
scores would be recorded as z(A) = 1.5, z(B) = 1.0. If z(A) = 1.2 and z(B) _ -1.0, the z scores
would be recorded as originally reported since za;rr is not significant. The same would be true if
z(A) _ -2.0 and z(B) = 2.0. (Fortunately, such problems of interpretation were not often
encountered in the survey.)
As another example, if groups A, B, and C all tried to influence an RNG in a particular way,
and no predictions were made as to interactions, then. their overall result would be combined as one
experiment. In this way, we attempted to emphasize in the meta-analysis the underlying question
of whether or not observers could influence or otherwise affect the statistical output of an RNG
according to the stated intention of the experimenter.
Results oaf literature review
We found 73 pertinent references in the journals and reports? These references included 381
experiments contributed by 38 different principal investigators, representing about 10 different
laboratories around the world. We say "about 10 laboratories" because over the years labs have
come and gone, researchers have moved among different labs, and in many cases, one or two
individuals at an academic or private research institution are considered a "laboratory."
Breakdown of experiments Of the 381 experiments found, 332 (in 56 remaining references)
were described as using binary generators based on either radioactive decay or electronic noise.
For this meta-analysis, we considered only studies using binary 1tNGs (or any study in which the hit
rate was defined or could be interpreted as 50%) for three reasons: First, since 87% of the
experiments (332!381) employed binary generators, we felt that this sample was representative of
the entire RNG database; second, for the sake of simplicity; and, third because the test of a model
2. These references are listed under the heading "Meta-Analysis" in the references at the end of this paper.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1 A'~' e ;c' n
Approved For Release ~OD/DB~fl~':y~ ~~~G-007898002200400001-3
we developed (May, Raclin et al, 1985) requires binary statistics. In addition, to avoid the
possibility that reported p values or z scores were rounded up, whenever possible we recorded the
reported number of trials (bits generated in an experiment) and hits (number of times the
designated bit was obtained) in these experiments.
Of these 332 binary experiments, 188 were reported in journals and conference proceedings, of
which 58 were reported significant at p < .05, 2-tailed, (against 9.4 expected by chance). The
probability of observing 58 significant studies out of 188 is less than 10-sr. We refer to this body of
data as the "survey." The remaining 144 experiments were obtained from the Princeton
Engineering Anomalies Research Laboratory (Dunne, Jahn and Nelson, 1982). Of these
experiments, 13 were significant 2-tailed, resulting in p < .04 (corrected for continuity). We refer
to these experiments as the "Princeton" data.!
t
Experiments with incomplete descriptions Of the 188 survey studies, 30 were simulated by
Monte Carlo techniques because the experiment was reported as nonsignificant but neither the z
score nor the number of trials and hits were provided. To perform the simulation, we had a
pseudorandom generator (cf. May, Humphrey and Hubbard, 1980) choose a z score at random
from a normal distribution [N(0,1)], but bounded between 10-~ to 1.64 and -10 -es to -1.64.3
In five additional studies, the results were reported as significant and p oY z values were provided,
but the number of trials or hits were not given. For these five studies, since~the z score was known
or could be calculated from a p value, the trials or hits (whichever was missing) were calculated.
Table 1 shows a breakdown of the number of experiments reported in each of the seven
reference sources we used. It is clear that the reports provided in the Research in Parapsychology
series are not as detailed as one might have wished, but it is not surprising since the contents of this
reference are only abstracts of the full papers presented at the annual Parapsychological Association
conventions.
%r Table 1. Experiment breakdown by source of reference.
Reference
Experiments
with full detail
Experiments
with partial detail
Journal of the American Society for Payehiul Aesearcb
5
European Journal of Parapsychoio=y
6
Journal of the Society for Paycbical IIewareb
9
ProeeedinSe of the ParapaycholoSical Auociation?
32
Journal of Parap:ycbolo~y
49
1
Research in Parap~ycholoty
52
34
Princeton Entineerint Anomalies Research Lab
144
3. We did not generate z scores of zero because this data was ultimately used in an evaluation of our model
(May, Raclin et al, 1985), in which log(z) is taken.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 (~ ~ C LA S I F I E D Meta-Analysis Pan 1
Approved For Release ~00070$TO$ :~fA-RDP96-007898002200400001-3
In summary, of the 332 experiments we considered (188 from the survey and 144 from
Princeton), 71 were reported significant at p < .05, 2-tailed, for an overall binomial probability of
p< 5.4 x10-43.
ADDRESSING CRITICISMS CAF THE DATA
Taken as prima facie evidence, one might think that this body of published data provides
indisputable evidence that an anomaly exists. But there are numerous reasons why the data may be
suspect. The main criticisms (Akers, 1984; Hansel, 1980; Hyman, 1985; Kurtz, 1981) include
1. Results are due to chance
2. Basic statistical assur~iptions are violated
3. Only significant studies are published
4. Experiments are not replicable
5. RNGs are nonrandom
6. Poorer studies are included with better studies
Let us consider each of these six steps as successive filters for the reliability of the data. If each
criticism can be satisfactorially refuted or countered, then a persuasive case for an anomalous effect
can be made.
1. Results are due to chance
In any one experiment we cannot establish the reality of a phenomenon, regardless of the
significance level, unless strong theoretical predictions have preceeded the experiments. For
example, the recent experiments suggesting that Bell's inequality is violated (e.g. Aspect, Dalibard,
and Roger, 182; Aspect, Grangier and Roger, 1982) have been widely accepted within the physics
community on the basis of only a few empirical studies despite its profound implications on our view
of the nature of reality (cf. d'Espagnat, 1979; Mermin, 1985; Rohrlich, 1983). Parapsychology,
however, has had the disadvantage of not having a firm theoretical base on which to stand. Thus
the nature of the claim (any claimed psi effect) understandably requires extremely persuasive
evidence.
One wonders how statistically strong an effect must be to bring about a consensual agreement
within the scientific community that a psi effect on RNGs is real. Would p < 10-43 be sufficient?
If this figure were revised to take into account all of the criticisms noted above, and the end result
were say, 10-s, would that be sufficient? Clearly an overall p = .1 would not satisfy anyone, so
there is a decision curve related to this question. This curve is probably different according to
individual prejudices and predilictions, but the resolution of this question is beyond the scope of the
present paper. Note that if an anomaly did exist, it would not necessarily imply that psi was the
mediating factor. Such an anomaly may, for example, reveal some heretofore unknown statistical
peculiarities about random numbers.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Medta- rlovedP~oriRelease
App ~~1~~:~~t~~-007898002200400001-3
2. Basic statistical assumptions are violated
This criticism incorporates such problems as the improper application of statistics to a particular
experimental design, violation of assumptions of independence, performing multiple analyses on the
same data, and so on. In this meta-analysis, one of the reasons we only considered binary
generators was to simplify the statistical assumptions to the point where we could avoid many such
problems. Another reason was to avoid the "apples vs. oranges" comparison problem we
mentioned earlier. Because we were interested only in RNG experiments that reported (or where
we could calculate) the number of hits and trials, we were in fact comparing apples only with apples
(actually bits with bits). While it is true t1Aat there were many different psychological and
physiological conditions involved in these experiments, as well as human and non-hut4tan subjects,
the underlying question we asked was the same for each experiment: What was the behavior of the
RNG as campared to the pre-specified direction of effort defined in the experimental task?
The statistics in these RNG experiments are described by the well understood binomial
distribution, and the central limit theorem allows us to use the normal approximation to further
simplify the statistical treatment for the range of trials observed in the data (200 to 2 million trials in
a single experiment).
Violation of the assumption of independence can be the downfall of an otherwise tightly
controlled experiment. In the present case, however, the random events are based on sources that
are quantum-mechanical (QM) in nature -- radioactive decay of alpha, beta, or gamma particles,
or electronic noise from various semiconductor devices such as tunnel diodes. QM theory states
that random numbers based on QM events are in principle indeterminant and therefore
independent of each other, provided that the RNG device is properly designed and constructed.4
In this mega-analysis, under the null hypothesis of no psi effect we can assume independence of
random bits. Note that the assumption of independence among bits does not override proper
concern about whether the RNGs used in the experiments produced bits with equal probabilities. It
is entirely possible, for example, to produce bits that are completely independent, but with p(1) _
.6 and p (0) _ .4. This is addressed in point 5 below.
3. Only significant studies are published -the Filedrawer problem
The filedrawer problem, in which only significant studies are reported and the nonsignificant
studies languish in filedrawers, will inflate the results of ameta-analysis because there will be too
many small p values (or equivalently, to many large z scores) . To address this problem, we
followed a procedure proposed by Rosenthal (1984, p. 108), in which the average z score for all
combined studies is applied to the formula:
4. Note that some of the diodes used in noise-based RNGs are not QM in nature. RNGs that use avalanche .
diodes, for example, derive their noise from fluctuations in charge carrier multiplication, which can be
described by classical electromagnetic theory.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release g0~~1-00789R0~~0~~~~~'~'~31
K[KZ 2- 2.72]
X=
2.72
(1)
where K is the number of studies combined, Z is the mean Z obtained for the K studies, and X is
the number of new, filed, or unretrieved studies averaging null results required to bring the new
overall p level to a designated level. The value 2.72 in equation (1) is the square of 1.65, the z
value for p = .OS (the p level that Rosenthal uses). To make our filedrawer estimate more
conservative, we chose a 2-tailed p = .05, z = 1.96. Thus the formula we used was
K[KZ 2- 3x92]
r (2~
We shall consider the Princeton studies separately from the rest of the survey because we have
good reason to believe that all of the Princeton data was, in fact, published, thus their data has no
filedrawer problem. [ (Publishing all data is a part of the Princeton Laboratory's philosophy (Jahn,
1982)].
In the 188 survey experiments, the mean. zscore = 0.738. A mean z of this value over 188
experiments produces an overall z = 10.114, fora 2-tailed p < 4.9 x 104 (fee Table 1). Note that
this method of estimating the overall probability is more accurate than determining the binomial
probability of 71 successes out of 188 samples at p < .05, as described earlier in this paper.
Applying Z = .738 and K = 188 to formula (2) results in X = 4723. This means that 4723
additional studies averaging null results would have to be filed away in researchers' filedrawers to
bring the overall z score down to a 2-tailed nonsignificant level.
According to Rosenthal (1984), the number X has different meanings depending on the
research context. In some areas of research (say genetic engineering), perhaps 10 or 12
unpublished''or unretrieved studies might be considered reasonable. In other areas (say child
development), perhaps 200 to 500 filedrawer studies might be a reasonable estimate. Rosenthal
(1984, p.110) proposes the following general guideline: "Perhaps we could regard as robust to the
file drawer problem any combined results for which the tolerance level (X) reaches 5 K + 10."
Thus -- not counting the Princeton data -- since X is more than 25 times Iarger than the
observed number of studies, we could state, based on Rosenthal's guideline, that the observed
effect is robust. Indeed, for this many unpublished or unretrieved studies to exist it would have
required each of 10 parapsychology laboratories to have continuously produced nonsignificant
studies at the rate of 2.6 per month over the 15 years surveyed. This is an unlikely scenario given
the limited number of researchers performing these experiments over the years and the time and
effort typically required to perform a single study.
If we apply the same procedure to the Princeton data of 144 studies, we find the mean z = .339,
overall z = 4.063, and p < 4.85 x 10 -5. Plugging these values into formula 2, we find. we would
need X = 476 additional unpublished or unretrieved studies averaging null results. But as previously
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1 (' ~c C~~_
Approved For Release LQ +.~ 007898002200400001-3
mentioned (vide supra), the Princeton lab has claimed that they have no unpublished or filed
studies, thus this estimate of the filedrawer size is purely academic.
Another way of looking at the Princeton data is shown in Figure 1. This shows a histogram of
the absolute value of the observed z scores in light-colored bars, and a best Gaussian fit in dark
bars. As is apparent from the figure, the observed z scores are a good Gaussian fit, but the
standard deviation of the fit is not 1.0, as one would expect under the null hypothesis of z scores
chosen at random from a normal distribution, but rather the best fit Gaussian standard deviation is
1.17. A variance test between these two variances results in z = 2.90, p < .004 (2-tailed). Thus
the distribution of z scores is significantly altered from that expected by chance. This interesting
effect is discussed in more detail by Jahn, Ne~son and Dunne (1985) and May, Radin, Hubbard,
Humphrey, and Utts (1985). t
Figure 1. ~z~ score distribution for Princeton data
num~ier
of
studies
Gaussian fit
Data (144)
i i I I '1i' ~,.--
.675 1.275 1.875 2.475 3.075 3.675 4.275
~z~ scores
Estimating the actual number of filedrawer studies What if we wished to make an estimate of
the actual size of the filedrawer for the rest of the survey data? We would not be surprised to learn,
for example, that there are indeed some unpublished or unretrieved nonsignificant studies we may
have missed in our survey. To do this, we postulated what a z-score distribution might look like if
there were a filedrawer problem. Figure 2 (next page) shows a histogram of the absolute value of
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 A roved For Release Meta-Anal sis Part 1
pp ~Q,~(~~~1~6-00789R0022004~0001-3
hypothetical z scores with a filedrawer problem. Notice the discontinuity at the magic number z =
1.65 (p < .OS), which is what one would expect if nonsignificant studies remained unpublished.
Figure 2. ~z~ score distribution with filedrawer problem
IS
number
of
studies
10
~z~ scores
In Figure 3, we plot a histogram of z scores from the 188 survey studies. We also plot a double
Gaussian curve, assuming that the observed z-score curve is actually the sum of two Gaussians. The
resulting two-Gaussian curve is a good fit to the data; in fact, the sum of two Gaussians is a
significantly better fit than a single Gaussian curve (za~rr = 1.718, p < .04, 1-tailed, determined by
transforming chi-square goodness-of-fit values for one vs. two Gaussian fits into z scores, and
comparing those two z scores.)
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1
Approved For Release ~~'~SIL~~~t~-007898002200400001-3
Figure 3. ~z~ score distribution for 186 survey experiments
15
number
of
studies
Gaussian fit
Data (18b)
fl
~z~ scores
In Figure 4, we show how we estimated the actual number of filedrawer studies. We have
assumed that the observed curve (Figure 3 above) is the sum of two Gaussians (Figure 4b), shown
as two separate curves (1 and 2) in Figure 4a. We obtained estimates of the amplitude and
variance of these curves by allowing acomputer-based curve-fitting routine the freedom to vary the
amplitude and variance of each curve so the obtained fit to the curve shown in Figure 3 would be
the best possible. Under these conditions, the standard deviation (sd) of curve 1 was found to be
0.9256 and sd of curve 2 was 2.024.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 A roved For Release Meta-t~nalysis Part 1
pp ~p~L~~l,l~~l~6-007898002200400001-3
Figure 4. Method of estimating filedrawer size (see text)
(4a)
(4b)
Now that we had a full description of curves 1 and 2, we assumed that the area labeled "b" in
Figure 4a was the number of observed studies with ~z~ < 1.65 (1$8 - 76 = 112), that area "c + d"
was composed of 76 observed studies with ~z~ Z 1.65, and that the total area "a + b + c + d" was
calculated at 283 studies 5. Doing the subtraction 283 - 112 - 76 = 95, we estimate 95 unreported
or unretrieved nonsignficant studies in the actual filedrawer. We believe that this number is a more
realistic estimate than the 4700 studies determined by equation (2). In fact, 95 studies would
require each of 10 parapsychology laboratories to have filed only about 0.6 studies per year over
the 15 year .survey period (as opposed to 2.6 per month, as 4700 studies would require).
a
Now if we combine the 188 observed survey studies with 95 new, nonsignificant z scores
(generated by Monte Carlo technique with z chosen at random from a normal distribution, and
bounded between 10-25 and f1.64), we find of the 283 resulting studies, mean z = .462, overall
z = 7.768, and p < 8.03 x 10 -15. Again applying formula (2) to the new values (for the sake of
curiosity), we find X = 4078 additional nonsignificant studies needed to bring this overall p value
down top = .05, 2-tailed.
Finally, combining all survey, newly estimated, and Princeton studies (188+144), we find that
for the 425 total studies the mean z = .420, overall z = 8.684, and p < 3.9 x 10'18. Applying
formula (2), we find we would need 7778 additional nonsignificant studies in the filedrawer. Thus,
from several different perspectives, it seems that the filedrawer issue is not as serious a problem as
many have thought.
5. This calculation was based on the curve-fitted standard deviations for the two Gaussian curves and the
observed number of studies in areas b and c t d.
B-13
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 ~ ~~ ~telease ~~l~l(~~~~6-007898002200400001-3
Incidentally, testing the standard deviation of the z scores observed in these 425 studies (sd =
1.414) against the expected variance of 1.0 for a normal, unperturbed z distribution, results in a
chi-square value of 853.7 (424 df), for a p < 5.9 x 10-x. Table 1 (below) and 7 (at end of the
paper) summarize these findings.
Table 1. Summary of z score analyses
variance test
~ z's
against o' = 1
Source of
studies
z score
x
reference
(N)
z
~~
~p (2-tail)
sd
X p (2-tail)
Survey
188
0.738
10.114
4.9 x 10 - s4
1.739
- a~
568.5 4.9 x 10
Princeton
144
0.339
4.063
4.9 x 10 - s
1.184
201.9 0.001
Estimated
(simulated)
95
-0.084
-0.820
0.412
0.661
41.5 0.51
filedrawer
Combined
425
0.420
8.684
3.9 x 10 - 18
1.414
853.7 5.9 x 10 34
4. Experiments are not replicable
Occasional significant effects may be impressive, but the existence of the claimed anomaly
cannot be established on the basis of results reported by only a few individuals.g The same effect
must be replicated by many others. Is it true, as Kurtz (1980) claims, that
The basic problem ... is the lack of replicability by other experimenters. Apparently,
some experimenters -- a relative few -- are able to get similar results, but most are
unable to do so. (Italics in the original, p.12)
In fact, of the 332 experiments we considered, 78.6% failed to reach significant levels. It is
hardly surprising, then, that on the basis of examining individual experiments it is easy to reach the
conclusion that the effect is elusive and non-replicable. At this failure rate, nearly 4 out of 5
experiments will fail to reject the null hypothesis. (Of course, if just chance were operating, 19 out
of 20 experiments would fail to reject the null hypothesis.)
6. Actually, compared to experimental psychology, experimental parapsychology is in much better
shape as far as replication rates go. Honorton (1975), for instance, describes a study by Bozarih
and Roberts (]972), who, in a survey of 7334 articles from psychology journals, found only eight
articles involving replications of previously published work. In this present meta-analysis alone,
parapsychology is a factor of 40 ahead of psychology.
UNCLASSIFIED
Approved. For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release kdi~-~stc~-:S-Sds-00789R~~t~0`an~~(~'~~1 31
Another reason why it may be difficult to produce significant experiments at will is the
well-known "experimenter effect" (Rosenthal, 1976).. This effect is ubiquitous to all the sciences,
but parapsychology seems to be especially vulnerable (see, e.g. White, 1977). The experimenter
effect may help explain why some critics of parapsychology claim that they have never obtained
significant results in their attempts to replicate psi experiments (e.g. Kurtz, 1981, p.16; Neher,
1980, p.147). Of course, the odds of never obtaining a significant study can be astronomical,
depending on the number of studies conducted. Unfortunately, critics rarely report the number
and details of their claimed replications, so a goad estimate of the probability of their never seeing a
significant result cannot be made.
It should be noted that experimenter effect~is only one of many confounding problems involved
in the quest for the significant replication. For example, selection of subjects, experimenters, task
conditions, experimental protocols, statistical procedures, environmental conditions, feedback
techniques and generation of random numbers are all reflected in the ultimate outcome of an
experiment. Regardless of how well controlled an experiment may be, a change in any one of these
factors will affect the entire experiment in a complex, poorly understood way.
In any case, experimenter bias is unavoidable, and we must rely on well-controlled experiments
with features like automated data recording to help eliminate this bias. Ir? spite of tight controls,
however, it is known that even parapsychologists who would like to replicate RNG studies cannot
guarantee significant results. Thus, critics would perhaps claim that any reported significant studies
are due more to unconscious or intentional experimenter bias (i.e. fraud or carelessness) rather
than there being a real effect.
To address the issue of what effect different experimenters may have had in the reported RNG
experiments, we ran two analyses on the survey data. The first involved calculating the overall z
score obtained by each principal investigator; the second was a test of the homogeneity of mean z
scores reporied by different investigators.
Combined z score results Table 2 shows a combined z and mean z calculated for each of 28
different principal investigators. This list is comprised of only those studies where sufficient detail
was published for us to calculate z scores from the number of trials and hits in an experiment (332
total - 35 partially detailed experiments = 297 experiments). The z(overall) scores per investigator
were calculated by summing the z scores for all experiments contributed by that investigator and
dividing by the square root of the number of experiments. In effect, this weights each experiment
equally, regardless of the number of trials (bits) actually used in the experiment. (The number of
trials run in these experiments ranged between 144 and 2 million.)
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 rQ~ta- rov~etd ~orlRelease
pp ~~Q~L~S~~R~ 6-007898002200400001-3
Table 2: Overall z score per investigator
Principal t
investigator
References
Experiments
z(overall)
Andre
1
4
2.413
Bierman
1
2
3.899
Braud
2
4
3.760
Broughton
1
4
-0.470
Debes
1
8
0.35 6
Dunne
1
144
4.063
Edge
1
10
0.369
Giesler
1
12
2.694
Heseltine
4
19
-0.386
Hill
1
~ 1
2.950
Honorton
4
14
1.523
Houtkooper
1
4
0.981
Jungerman
1
1
2.332
Kelly
1
2
3.366
Matas
1
2
0.513
May z
1
1
-2.384
Millar
2
2
-0.875
Morris
1
5
1.835
Morrison
1
3
1.342
Palmer
1
1
1.750
Pantas
1
4
1.525
Radin
1
4
4.343
Randall
1
6
-0.029
Schechter
1
2
-1.060
Schmeidler
1
1
-1.273
Schmidt
9
30
13.224
Shafer
1
2
-1.440
Winnett
1
5
-0.089
TOTAL 44 297 8.548
t This is the name of the first author as listed in the references.
~2 The study by May, Humphrey, and Hubbard (1980) is not included in this survey
because their sequential analysis data collection technique is not amenable to z score analysis.
As seen in Table 2, the overall z scores for these investigators ranged between -2.384 to 13.224.
The grand total z score, obtained by summing the 28 z scores and dividing by 28 is z = 8.548,
for an overall p < 1.27 x 10 ~~ (2-tailed). If we remove Schmidt's 30 studies, since he obtained the
largest overall z score and is responsible for the largest number of references in our survey, we find
the grand total z = 6.160, p < 7.31 x 10~e (2-tailed). If we also remove the Princeton data, which
comprise nearly half of the reported experiments, we get a grand total z = 5.480, p < 4.25 x 10-g
(2-tailed). Thus, after removing the two largest contributors to the database, we are left with a
fairly impressive overall result: Odds against chance of about 1 in 23,000,000. In addition, we find
that 39% (11/28) of the experimenters obtained overall 2-tailed significance and 68% (19/2$)
obtained positive z scores.
Test for homogeneity of effect size Do different experimenters tend to observe about the same
effects in their experiments? Or are there some individuals who consistently obtain significant
r results and others do not? In the present context, to test for homogeneity of effect size among
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Ana,ysis Part 1
Approved For Release~~Q,~~St~l~~l~6-007898002200400001-3
different experimenters, we believe it makes more sense to test the individual z scores obtained in
each experiment rather than use effect sizes such as d, d', r, or so on, as discussed by Rosenthal
(1984) and others.
The reason is the following: Effect size may be defined as
significance test = [effect size] x [size of study]
where "significance test" can be a z, t, r, chi-square, or any other statistical test. In the studies we
found in the literature, it is clear that if the effect size were constant regardless of the size of the
study (say, N trials), we should be observing enormous z scores when N, is even moderately large.
For example, if an investigator ran a study with N = 100 and obtained a zscore = 2t0, this would
imply that the effect size (defined as r = 2~p = z/~f N for a binary RNG) would be
r = 2.0/ 100 = 2.0/10.0 = .2. If this effect size were constant, then if we ran the same experiment
again but with N = 10000, the z score for this experiment would be z = (2Op) 10000 = .2( 100) _
20.0. Z scores of this magnitude are simply not reported in individual experiments, thus our effect
size is almost certainly n-dependent. Indeed, this phenomenon has been observed repeatedly in a
variety of experiments and has been called agoal-directed effect (e.g. Kennedy, 1978; May, Rodin
et al, 1985; Schmidt, 1974). ~
To take the effect size n-dependence into account, we must multiply the effect size by a
function of the size of the study, which brings us back to a significance test, as noted above. For
the sake of convenience, we can use the z score calculated for each experiment. To see whether
different experimenters reported about the same magnitude z scores, we performed an analysis of
variance; the results are shown in Table 3 (on the next page).
It is clear from the results of the ANOVA that different experimenters do indeed obtain
different me,~n z scores, although with 25% (7/28) of the principal investigators reporting mean z
scores greater than 2 or less than -2, it is not the case that only one or two experimenters have
obtained large mean z scares.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-00789800220040000'1-3
 Meta-Analysis Part 1
Approved For Release 2~{~~~t~1~00789R002200400001-3
Table 3: Results of one-way analysis of variance
N
297
MEAN Z
0.5979
SD
1.5823
SE
0.0918
Person
N
MEAN Z
SD
SE
Andre
4
1.2065
1.9091
0.9546
Bierman
2
2.7570
1.3863
0.9803
Braud
4
1.879*7
0.9373
0.4887
Broughten
4
-0.2347
0.3048
0.1524
Debes
8
0.1260
1.8205
0.6437
Dunne
144
0.3386
1.1842
0.0987
Edge
10
0.1166
2.0067
0.6346
Giesler
12
0.7778
0.8011
0.2313
Heseltine
19
-0.0885
1.7124
0.3,928
Hill
1
2.9498
}
Honorton
14
0.4071
1.1328
0.3028
Houtkooper
4
0.4906
1.4944
0.7472
Jungerman
1
2.3322
Kelly
2
2.3799
0.3015
0.2132
Matas
2
0.3625
2.9522
2.0875
May
1
-2.3841
Millar
2
-0.6187
1.6406
1.1601
Morris
5
0.8206
1.0562
0.4723
Morrison
3
0.7746
0.4926
0.2844
Palmer
1
1.7500
Pantas
4
0.7625
2.4453
1.2226
Kadin
4
2.1712
0.$822
0.4411
Randall
6
-0.0120
1.1753
0.4798
Schechter
2
-0.7496
4.2411
2.9989
Schmeidler
1
-1.2728
Schmidt
30
2.4144
2.0341
0.3714
Shafer
2
-1.0178
1.1158
0.7890
Winnett
5
-0.0396
0.5795
0.2592
SOURCE SS df MS F p
person 197.9629 27 7.3320 3.631 2.78 x 10 e
error 543.1467 269 2.0191
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-00789R002200400Q01-3
 Approved For Release 2~~~'ft8A~is~~00789R00'4'~@~~I~ 1
To see whether the mean z score might be related to the number of experiments each investigator
ran, we performed a correlation between N and MEAN Z (in Table 3). Results were as follows:
Correlation r-squared t(21) p
-0.0185 0.0003 -0.0941 0.9257
In summary, taking the data taken at face value (i.e. not weighted by quality analysis), we can
make taro statements: First, considering all available data, there do appear to be significant
differences among mean z scores obtained} by different experimenters. Second, (here is a
nonsignificant correlation between the number of experiments run by principal insestigators and
their mean z scores. So to return to the question at the beginning of this section: Do different
experimenters obtain about the same results? The answer is no -- experimenters in this survey
showed mean z scores ranging from -2.38 to 2.95. As to the question of whether only one or two
individuals may be responsible for the overall significance, the answer is also no; 25% of the
experimenters in our survey reported mean z scores beyond 2 and -2.
5. RNGs were nonrandom
This criticism may be addressed by examining the results of control studies reported in the
literature. The results shown in Table 4 were compiled from 14 of the 44 detailed references
referred to in Table 3, and were contributed by the following twelve authors: Dunne (Princeton),
57 control studies; Schmidt, 23; Broughten, 8; Braud, 2; and one each for Bierman, Hill, May,
Millar, Ivlorris, Schechter, Honorton, and Palmer. The other references did not report control
results in detail and could not be used.
.ir
Table 4: Combined control studies
Data
Number of
control studies
~ z's
z
overall z
p (2-tail)
sd
Survey
41
-0.012
-0.0003
-0.002
0.999
1.036
Princeton
57
2.829
0.0496
0.375
0.708
.806
Combined
98
2.817
0.0287
0.285
0.776
.905
A variance test of the observed standard deviation (sd = .905) against the expected variance of
1.0 for 98 samples results in achi-square = 80.2645 (97 df), z = -1.22, and p < .222 (2-tailed).
Thus, for the references where control runs were described in sufficient detail to determine the
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 ~pprove~c~s~or F~elease 2~~~~~1'~I~~-007898002200400001-3
number of binary hits and trials, there is no evidence of systematic (mean or variance) bias in the
RNG equipment.
QUALITY ANALYSIS: A PROPOSAL
In this section, we address how we plan to judge the quality of the published experiments.
Quality analysis in effect adds a weighting factor to each experiment's reported z, t, or p value,
depending on the assessed quality of that experiment. To avoid making a subjective quality
assessment for each experiment, criteria and associated weights can be defined such that if a
criterion is met, the weight associated with that criterion is added to that experiment's overall
weighting factor. Rosenthal (1984, p.46-48) describes a variety of factors one fight want to
consider when performing quality analyses, but it is clear that the choice of weighting criteria
depends on the research context. For the present analysis, Table 5 shows our initial proposal for
criteria and associated weights; these are explained following the table.
Table 5: Weighting criteria for RNG quality analysis
Criteria* Weighting factors
Controls With data Without data
local control runs 30 15
global control runs 20 10
other control/random tests 10 5
target bit oscillation 10 5
Data Integrity
automatic hit/trial counters 5
tamper resistant equipment 5
automatic data recording 10
Statistical Integrity
pre-specified analysis 10
fixed run lengths 10
direction of effort stated 5
Subject type
ordinary subjects !0
special subjects 4
experimenter as subject 2
Reporting clarity
fully reported hits or trials and z 10
report of z, p, or t only 4
report of other statistics -2
"significant" only 2
"nonsignificant" only 4
' See text for explanation of criteria.
Explanation of RNG weighting criteria
Controls In Table S, a "local" control means the equipment was checked for randomness as
part of the experimental protocol. Atypical design is to have an experimental run followed by a
B-20
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1
Approved For Release 2~~~~~00789R002200400001-3
control run equivalent in all respects to the experiment run, but where the subject applies no
"effort" to the task or is absent. A "global" control means the equipment (RNG, computer, etc.)
was tested under the same conditions as used in the experiment, but separate from the
experimental sessions. "Other" control or randomness tests meant that some reference was made
to control runs or randomness tests, but the detailed results were either (a) not in the report or (b)
the explanation of the controls were referenced or related to a description in another article. The
columns labeled "With data" and "Without data" show different weights assigned to control runs
depending on whether actual data were reported. "Target bit oscillation" means the assigned "hit"
bit alternated with each newly generated bit to counterbalance any possible RNG bias.
Data Integrity The "automatic hit/trial counters" criterion is satisfied if the RNG equipment
has an automated method of keeping track of its and trials. "Automatic data record ng" requires
use of punched paper tape, magnetic tape, computer disk, ar so on, to automatically record the
data collected in the experiment. There are instances in the literature (especially ~t .reports from
the early 1970's) where the automatic counter criterion is met, but not automatic data recording.
"Tamper resistant equipment" requires either that the RNG was (a) in a locked laboratory and
inaccessible to subjects at any time, (b) the experiment was under the immediate supervision of an
experimenter, (c) the equipment had a "fail-safe" or interlock system ttlat prevented disruption of
or tampering with the data collection process, or (d) the device was a computer with software data
protection such as a password, protected files, or so on.
Statistical Integrity "Pre-specified analysis" means it is clear from the report that the
statistical analysis method was defined before data was collected. "Fixed run lengths" means the
total number of trials was specified in advance of data collection. "Direction of effort stated"
requires that it was clear whether the planned test was one-tailed or two-tailed, and what direction
of "effort" subjects were to aim for during the experiment.
Subject Integrity This category checked whether the subjects used in the experiment were
ordinary, selected or special in some other way, or the experimenter(s). Stronger weight was
applied to unselected subjects because it was felt they would have less invested in the experimental
outcome and would be less likely to intentionally or unintentionally interfere with the equipment or
procedures.
Reporting Integrity If the report included the actual number of trials and hits, or the number
of trials and a z, p, or t score, this was assigned the greatest weight. If it included only z, p, or t
scores, this was assigned less weight.. Report of any other statistics that we had to transform into the
equivalent. of z scores were assigned the lowest weight. In addition, reports consisting only of the
statement "significant," without supporting data, were assigned a weight of 2 and similarly, the
statement "nonsignificant" was assigned a weight of 4.
Method of calculating quality-weighted analysis
The weighting factor per experiment would be calculated as follows: If the criterion was clearly
present in the published report, the associated weighting factor would be added to that experiment's
T.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-f~nalysis Part 1
Approved For Release 2~D1~OY~I~SI~~i.~~00789R002200400001-3
weight. If the criterion was not met, the weight assigned for that factor would be zero (0). The sum
of the individual weights would be the overall weight per experiment, and the final overall weighted
z score is then calculated as
Weighted Z =
(5)
Thus the minimum weight per experiment would be 0 if there were no mention of control tests,
no description indicating that data collection was protected in some way, no evidence that statistical
tests were pre-planned, insufficient report on who the subjects were, and no report of results. The
maximum weight would be 125 (sum of thre8~ control weights, three data integrity weights, three
statistical integrity weights, use of ordinary subjects, and full report of data). ~
Weighting the filedrawer estimate
We propose to weight our estimated 95 nonsignificant filedrawer studies with the average weight
found in the rest of the studies. This proposal has a potential criticism; however. Our means of
estimating the filedrawer size depends on the observed z score distribution} Since the individual z
scores depend on the weighting factors (which were in effect all 1's in the analysis reported in this
paper), the unweighted filedrawer estimate may be smaller than a similar estimate made with
weighted z scores, thus inflating the final results. In response to this criticism, we would point out
that the quality analysis is actually orthogonal to the filedrawer estimate because the actual
magnitude of a z score does not change with our quality analysis, instead the importance of the z
score is affected, and the importance of a z score is not considered in our filedrawer estimation
method, only in the final estimate of overall significance.
In addition, by adding a group of nonsignificant studies (the filedrawer estimate by definition is
composed of nonsignificant studies) into a pool of z scores that have already been weighted
according to quality, we are in effect creating an ultra-conservative test. A case could be made, for
instance, on why a filedrawer estimate should not be added into aquality-weighted analysis at all,
but to take the conservative approach given the nature of the claim, we will pool the 95 estimated
studies along with the quality-weighted z scores.
Defining experiments in the Quality Analysis
Although adequate for afirst-pass analysis, the method of selecting experiments described
above would be less than perfect for aquality-weighted analysis. The main objection that could be
raised is that the decision on what constitutes the subjects' "direction of effort" is dependent on the
reviewer's interpretation of the experimental procedure. In many articles, we took educated
guesses to decide what were the actual conditions, what were the subjects' intentions, did the
authors in fact predict in advance the outcome, and so on.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1
Approved For Release 2~~1~J~~1~~~00789R002200400001-3
To address this problem in Part 2 of this meta-analysis, we will actually be performing two
separate meta-analyses. The first will take into account the minimum number of experiments that
we decide is a reasonable partitioning, and the second meta-analysis will be for the maximum
number of experiments. The two end results will be compared, and the more conservative of the
two will be used as the overall result.
Deciding on a range of possible experiments allows us to form an "uncertainty" factor for each
reference. If a reference's maximum-minimum experiment range is large as compared to the
average observed range, we must consider that the quality of that reference, at least for our
purposes, is poor. We plan on presenting a breakdown of each reference's uncertainly in the Part 2
meta-analysis to judge how clear each reference was in this study.
Example of reference source quality analysis
In Table 5 we present an example of a preliminary quality analysis applied to the source of
reference. We assigned arbitrary weights according to our perception of the quality of average
papers published in each parapsychological reference source (not counting the Princeton data).
Then, after making guesses for-these weights, we calculated a combined z;score contributed by each
journal and compared it to a weighted z score according to equation ($). `As seen in Table 5, the
original combined z score dropped by 4 orders of magnitude in significance, but the weighted z
score is still quite significant. We expect that the wider range of quality weights, as we have
proposed above, will make a larger difference in a weighted analysis, but it would appear that most
of the reports would have to be extremely poor in quality to nullify the overall p value.
Table 5. Exploratory quality analysis of reference sources
Studies Overall z
p(2-tail)
Assigned weight
Journal of Parapsychology
49 7.0$$
3.30 x 10 -13
10
Proceedings of the PA
32 $.036
4.76 x 10 - ~
2
Research in Parapsychology
$2 4.0$2
$.0$ x 10 - s
1
Journal of the ASPR
$ 4.10$
4.04 x 10 - s
$
European Journal of Parapsychology
6 3.0$2
0.002
$
Journal of the SPR
9 1.692
0.091
$
Combined unweighted result = 10.27. p < 9.45 x 10 ' is
Combined weighted result = 9.$3 p < 1.60 x 10 ' zt
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1
Approved For Release 2bl~,~~~-~I~['~,~'-007898002200400001-3
Example of chronological analysis
In Table 6 we show an analysis of variance of the 297 detailed experiments grouped according
to year of publication.
Table 6. Chronological analysis of variance
SOURCE: grand mean
year N MEAN Z
~ SD
SE
297
0.5979
1.5823
0.0918
SOURCE: year
year N
MEAN Z
SD
SE
1970 5
0.8247
3.0969
1.3850
19.71 6
0.6292
2.3180
0.9463
1972 9
1.3565
1.4253
0.4751
1973 B
4,1239
1.4665
0.5987.
1974 10
1.1539
1.9879
0.6286 ~
1975 9
1.5804
2.1841
0.7280
1976 17
0.7366
1.5784
0.3828
1977 23
0.5895
1.8333
0.3823
1978 9
-0.2520
1.1482
0.3827
1979 7
0.6012
1.2325
0.4658
1980 5
-1.1411
1.3480
0.6029
1981 7
2.3437
0.7836
0.2962
1982 164
0.3098
1.2595
0.0983
1983 1
1.7500
^1984 19
0.8492
1.0779
0.2473
.SOURCE SS df MS F p
year 151.2565 14 10.8040 5.165 1.14 x 10 ~
error 589.8531 282 2.0917
This ANOVA shows that mean z scores differ significantly from year to year. We then looked
for trends in z scores by performing a correlation between year and mean z. Results showed that
r = -0.205, t(13) _ -0.756, p = 0.463, i. e. there was no significant correlation between year of
publication and mean z score observed for that year..
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-00789R002200400Q01-3
 -~ l~~~i cc cC ~~11 Meta-Analysis Part 1
Approved For Release 2V~O~QI~I~I~D~~ 007898002200400001-3
A planned quality vs. z score correlational study
Once we perform the quality analysis and have a list of raw z scores and associated quality
weights, we plan on performing a correlation between these pairs of numbers. If the correlation is
significantly negative, it would suggest that the better the quality of a study, the lower the z score.
This would be in accordance with what some critics have claimed, namely that "there is a strong
tendency for the rate of success to increase with the number of obvious defects" (Hyman, 1983,
p.23). If a significant positive correlation is seen, however, this criticism can be refuted.
CONCLUSION
In an initial meta-analysis of psi experiments involving binary RNGs, we have identified 332
experiments published over the years 1969-1984 in 56 references. Based on an analysis of 188 of
these experiments reported in parapsychological journals, we estimated the actual number of
nonsignificant, unreported or unretrieved experiments to be 95. We found a total of 98 reported
control studies in 14 of these references. A summary of the meta-analytic results is shown in Table
7 (on the following page). .
In agreement with a hypothesis of a "psi effect" on RNGs, the combined data indicate that, in
the aggregate, the experimental conditions resulted in anomalous statistical behavior of the Rl~'G in
the direction of effort specified by the task, and the control conditions resulted in expected
binomial statistics for both mean z scores and standard deviations.
The combined data shows an interesting effect on the distribution of z scores. We find that in
the experimental condition the mean z score has been increased significantly from chance
expectation,~which in the present context is in accord with the underlying hypothesis that the z
score will shift according to the direction of the subject's effort. In the control condition we find
the z mean shifted slightly, but not significantly so. We also find that the standard deviation of the
combined distribution of experimental z scores has become significantly fatter than chance
expectation, and that the combined control standard deviation is as expected. Both of these effects
-- a shifting of the mean and fattening of the standard deviation, are accounted for in a model
discussed by May, Radin et al (1985).
Part 2 of this study will report on aquality-weighted analysis of this same data. By weighting
each study according to asemi-objective quality assessment scale, we will- address the major
criticisms of such experiments in a quantified way, and the overall experimental vs. control result
will provide a basis for discussion on whether or not this anomaly is, in fact, real.
We urge readers to comment on and criticize the method described here, and especially on the
proposed weighting criteria presented in the Quality Analysis section above. We plan to gain a
consensus opinion among informed scientists on what constitutes an agreeable, conservative
weighting scheme before we perform the. quality analysis. In this way, the combined results
observed in the weighted data will be less subject to post hoc. debate over the adequacy of xhe
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Anal;~sis Part i
Approved For Release 2~~~s1~-~1~~00789R002200400001-3
analysis method, we will avoid an enormous amount of pointless work, and we can proceed with
constructive discussion. We are especially interested in comparing the quality weights proposed by
parapsychologists, critics of psi research, and "neutral" scientists, as this may give us a clue as to
what is considered to be important in establishing consensus agreement among these different
groups.
Table 7. Summary of RNG meta-analysis
Ez's
standard
variance test
studies
!
deviation
agai
nst v = 1
Source
(N)
z
~N
p (2-tail)
Z
z scores
X
~ (2-tail)
SURVEY
Experiment
188
0.738
10.114
4.9 x 10 - Za
1.739
568.5
- a~
4.9 x 10
Control
41
-0.0003
-0.002
0.999
1.036
44.0
0.62
PRINCETON
Experiment
144
0.339
4.063
4.9 x 10 - s
1.184
201.9
0.001
Control
57
0.050
0.375
0.708
.806
~ 37.0
0.05
FILEDRAWER
ESTIMATE
Experiment
95
-0.084
-0.820
0.412
0.661
41.5
0.51
COMBINED
Experiment
427
0.420
8.684
3.9 x 10 - 18
1.414
853.7
5.9 x 10 ' sa
Co~ttrol
98
0.029
0.285
0.776
,905
80.3
0.22
' This "too small" variance in the control data is compatible with a model proposed by May, Radin et al (1985)
and is also discussed by Jahn, Nelson and Dunne (]985).
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Anal sis Part 1
Approved For Release ~~(~~&~I~C~~~-00789R00220040~001-3
REFERENCES
General References
Akers, C. Methodological criticisms of parapsychology. In S. Krippner (Ed.), Advances in
parapsychological research, Volume 4. Jefferson, NC: McFarland & Company, Inc., 1984.
Aspect, A., Dalibard, J. and Roger, G. Experimental test of Bell's inequalities using
time-varying analyzers, Physical review letters, 49, 1982, 1804-1807.
!~
Aspect, A., Grangier, P., and Roger, G. Experimental realization of
Einstein-Podolsky-Rosen-Bohm Gedankenexperiment: Anew violation of Bell's inequalities.
Physical review letters, 49, 1982, 91-94.
Beloff, J. Parapsychology and philosophy.. In B. Wolman (Ed.), Handbook of
parapsychology, New York: Van Nostrand, 1977.
Bower, B. Getting into Einstein's brain. Science News, Vol. 127, No. 21, May 25, 1985, p.
330.
Bozarth, J. D. and Roberts, R. R. Signifying significant significance. A~herican Psychologist,
Vol. 27, ll972, 774-775.
Carpenter, J. C. Intrasubject and. subject-agent effects in ESP experiments. In B. Wolman
(Ed.), Handbook of parapsychology, New York: Van Nostrand, 1977.
Cooper, H. M. & Rosenthal, R. Statistical versus traditional procedures for summarizing
research findings. Psychological bulletin, Vol. 87, 1980, 442-449.
d'Espagnat, B. The quantum theory and reality. Scientific American, November 1979,
158-181.
Dunne, B. f., Jahn, R. G., Nelson, R. D. An REG experiment with large data-base
capability, II: Effects of sample size and various operators. Technical Note PEAR 82001,
Princeton Engineering Anomalies Research Laboratory, Princeton University, School of
Engineering /Applied Science, 1982.
Dunne, B. J., Jahn, R. G., Nelson, R. D. Precognitive remote perception. Technical Note
PEAR 83003, Princeton Engineering Anomalies Research Laboratory, Princeton University,
School of Engineering /Applied Science, 1983.
Epstein, S. The stability of behavior. II. Implications for psychological research. American
Psychologist, 35, 9, September 1980, 790-806.
Fisher, R. A. Statistical methods for research workers, (2nd ed.). London: Oliver & Boyd,
1928.
Hansel, C. E. M. ESP and parapsychology: Acritical reevaluation. Buffalo, NY: Prometheus
Books, 1980.
Honorton, C. Error some place! Journal of communication, Vol. 25:1, Winter 1975.
Honorton, C. Replicability, experimenter influence, and parapsychology: An empirical context
for the study of mind. Paper presented at the meeting of the American Association for the
Advancement of Science, Washington, D. C., 1978.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analys.s Part 1
Approved For Release ~Oulp~~~`I~~~~-007898002200400001-3
Honorton, C. Meta-analysis of psi ganzfeld research: A response to Hyman. Journal of
Parapsychology, Vol. 49, No. 1, 1985, 51-92.
Hyman, R. Does the ganzfeld experiment answer the critics' objections? In W. G. Roll, J.
Beloff, & R. A. White (Eds.), Research in Parapsychology 1982. Metuchen, NJ: Scarecrow
Press, 1983, 21-23.
Hyman, R. The ganzfeld psi experiment: A critical appraisal. Journal of Parapsychology, 49,
1985, 3-50.
Jahn, R. G. The persistent paradox of psychic phenomena: An engineering perspective.
Proceedings of the IEEE, Vol. 70, No. 2, February 1982.
Jahn, R. G., Nelson, R. D., and Dunne, B. J. Variance effects in REG series score
distributions. Proceedings of the 28th Annual Parapsychological Association Convention,- Tufts
University, Medford, Massachusetts, August 12-16, 1985. t
Kennedy, J. E. The role of task complexity in PK: A review. Journal of Parapsychology, 42,
1978, 89-122 .
Kurtz, P. Is parapsychology a science? In K. Frazier, Paranormal borderlands of science.
Buffalo, NY: Prometheus Books, 1981.
May, E. C., Humphrey, B. S., and Hubbard, G. S. Electronic system perturbation
techniques. SRI International Final Report, September 30, 1980.
t
May, E. C., Radin, D. I., Hubbard, G. S., Humphrey, B. S. and Utts, J. M. Psi
experiments with random number generators: An informational model. Proceedings of the
Presented Papers of the 28th Annual Parapsychological Association Convention, Tufts
University, Medford, Massachusetts, August 12-16, 1985.
Mermin, N. D. Is the moon there when nobody looks? Reality and the quantum theory.
Physics today, April 1985, 38-47.
Neher, A~ The psychology of transcendence, Englewood Cliffs, NJ: Prentice-Hall, 1980.
Nelson, R. ~., Dunne, B. J. and .Jahn, R. G. An REG experiment with large data-base
capability, IPA: Operator related anomalies. Technical Note PEAR 84003, Princeton
Engineering Anomalies Research Laboratory, Princeton University, School of Engineering /
Applied Science, September 1984.
Palmer, J. ESP research findings: 1976-1978. In S. Krippner (Ed.), Advances in
parapsychological research, Volume 3. New York: Plenum Press, 1982.
Rohrlich, F. Facing quantum mechanical reality. Science, Vol. 221, No. 4617, September
23, 1983, 1251-1255.
Rosenthal, R. Experimenter effects in behavioral research, (rev. ed.). New York: Irvington,
1976.
Rosenthal, R. Meta-analytic procedures for social research. Beverly Hills, CA: Sage
Publications, 1984.
Rush, J. H. Problems and methods in psychokinesis research. In S. Krippner (Ed.), Advances
in parapsychological research, Volume 3. New York: Plenum Press, 1982.
Schechter, E. I. Hypnotic induction vs, control conditions: Illustrating an approach to the
evaluation of replicability in parapsychological data. Journal of the American Society far
Psychical Research, 78, 1-28, 1984.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Meta-Analysis Part 1
Approved For Release ~bA+ld~si~~-S~I~E~-007898002200400001-3
Schmeidler, G. R. Psychokinesis: The basic problem, research methods, and findings. In S.
Krippner (Ed.), Advances in parapsychological research, Volume 4. Jefferson, NC: McFarland
& Company, Inc., 1984.
Schmidt, H. Comparison of PK action on two different random number generators. Journal of
Parapsychology, 1974, 38, 47-55.
Stanford, R. G. Experimental psychokinesis: A review from diverse perspectives. In B. B.
Wolman (Ed.), Handbook of parapsychology, NY: Van Nostrand Reinhold Company, 1977.
Stanford, R. G. Recent ganzfeld-ESP research: A survey and critical analysis. In S. Krippner
(Ed.), Advances in parapsychological research, Volume 4. Jefferson, NC: McFarland &
Company, Inc., 1984.
Tart, C. T. Laboratory PK: Frequency of m$nifestation and resemblance to precognition. In
W. G. Roll, J. Beloff, & R. A. White (Eds.), Research in Parapsychology 1982. Ivjetuchen,
NJ: Scarecrow Press, 1983, 101-102.
White, R. A. The influence of the experimenter motivation, attitudes and methods of handling
subjects in psi test results. In B. B. Wolman (Ed.), Handbook of parapsychology, NY: Van
Nostrand Reinhold Company, 1977.
Meta Analysis References
Note: The following references describe psi experiments using RNGs in various ways. In this
list, the following codes are used: A ~* means the reference was used in our binary RNG
meta-analysis; a rr means the reference was also used, but the experiments mentioned in the
report were simulated due to lack of sufficient detail; ~kcr means this report contained both
detailed and non-detailed studies.
* Andre, E. Confirmation of PK action on electronic equipment. Journal of Parapsychology,
1972, 36, 283-293.
Bierman, R'f J., and Wout, N. V. T. The performance of healers in PK tests with different
RNG feedback algorithms. In J. D. Morris, W. G. Roll, & R. L. Morns (Eds.), Research in
Parapsychology 1976. Metuchen, NJ: Scarecrow Press, 1977, 131-133.
~* Bierman, R. J., and Houtkooper, J. M. Exploratory PK tests with a programmable high
speed random number generator. European Journal of Parapsychology, 1975, 1, 3-14.
rr Braud, W. Allobiofeedback: Immediate feedback for a psychokinetic influence upon another
person's physiology. In W. G. Roll (Ed.), Research in Parapsychology 1977. Metuchen, NJ:
Scarecrow Press, 1978, 123-134.
~* Braud, L., and Braud, W. Psychokinetic effects upon a random event generator under
conditions of limited feedback to volunteers and experimenter. Journal of the Society for
Psychical Research, 1979, 50, 21-30.
rr Braud, W., and Schroeter, W. Psi tests with Algernon, a computer oracle. In W. G. Roll, J.
Beloff, & R. A. White (Eds.), Research in Parapsychology 1982. Metuchen, NJ: Scarecrow
Press, 1983, 163-165.
~iFrr Braud, W. G., Smith, G., Andrew, K., and Willis, S. Psychokinetic influences on random
number generators during evocation of "analytic" vs. "nonanalytic" modes of information
processing. In J. D. Morris, W. G. Roll, & R. L. Morris (Eds.), Research in Parapsychology
1975. Metuchen, NJ: Scarecrow Press, 1976, 85-88.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 ~~~i~~F~elease ~p~p~~q,8Jp$~ ~~I~Q~~ 00789R002200400001-3
cr Broughton, R. S., and Millar, B. A PK experiment with a covert release-of-effort test. In
J. D. Morris, W. G. Roll, & R. L. Morris (Eds.), Research in Parapsychology 1976.
Metuchen, NJ: Scarecrow Press, 1977, 28-30.
rr Broughton, R., Millar, B., Beloff, J., and Wilson, K. A PK investigation of the
experimenter effect and its psi-based component. In W. G. Roll (Ed.), Research in
Parapsychology 1977. Metuchen, NJ: Scarecrow Press, 1978, 41-48.
~* Broughton, R. S., Millar, B., and Johnson, M. An investigation into the use of aversion
therapy techniques for the operant control of PK production in humans. Proceedings of the
Parapsychological Association Convention, 1979, 1-18.
Broughton, R. S. and Perlstrom, J. R. Results of a special subject in a computerized PK
game. Proceedings of the Parapsychological Al,ssociation Convention, 1984, 411-419. _
Camsira? B. PK conditioning. In W. G. Roll, R. L. Morris, & J. D. Morris (Eds.S, Research
in Parapsychology 1972. Metuchen, NJ: Scarecrow Press, 1973, 25-27.
r~ Davis, J. W. and Morrison, M. D. A test of the Schmidt model's prediction concerning
multiple feedback in a PK test. In W. G. Roll (Ed.), Research in Parapsychology 1977.
Metuchen, NJ: Scarecrow Press, 1978, 163-168.
~* Debes, J. and Morris, R. L. Comparison of striving and nonstriving instructional sets in a PK
study. Journal of Parapsychology, 1982, 46, 297-312. r
~k Dunne, B. J., Jahn, R. G., and Nelson, R. D. An REG experiment with large data-base
capability. In W. G. Roll, R. L. Morris, & R. A. White (Ed.), Research in Parapsychology
1981. Metuchen, NJ: Scarecrow Press, 1982, 50-51. Main reference is Nelson, R. D.,
Dunne, B. J. and Jahn, R. G. An REG experiment with large data-base capability, III:
Operator related anomalies. Technical Note PEAR 84003, Princeton Engineering Anomalies
Research Laboratory, Princeton University, School of Engineering /Applied Science, September
1984.
Dunne, B. J., Jahn, R. G., and Nelson, R. D. An REG experiment with large data-base
capability, II: effects of sample size and various operators. In W. G. Roll, J. Beloff, & R. A.
White (Eds.), Research in Parapsychology 1982. Metuchen, NJ: Scarecrow Press, 1983,
154-157. ~
~k Edge, H. L. Plant PK on an RNG and the experimenter effect. In W. G. Roll (Ed.), Research
in Parapsychology 1977. Metuchen, NJ: Scarecrow Press, 1978, 169-174.
~t Giesler, P. V. Differential micro-PK effects among Afro-Brazilian Caboclo and Candomble
cultist using trance-significant symbols as targets. Proceedings of the Parapsychological
Association Convention, 1984, 87-105.
* Heseltine, G. L. Electronic random number generator operation associated with EEG activity.
Journal of Parapsychology, 1977, 41, 103-118.
~* Heseltine, G. L., and Mayer-Oakes, S. A. Electronic random generator operation and EEG
activity: further studies. Journal of Parapsychology, 19?8, 42, 123-136.
~* Heseltine, G. L. and Kirk, J. H. Examination of a majority-vote technique. Journal of
Parapsychology, 1980, 44, 167-176.
~* Heseltine, G. L. PK success during structured and non structured RNG operation. Proceedings
of the Parapsychological Association Convention, 1984, 379-388.
~t Hill, S. PK effects by a single subject on a binary random number generator based on
electronic noise. In J. D. Moms, W. G. Roll, & R. L. Morris (Eds.), Research in
Parapsychology 1976. Metuchen, NJ: Scarecrow Press, 1977, 26-28.
&-30
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-00789R0022004000~1-3
 Approved For Release ~~~~~S~~I~~-00789R0'~TF~'6~O~~a3 1
-* Honorton, C. Effects of meditation and feedback on psychokinetic performance: a pilot study
with an instructor of transcendental meditation.In J. D. Morris, W. G. Roll, & R. L. Morris
(Eds.), Research in Parapsychology 1976. Metuchen, NJ: Scarecrow Press, 1977, 95-97.
~t Honorton, C., Barker, P., and Sondow, N. Feedback and participant-selection parameters in
a computer RNG study. In W. G. Roll, J. Beloff, & R. A. White (Eds.), Research in
Parapsychology 1982. Metuchen, NJ: Scarecrow Press, 1983, 157-159.
~* Honorton, C. and Barksdale, W. PK performance with waking suggestions for muscle tension
versus relaxation. Journal of the American Society for Psychical Research, 1972, 66, 208-214.
rs Honorton, C. and Tremmel, L. Psi correlates of volition: a preliminary test of Eccles'
"Neurophysiological Hypothesis" of mind-brain interaction. In W. G. Roll (Ed.), Research in
Parapsychology 1978. Metuchen, NJ: Scarec~,ow Press, 1979, 36-38.
~* Honorton, C. and May, E. C. Volitional control in a psychokinetic task with audit~iry and
visual feedback. In J. D. Morris, W. G. Roll, & R. L. Morris (Eds.), Research in
Parapsychology 1975. Metuchen, NJ: Scarecrow Press, 1976, 90-91.
~* Houtkooper, J. M. A study of repeated retroactive psychokinesis in relation to direct and
random FK effects. European Journal of Parapsychology, 1977, 4, 1-20.
* Jungerman, R. L., and Jungerman, J. A. Computer controlled random number generator PK
tests. In W. G. Roll (Ed.), Research in Parapsychology 1977. Metuchen, NJ: Scarecrow
Press, 1978, 157-162. ,
~It Kelly, E. F. and Kanthamani, B. K. A subject's efforts toward voluntary control. Journal of
Parapsychology, 1972, 36, 185-197.
Levi, A. The influence of imagery and feedback on PK effects. Journal of Parapsychology,
1979, 43, 275-289.
~* Matas, F. and Pantas, L. A PK experiment comparing meditating versus nonmeditating
subjects. Proceedings of the Parapsychological Association Convention, 1971, 12-13.
~k May, E. C. and Honorton, C. A dynamic PK experiment with Ingo Swann. In J. D. Morris,
W. G. Roll?y& R. L. Morris (Eds.), Research in Parapsychology 1975. Metuchen, NJ:
Scarecrow Press, 1976, 88-89.
Millar, B. A covert PK test of a successful psi experimenter. In J. D. Morris, W. G. Roll, &
R. L. Morris (Eds.), Research !n Parapsychology 1976. Metuchen, NJ: Scarecrow Press, 1977,
111-113.
~t Millar, B. and Broughton, R. A preliminary PK experiment with a novel computer-linked
high speed random number generator. In J. D. Morris, W. G. Roll, & R. L. Morris (Eds.),
Research in Parapsychology 1975. Metuchen, NJ: Scarecrow Press, 1976, 83-84.
*~ Millar, B. and Mackenzie, P. A test of intentional versus unintentional PK. In J. D. Morris,
W. G. Rall, & R. L. Morris (Eds.), Research in Parapsychology 1976. Metuchen, NJ:
Scarecrow Press, 1977, 32-35.
~k Morrison, M. D., and Davis, J? W. PK .with immediate, delayed, and multiple feedback: a
test of the Schmidt model's predictions. In W. G. Roll (Ed.), Research in Parapsychology
1978. Metuchen, NJ: Scarecrow Press, 1979, 117-120.
~* Morris, R. L., Nanko, M., and Phillips, D. A comparison of two popularly advocated visual
imagery strategies in a psychokinesis task. Journal of Parapsychology, 1982, 46, 1-16.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 ~pprove~'~or F~elease 2~~l~~SJI~~00789R002200400001-3
~*~ .Palmer, J. and Kramer, W. Internal state and temporal factors in RNG PK. In R. A. White
& R. S. Broughton (Eds.), Research in Parapsychology 1983. Metuchen, NJ: Scarecrow Press,
1984, 28-30.
~k Pantas, L. PK scoring under preferred and nonpreferred conditions. Proceedings of the
Parapsychological Association Convention, 1971, 47-49.
~k?~ Rodin, D. I. Mental influence on machine-generated random events: six experiments. In W.
G. Roll, R. L. Morris, & R. W. White (Eds.), Research in Parapsychology 1981. Metuchen,
NJ: Scarecrow Press, 1982, 141-142.
~* Randall, J. L. An extended series of ESP and PK tests with three English schoolboys. Journal
of the Society for Psychical Research, 1974, 47, 485-494.
r~ Schechter, E. I., Honorton, C, Barker, P.,~and Varvoglis, M. P. Relationships petween
participant traits and scores on two computer-controlled RNG-PK games. In R. A. White & R.
S. Broughton (Eds.), Research in Parapsychology 1983. Metuchen, NJ: Scarecrow Press, 1984,
32-33.
~ Schechter, E., Barker, P., and Varvoglis, M. P. A second study with the "psi ball" RNG-PK
game. In R. A. White & R. S. Broughton (Eds.), Research in Parapsychology 1983.
Metuchen, NJ: Scarecrow Press, 1984, 93-94.
~* Schechter, E. I., Barker, P., and Varvoglis, M. A preliminary study with a PK game
involving distraction from the psi task. In W. G. Roll, J. Beloff, & R. A, White (Eds.),
Research in Parapsychology 19$2. Metuchen, NJ: Scarecrow Press, 1983, 152-154.
~k Schmeidler, G. R. and Borchardt, R. Psi scores with random and pseudo-random targets. In
W. G. Roll (Ed.), Research in Parapsychology 1980. Metuchen, NJ: Scarecrow Press, 1981,
45-47.
Schmidt, H. Precognition of a quantum process. Journal of Parapsychology, 1969, 33, 99-108.
~* Schmidt, H. A PK test with electronic equipment. Journal of Parapsychology, 1970a, 34,
175-181.
~*~ Schmidt, H.;.PK experiments with animals as subjects. Journal of Parapsychology, 1970b, 34,
255-261.
Schmidt, H. An attempt to increase the efficiency of PK testing by an increase in the
generation speed. In W. G. Roll, R. L. Morris, & J. D. Morris (Eds.), Research in
Parapsychology 1972. Metuchen, NJ: Scarecrow Press, 1973a, 65-67.
~* Schmidt, H. PK tests with ahigh-speed random number generator. Journal of Parapsychology,
1973b, 37, 105-118.
~* Schmidt, H. PK effect on random time intervals. In W. G. Roll, R. L. Morris, & J. D. Morris
(Eds.), Research in Parapsychology 1973. Metuchen, NJ: Scarecrow Press, 1974a, 46-48.
~* Schmidt, H. Comparison of PK action on two different random number generators. Journal of
Parapsychology, 1974b, 38, 47-55.
Schmidt, H. Observation of subconscious PK effects with and without time displacement. In J.
D. Morris, W. G. Roll, & R. L. Morris (Eds.), Research ~in Parapsychology 1974. Metuchen,
NJ: Scarecrow Press, 1975, 116-121.
~* Schmidt, H. PK experiment with repeated, time displaced feedback. In J. D. Morris, W. G.
Roll, & R. L. Morris (Eds.), Research in Parapsychology 1975. Metuchen, NJ: Scarecrow
Press, 1976a, 107-109.
~.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 Approved For Release ~lob0,~e'i6,aASISII~bE~-00789RO~~~~C~~~a3 1
~k Schmidt, H. PK effect on pre-recorded targets. Journal of the American Society for Psychical
Research, 1976b, 70, 267-291.
~t Schmidt, H. A take-home test in PK with pre-recorded targets. In W. G. Roll (Ed.),
Research in Parapsychology 1977. Metuchen, NJ: Scarecrow Press, 1978, 31-36.
Schmidt, H. Use of stroboscopic light as rewarding feedback in a PK test with prerecorded and
momentarily-generated random events. In W. G. Roll (Ed.), Research in Parapsychology
.1978. Metuchen, NJ: Scarecrow Press, 1979a, 115-117.
a~ Schmidt, H. Search for psi fluctuations in a PK test with cockroaches. In W. G. Roll (Ed.),
Research in Parapsychology 1978. Metuchen, NJ: Scarecrow Press, 1979b, 77-78.
~* Schmidt, H. PK tests with pre-recorded and ~re-inspected seed numbers. Journal of
D.......,,..,,,u,,r,.,... ~no~ ec o~ no
~k Schmidt, H. Addition effect for PK on pre-recorded targets. Proceedings of the
Parapsychological Association Convention, 1984, 136-139.
Schmidt, H. and Pantas, L. Psi tests with psychologically equivalent conditions and internally
different machines. Proceedings of the Parapsychological Association Convention, 1971, 49-51.
Schmidt, H. and Pantas, L. Psi tests with internally different machines. Journal of
Parapsychology, 1972, 222-232.
* Schmidt, H. and Terry, J? C. Search for a relationship between brainwa'ves and PK
performance. In J. D. Morris, W. G. Roll, & R. L. Morris (Eds.), Research in Parapsychology
1976. Metuchen, NJ: Scarecrow Press, 1977, 30-32.
~k Shafer, M. G. A PK experiment with random and pseudorandom targets. In W. G. Roll, J.
Beloff, & R. A. White (Eds.), Research in Parapsychology 1982. Metuchen, NJ: Scarecrow
Press, 1983, 64-66.
Stanford, R. G., Zenhausern, R., Taylor, A., and Dwyer, M. A. Psychokinesis as
psi-mediated instrumental response. Journal of the American Society for Psychical Research,
1975, 69, 127-133.
Stanford, R~. G. "Associative activation of the unconscious" and "visualization" as methods for
influencing the PK target: a second study. Journal of the American Society for Psychical
Research, 1981, 75, 229-240.
Tart, C. T. Are prepared random sequences and real time random generators interchangeable?
In W. G. Roll, J. Beloff, & J. McAllister (Eds.), Research in Parapsychology 1980.
Metuchen, NJ: Scarecrow Press, 1981, 43-47.
Terry, J. and Schmidt, H. Conscious and subconscious PK tests with pre-recorded targets. In
W. G. Roll (Ed.), Research in Parapsychology 1977. Metuchen, NJ: Scarecrow Press, 1978,
36-41.
rT Talbert, R. and Debes, J. Time-displacement psychokinetic effects on arandom-number
generator using varying amounts of feedback. In W. G. Roll, R. L. Morris, & R. A. White
(Eds.), Research in Parapsychology 1981. Metuchen, NJ: Scarecrow Press, 1982,- 58-61.
rr Varvoglis, M. P. and McCarthy, D. Psychokinesis, intentionality, and the attentional object.
In W. G. Roll, R. L. Morris, & R. A. White (Eds.), Research in Parapsychology 1981.
Metuchen, NJ: Scarecrow Press, 1982, 51=55.
-* Winnett, R. and Honorton, C. Effects of meditation and feedback on psychokinetic
performance: results with practitioners of Ajapa yoga. In J. D. Morris, W. G. Roll, & R. L.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 ~v~- P~ Release ~~~$~~~~-007898002200400001-3
Morris (Eds.), Research in Parapsychology 1976. Metuchen, NJ: Scarecrow Press, 1977,
97-98.
UNCLASSIFIED
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
 q. 'w
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3
Approved For Release 2000/08/08 :CIA-RDP96-007898002200400001-3