AN AUTOMATED RV EVALUATION PROCEDURE

Document Type: 
Collection: 
Document Number (FOIA) /ESDN (CREST): 
CIA-RDP96-00789R003800310001-6
Release Decision: 
RIPPUB
Original Classification: 
S
Document Page Count: 
29
Document Creation Date: 
November 4, 2016
Document Release Date: 
October 27, 1998
Sequence Number: 
1
Case Number: 
Publication Date: 
May 1, 1985
Content Type: 
REPORT
File: 
AttachmentSize
PDF icon CIA-RDP96-00789R003800310001-6.pdf1.02 MB
Body: 
Approved For Release 2000/08/10 : A- 96-00789R003800310001-6 May 1985 Final Report AN AUTOMATED RV EVALUATION PROCEDURE (U) By: EDWIN C. MAY BEVERLY S. HUMPHREY HAROLD E. PUTHOFF Prepared for: DEFENSE INTELLIGENCE AGENCY WASHINGTON, D.C. 20301 SG1J CONTRACT DAMD 17-83-C-3106 SPECIAL ACCESS PROGRAM FOR GRILL FLAME. RESTRICT DISSEMINATION TO ONLY INDIVIDUALS WITH VERIFIED ACCESS. NOT RELEASABLE TO FOREIGN NATIONALS 333 Ravenswood Avenue Menlo Park, California 94025 U.S.A. (415) 326-6200 Cable: SRI INTL MPK oved ForTK6ledi%-31N8)b8/10 : CIA- R003800310001-6 roved For Release 2000/08/10 : 60%. MOL-Awk ~0789R003800310001-6 Final Report May 1985 Covering the Period October 1983 to October 1984 ^o C~=D ~~ V - AN AUTOMATED RV EVALUATION PROCEDURE (U) By: EDWIN C. MAY BEVERLY S. HUMPHREY HAROLD E. PUTHOFF DEFENSE INTELLIGENCE AGENCY CONTRACT DAMD17-83-C-3106 WASHINGTON, D.C. 20301 Attention: SG1J SPECIAL ACCESS PROGRAM FOR GRILL FLAME. RESTRICT DISSEMINATION TO ONLY INDIVIDUALS WITH VERIFIED ACCESS. 0 ROBERT S. LEONARD, Director Radio Physics Laboratory DAVID D. ELLIOTT, Vice President Research and Analysis Division CLASSIFIED BY: DT-5A REVIEW ON: 31 May 2005 Copy No. ...... .`:`.'..... This document consists of 28 pages. SR I /G F-0275 NOT RELEASABLE TO FOREIGN NATIONALS ftf a. K% 16 1 pproved,V6r8RLbrlft,q$"OO/081 :Fe ARl P 6-O 7898-0 3800310001-6 (415) 326-6200 1 Cable: SRI INTL MPK ? TWX: 910-373-2046 Approved For ReIUN ccp( 5WI:F1 f-DDP96-00789R003800310001-6 CONTENTS LIST OF ILLUSTRATIONS . . . . . . . . . . . . . . . . . . . . . . iii LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . I OBJECTIVE . . . . . . . . . . . . . . . . . . . . . . . . . . 1 II SUMMARY OF RESULTS . . . . . . . . . . . . . . . . . . . . 2 III BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . 3 IV METHOD OF APPROACH . . . . . . . . . . . . . . . . . . . . 5 A. The Princeton Evaluation Procedure (PEP) . . . . . . . . . . . . 5 1. Target Information . . . . . . . . . . . . . . . . . . . 5 2. Response Definition . . . . . . . . . . . . . . . . . . . 6 3. Analysis . . . . . . . . . . . . . . . . . . . . . . 6 C. The SRI Evaluation Procedure (SEP) . . . . . . . . . . . . . . 7 1. Target Information . . . . . . . . . . . . . . . . . . . 8 2. Response Definition . . . . . . . . . . . . . . . . . . . 10 3. Analysis . . . . . . . . . . . . . . . . . . . . . . . 10 a. Target-Pool Dependent Scoring Algorithm . . . . . . . . 11 b. Target-Pool Independent Scoring Algorithm. . . . . . . . 15 c. Absolute Figure of Merit (FM) . . . . . . . . . . . . 19 4. Testing . . . . . . . . . . . . . . . . . . . . . . . . 20 Approved For ReWsb" SIb 1 EIIfFQP96-00789R003800310001-6 Approved For Rele N eltlXl~(g,:~itEP196-00789ROO3800310001-6 ILLUSTRATIONS 1 Score Distribution for 4995 Cross Matches . . . . . . . . . . . . . . . 16 1 Descriptor-Bit Definition . . . . . . . . . . . . . . . . . . . . . . 9 2 Binary-to-Octal Conversion . . . . . . . . . . . . . . . . . . . . . 10 3 Descriptor-Bit Weighting Factors for 112 Targets . . . . . . . . . . . . 12 4 Single Description Bit Scoring . . . . . . . . . . . . . . . . . . . . 13 5 Bit-Dependent Figures of Merit . . . . . . . . . . . . . . . . . . . 18 6 RV Evaluation Procedures Under Test . . . . . . . . . . . . . . . . 21 7 0 to 7 Point Assessment Scale . . . . . . . . . . . . . . . . . . . . 22 8 Z Scores Correlated Against the 0-to-7-Point Scale . . . . . . . . . . . 23 Approved For ReleU"%/A8G:jiffFOP96-00789R003800310001-6 Approved For Relean A?J lA pIpP96OO789ROO38OO31 0001-6 U_ CL (U) The objective of this task was to improve and automate the remote viewing (RV) evaluation procedures. (U) RV (remote viewing) is the acquisition and description, by mental means, of informa- tion blocked from ordinary perception by distance or shielding. Approved For ReleaUNJ Ji9A S4r4ID96-00789R003800310001-6 I OBJECTIVE (U) Approved For Release 2I 1 ?CIA-RDP96-00789R003800310001-6 II SUMMARY OF RESULTS (U) (S/NF) We have modified a computer-automated remote viewing analysis procedure, first developed at Princeton University, to be more responsive to the needs of the intelligence community. Our procedure is based upon defining the information content in both a RV response and its associated target as the presence or absence of a series of items (called descriptors). Various mathematical comparisons can be made between responses and targets. By defining RV accuracy as the fractional part of the target information that was correctly perceived, and defining RV reliability as the fractional part of the response that was correct, we are able to construct an RV "figure of merit " as the product of the two. The RV figure of merit is a sensitive, target-pool-independent assessment of the quality of a single, remote- viewing response. (U) We have developed a technique to assess an analysts' RV judging ability by using a standardized test case of a series of remote viewings. Judging consistancy in a training environment is the most important factor in assessment ability. Thus, it is a requirement that the same analyst assess the information content in both the response and the target. In a training environment, an analyst would first determine the information content in all of the targets in the target pool before assessing the information content in any RV response. All of the RV assessments are done without knowledge of the particular matching target. (S/NF) We have suggested ways in which a priori probabilities, on a descriptor- by-descriptor basis, can be used as RV assessments in the absence of any knowledge of the site. This technique requires the building of track records for each item on a viewer-by- viewer basis in operational settings. As the track records begin to stabilize, we will be able to integrate the analysis techniques described in this report into the operational environment. Approved For Release lA-RDP96-00789R003800310001-6 Approved For Release IA-RDP96-00789R003800310001-6 III BACKGROUND (U) (U) Since publication of the results from the initial remote viewing effort at SRI International,'* two basic questions remained about the evaluation of RV data: ? What is the definition of the target site? ? What is the definition of the RV response? For example, consider a typical IEEE-style, outbound-experimenter remote-viewing trial. After an experimenter travels to a randomly chosen location at a prearranged time, a remote viewer's (RVer) task is to describe that location. In trying to assess the quality of the RV descriptions (e.g., in a series of trials), an analyst must go to each of the sites and attempt to match responses to them. For example, while standing at a site, the analyst must decide not only the bounds of the site, but must also determine the site details that should be included in his/her analysis. While standing in the middle of the Golden Gate Bridge, should the analyst consider the buildings of downtown San Francisco, which are clearly and prominantly visible, as part of the Golden Gate Bridge target? Similarly, the RV response to the Golden Gate Bridge target might be 15 pages of dream-like free associations. A reasonable description of the bridge may be contained in the response; however, it might be obfiscated by a large amount of unrelated material. How should an analyst approach this problem? (U) The first attempt at quantitatively defining an RV response involved reducing the raw transcript to a series of declarative statements called concepts.2 Initially, it was determined that a coherent concept should not be reduced to its component parts. For example, a small red VW car should be considered as a single concept rather than four separate concepts: small, red, VW, and car. Once a transcript had been "conceptualized," that list of concepts constituted, by definition, the RV response. The analyst, then, rated the concept lists against the sites. Although this represented a major advance over previous methods, no attempt was made to define the target site. (S/NF) During the FY'82 program, we developed a procedure to define both the target and the response material.3 We learned that before a site could be quantified, a goal for the * (U) References are listed in the order of appearance at the end of this report. Approved For Release : CIA-RDP96-00789R003800310001-6 Approved For Release 2 M&E -RDP96-00789R003800310001-6 (S/NF) overall remote viewing must be clearly defined. If the goal is simply to demonstrate the existence of the RV phenomena, then anything that is perceived at the site is important. But if the goal is to gain information that is useful to the intelligence community, then specific items at the site are important, while others remain insignificant. For example, consider an office as a hypothetical target. Let us assume that a single computer in the office is of specific interest. Suppose an RVer gives an accurate description of the shape of the office, provides the serial number of the typewriter, and gives a complete description of the owner of the office. While this kind of a response might provide excellent evidence for remote viewing, the target of interest (the computer) is completely missed; thus, this response is of no interest as intelligence data. What is needed is a specific technique to allow assessments that are mission oriented. (S/NF) The procedure developed during FY'82 was a first attempt at solving the mission orientation problem. In this technique, the transcript is conceptualized as described above, and a similar process is applied to the sites. A target site is conceptualized as a set of target elements, which are to be considered "mission independent." In the office example above, target elements might be: desk, safe, window, telephone, computer, and chair. A second layer of conceptualization is then applied, which is "mission specific." Each target element is assigned a number between 1 and 5 corresponding to the mission's relevance. Again, in the office example, the computer would be assigned a relevance factor of 5 (most relevant), while all other target elements would be assigned a factor of 1 (least relevant). The target elements and their relevance factors constitute the site definition and mission orientation. The final report for the FY'82 task3 described in detail how a mission specific assessment was made. Although the procedure proved to be quite sensitive, it was nonetheless cumbersome and difficult to apply. (S/NF) This report describes a major advance over the FY'82 technique. The original idea, which involves computer-automated scoring of RV data, was developed at the Anomalies Laboratory of Princeton University.4 We have significantly extended and modified the Princeton technique and have developed procedures that can be used in actual intelligence applications. Approved For Release A-RDP96-00789R003800310001-6 Approved For Release 2 ? IA-RDP96-00789R003800310001-6 IV METHOD OF APPROACH (U) (S/NF) The overall method of approach was to begin with the Princeton group's known evaluation procedure, then determine what would be appropriate for our environment. The next step was to expand the analysis concept to be more responsive to intelligence requirements, and to integrate the entire procedure with our on-line data bases. A. (U) The Princeton Evaluation Procedure (PEP) (U) In general, the Princeton Evaluation Procedure (PEP) is based on comparing a priori, quantitatively-defined target information with similarly quantitatively-defined response information. (A complete description of this procedure can be found in Reference 4.) The procedure was developed for use as a research tool in the university environment, where complete knowledge of the target sites could be obtained. Once the target and response information was defined, the PEP applied various methods of mathematical comparisons to arrive at a meaningful assessment score. 1. (U) Target Information (U) The definition of a particular target site (usually outdoor sites in and around Princeton, New Jersey) was contained in the yes/no answers to a set of questions called descriptors. These descriptors were designed in such a way as to characterize the typical Princeton target. By definition, the only target information to be considered in the analysis was completely contained in the yes/no answers of the descriptor questions for that site. For example, one descriptor from their list, "Are any animals, birds, fish, major insects, or figures of these significant in the scene?" defines the animal content of the site. The question would be answered "yes" for a zoo and a pet store target, but answered "no" for a typical campus building target. Similarly a set (30 for the PEP) of yes/no responses constitutes the target information. Approved For Release 2 -RDP96-00789R003800310001-6 Approved For Release 2 / 1 IA-RDP96-00789R003800310001-6 fto no MW a 2. (U) Response Definition (U) The descriptor list for the target sites is used as a definition of the response as well. For a given RV session, an analyst (blind to the target site) attempts to answer the 30 questions based entirely on the single RV response. Using the same example above, an analyst would have to decide if a particular verbal passage or a quick sketch could be interpreted as animals or not. For some responses this might be an easy task, "I get a picture of a purple cow." Most responses, however, require a judgement, "I hear a funny sound and there may be an odd smell in the air." Nonetheless, the yes/no answers to the 30 questions constitute the only response information that will be used in the analysis. (U) For a given response/target combination, the information is strictly contained in the yes/no answers to the descriptors. A binary number (30 bits long for PEP) is constructed for the target and the response descriptor questions respectively. A yes answer is considered a binary "1" while a no answer is considered a binary "0." The resulting two 30-bit binary numbers can then be compared by a variety of mathematical techniques to form a score for that specific RV session. For a series of RV sessions, a quantitative assessment is made by comparing a given response (matched to its corresponding target site) against the scores computed by matching the response to all other targets used in the series. This procedure has the added advantage of a built-in, within-group control. In other words, this assessment determines the uniqueness of the target/response match compared with all other possible matches for the series. B. (U) Problems with the PEP (S/NF) There are a number of problems with the PEP when the conditions under which the PEP was developed are no longer valid. Because we are trying to develop an RV analysis procedure that is useful both in the RV training environment as well as in intelligence applications, we have identified four basic problems with using the PEP for our purposes: ? The bit descriptors were not appropriate for our training environment. ? The PEP was not interfaced to a standard data base management system (DBMS). Approved For Release 2~-RDP96-00789R003800310001-6 Approved For Release 20NIMIWW A-RDP96-00789R003800310001-6 400064 (S/NF) ? The cross-target scoring procedure was not sensitive to intelligence requirements. ? Any cross-target scoring procedure is inappropriate for a training environment. (U) As stated above, the PEP descriptors were optimized for natural outdoor sites in the Princeton area. Because we planned to use different target material, the PEP descriptor list was completely inadequate. Having obtained the computer codes used at Princeton, we noticed that the PEP required a special on-line, within-code data base. We felt this was an inefficient way to procede because we already had most of our data in a commercial DBMS, Ingres.5 (S/NF) One of the principal problems of RV used as an adjunct to conventional intelligence collection techniques is that RVers tend to add information, sometimes called analytical overlay (AOL), to the response. If training techniques are to be developed that are sensitive to intelligence requirements, they must attempt to inhibit AOL. Specifically, any training analysis procedure must be particularly sensitive to the addition of extraneous information. The PEP was completely insensitive to this requirement. (U) We also observed that for the purposes of training, any scoring procedure that cross compares a training response against all targets in the target pool, might penalize excellent RV simply because of the lack of target pool orthogonality (i.e., how different one target is to the next). For example, consider a typical National Geographic Magazine photograph of a flat desert showing few features. A very good description of this site will also match many other similar sites in the target pool. Thus, a comparison of the actual match with others in the pool will tend to reduce the score for reasons other than the quality of the particular RV response. (U) We, therefore, felt obligated to modify the PEP in such a way to address the above criticisms. C. (U) The SRI Evaluation Procedure (SEP) (S/NF) The SRI Evaluation Procedure (SEP) was developed to address not only various RV training programs, but also the potential application of the SEP to intelligence problems. Thus, it was recognized that the SEP must contain cross comparison analytical procedures that Approved For Release 20~A-RDP96-00789R003800310001-6 Approved For Release 20 1A-RDP96-00789R00380031 0001-6 (S/NF) were sensitive to AOL, and at the same time, provide a meaningful assessment of RV responses that were independent of other targets in the pool. (S/NF) As in the PEP, the SRI Evaluation Procedure quantifies the target material into binary numbers corresponding to yes/no answers to a set of descriptors. Before any of the training programs had begun, a descriptor list was developed on the basis of the target material (National Geographic Magazine photographs), and on the responses that might be expected for novice RVers. Table 1 shows the 20 questions (descriptors) that were used for the Alternate Training Task.6 This descriptor list, while applicable to a novice RV training environment, is not appropriate for either advanced training or intelligence applications. The questions are strongly oriented toward outdoor gestalts typical of National Geographic Magazine material. Each descriptor list must be tailored to the application requirements. The horizontal lines separating the descriptors in groups of three are an aid in translating binary numbers (derived from the yes/no answers to the questions) into an octal shorthand notation. (U) To illustrate exactly how a target might be coded into an octal number, let's consider a photograph of San Francisco on a clear day showing the bay, the central city sky- scrapers, and the centrally-located hill (Twin Peaks). Referring to Table 1 Bit Numbers 1, 6, 8, 9, 12, 13, 16 and 17 would all be answered "yes" and thus would be assigned a binary "1." The remaining questions would all be answered "no" and thus be assigned a binary "0." Starting with Bit Number 1 on the left, the binary number that defines the information for this target is 10000101100110011000. This representation, while convenient for computers, is difficult for humans; therefore, we convert it to the octal representation as a shorthand. Using the horizontal lines shown in Table 1 as divisions, we consider each triad of bits as a binary number ranging from 000 to 111. Table 2 shows the binary-number triad to octal conversion factors. (U) Rewriting the above binary number with triad separations for clarity, we have 10 000 101 100 110 011 000. Using Table 2, we find that this binary number converts to 20546308. This octal number is the shorthand notation for all the information contained, by definition, in the San Francisco target example. All targets in the data base are coded by the same technique. Approved For Release CFA-RDP96-00789R003800310001-6 Approved For Rele(aff ~ff/Q8,41? i f I(r.F QP96-00789R003800310001-6 (U) DESCRIPTOR-BIT DEFINITION Bit No. Descriptor 1 Is any significant part of the scene hectic, chaotic, congested, or cluttered? 2 Does a single major object or structure dominate the scene? 3 Is the central focus or predominant ambience of the scene primarily natural rather than artificial or manmade? 4 Do the effects of the weather appear to be a significant part of the scene? (e.g., as in the presence of snow or ice, evidence of erosion, etc.) 5 Is the scene predominantly colorful, characterized by a profusion of color, by a strikingly contrasting combination of colors, or by outstanding, brightly- colored objects (e.g., flowers, stained-glass windows, etc.--not normally blue sky, green grass, or usual building color)? 6 Is a mountain, hill, or cliff, or a range of mountains, hills, or cliffs a significant feature of the scene? 7 Is a volcano a significant part of the scene? 8 Are buildings or other manmade structures a significant part of the scene? 9 Is a city a significant part of the scene? 10 Is a town, village, or isolated settlement or outpost a significant feature of the scene? 11 Are ruins a significant part of the scene? 12 Is a large expanse of water--specifically an ocean, sea, gulf, lake, or bay--a significant aspect of the scene? 13 Is a land/water interface a significant part of the scene? 14 Is a river, canal, or channel a significant part of the scene? 15 Is a waterfall a significant part of the scene? 16 Is a port or harbor a significant part of the scene? 17 Is an island a significant part of the scene? 18 Is a swamp, jungle, marsh, or verdant or heavy foliage a significant part of the scene? 19 Is a flat aspect to the landscape a significant part of the scene? 20 Is a desert a significant part of the scene, or is the scene predominately dry to the point of being arid? Approved For ReI EO S IEEFBP96-00789R003800310001-6 Approved For Release 20AWN)o IA-RDP96-00789R003800310001-6 (U) BINARY-TO-OCTAL CONVERSION Binary Triad Octal Equivalent 000 0 001 1 010 2 011 3 100 4 101 5 110 6 111 7 2. (U) Response Definition (U) The descriptor list shown in Table 1 and the coding techniques described using Table 2 are prepared in exactly the same way to define each RV response. For a particular training program, however, a set of a priori guidelines must be defined in order to aid an analyst in interpreting the various aspects of the training procedure with regard to the descriptor list. For example, it might be correct within a given training context to advise the analyst to consider all isolated lines as a land/water interface, and set descriptor Bit Number 13 by definition. How this is done is completely dependent upon the particular training procedure in question. For an example see Alternate Training.8 3. (U) Analysis (S/NF) The SRI evaluation procedure involves two different types of analysis: ? Target-pool-dependent analysis (intelligence assessment) ? Target-pool-independent analysis (training). (U) The first of these involves descriptor weighting that gives more or less credit in the final score in accordance with an a priori defined algorithm. It is within this analysis that Approved For Release 2b A-RDP96-00789R003800310001-6 Approved For Releas - U ~ 1~ I ftFP9600789R00380031 0001-6 (U) penalties are levied for "inventing" information that is not present at the site. The target- pool-independent analysis involves a straightforward counting system that depends upon a single target/response information comparison. a. (U) Target-Pool-Dependent Scoring Alogrithm (U) Consider a finite set of targets, N, each of which has been coded in accordance with Table 1. Define a weighting factor Wj = where Pj is the probabililty of occurrence of Descriptor Bit j (j = 1,20) and is given by Pi = the number of targets that have Bit j present The weighting factors will be large for descriptors that are not common, and small for common elements in the target pool. Table 3 shows an example of a set of probability of occurrences and weighting factors taken from the Alternate Training Task. This table was derived from a set of 112 National Geographic Magazine photographic targets. We see from Table 3 that volcanos (Bit 7) are the rarest item in the target pool, and are thus allotted the highest weighting factor of 9.337. While correctly remote viewing a volcano will significantly increase an RVers score, inventing one where there is none will be heavily penalized. (U) Before we construct an assessment score for a single target/response, we must define the scoring algorithm, and determine a method by which scores can be compared. Consider a single target and RV response to that target. Suppose further that the information contained in each has been coded in accordance with methods described above. The scoring proceeds as follows. In considering a single descriptor bit, j, in an RV response, there are four possible ways to match (or not match) that bit with its corresponding bit in the target: Approved For Relea1Jt?ctcA S 1H.E 96-00789R003800310001-6 Approved For Release 00L%8I1 Q ~ffl(YP96-00789RO03800310001-6 UNC (U) DESCRIPTOR-BIT WEIGHTING FACTORS FOR 112 TARGETS Bit No. Probability Occurrence of Weighting Factor 1 0.4821 2.074 2 0.5089 1.965 3 0.5804 1.723 4 0.2857 3.500 5 0.1875 5.333 6 0.5893 1.697 7 0.1071 9.337 8 0.5268 1.898 9 0.2143 4.666 10 0.2768 3.613 11 0.1964 5.092 12 0.3125 3.200 13 0.5804 1.723 14 0.2768 3.613 15 0.1786 5.656 16 0.1786 5.656 17 0.1339 7.468 18 0.3482 2.872 19 0.3304 3.027 20 0.1786 5.599 (U) ? The target bit and the response bit are zero ? The target bit is one; the response bit is zero ? The target bit is zero; the response bit is one ? The target bit is one; the response bit is one. Approved For ReleaU NIUJS SdIF96-00789R003800310001-6 Approved For Reinif 1XV9 i FftYP96-00789R003800310001-6 (U) While there are a number of ways to proceed (the PEP considers them all), we will confine our discussion to that particular method of comparison that met the requirements stated above. Because it is difficult to know if a descriptor bit scored as zero is the result of correct or incorrect RV, a meaningful score can only be constructed from asserted information. Thus, the SEP only considers the case in which there is an assertive response (i.e., the RVer positively states that a particular descriptor is present). Table 4 shows the contribution to the assessment score for all four cases (single-bit comparison) shown above. (U) We see from Table 4 that if the RVer correctly identifies a target descriptor bit, he/she is awarded a large contribution to the score if the item is rare (i.e., the probability of occurrence is small), and not as much if the item is common. Likewise, if the RVers invent an item, they are penalized more if the item is rare. To analyze the complete response, the values shown in Table 4 are added to the score--depending upon the correctness of the bit-by-bit match. (U) SINGLE DESCRIPTION BIT SCORING Bit j Contribution Target Response to Score 0 0 0 1 0 0 1.0 0 1 - P. 1 1 1.0 P. J Approved For ReleaU tAGIoAl$S& Ii P96-00789R003800310001-6 Approved For Release 2Ep/4 gt.,$ I P96-00789R003800310001-6 (U) To complete the target-dependent scoring algorithm, it is necessary to normalize the score described above in such a way that comparisons can be made from session to session. In the PEP, a number of different normalizing factors were explored, but we have chosen to use the "perfect score" as our normalization. (U) Let Ti and Rj be the value of the target and the response bit j, respectively. The most negative score possible would result from inventing all items in the descriptor list not present in the target. Conversely, the most positive score possible would result in correctly identifying all present target descriptors. Let N and N be the most positive and the most negative score, respectively. They are given by + n T. N = j=0 P. n Ti N P. j=0 P. where n is the number of descriptors (20 in the example), and Tj and is zero when Tj is one. Thus, n T. X R. Tj X Rj Sr S P. P. is one when Tj is zero, Approved For ReleasA JiAtS:iiAC-IED6-00789R003800310001-6 Approved For ReIE /Q8J ? if (U) For the normalize score, S, to be in the range from -1, to 1, S = 2 ( Sr N ) 1 - N (U) To convert the normalized score for each RV session to a meaningful statistic, all sessions in a series are scored against all targets in the pool except for the matching target. Thus, for M RV sessions and a target pool of N targets, there would be (N X M) - M such cross matches. Figure 1 shows a sample distribution of scores for 4995 cross matches. The solid points are the data and the smooth curve is the best fit gaussian to the data. (U) Having completed the cross matches and constructed the best fit gaussian, statistical Z scores are calculated from the RV session scores by Z = Q where ? and v are the mean and the standard deviation of the cross-match best-fit gaussian, respectively. The Z score for each session is a measure of the uniqueness of the target/ response match compared with the remainder of the target pool, and it represents the final output of the target-pool dependent scoring algorithm. b. (U) Target-Pool-Independent Scoring Algorithm (U) The target-pool-independent scoring algorithm makes an assessment of the accuracy and reliability of a single RV response matched only against the target material used in the session. As in the case of the target-pool-dependent algorithm, the target and response materials are defined as the yes/no answers to a descriptor list (similar to that shown in Table 1). Once the session material is coded into binary, we define session reliability and accuracy as follows: Approved For Releas ?t*s 5I &-00789R003800310001-6 Approved For Relerasff //8~4 g 1 f I P96-00789R003800310001-6 0 -0.4 -0.2 UNCLASSIFIED 0.4 SCORE FIGURE 1 SCORE DISTRIBUTION FOR 4995 CROSS MATCHES Approved For ReleasLWWAS W-49 6-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) Reliability = number of correct response bits number of target bits = 1 number of correct response bits number of response bits = 1 In other words, the accuracy is the fraction of the target material that was correctly perceived, and the reliability is the fraction of the response that was correctly perceived. (U) Neither of these measures by themselves is sufficient for an RV assessment. Consider the hypothetical situation in which the RVer simply reads the Encyclopedia Britanica as his/her response. It is certain that the accuracy would be 1.0 simply because all possible target elements would have been mentioned, and thus would not be evidence of RV. Similarly, consider a response consisting of one correct word. The reliability would be 1.0, with little evidence of RV as well. We define the figure of merit (FM) as Figure of Merit = Accuracy X Reliability The figure of merit which ranges between zero and one, provides a more accurate RV assessment. In the example above where the Encyclopedia Britanica is the response, the FM will be low. Although the accuracy is one, the fraction of the response that is correct (the reliability) will be very small. Likewise, in the example of a single correct word as a response, the reliability is one, but the accuracy is low. (U) A figure of merit can be calculated for each RV session to assess the progress in an RV training environment. For a series of RV sessions, the FM may be used to assess a viewer's progress on a descriptor-by-descriptor basis as well. Table 5 shows an example of FMs calculated for 22 training sessions. The "bit number" corresponds to the descriptors shown in Table 1. The "number of responses" indicates the number of sessions (out of 22) that each descriptor was asserted; the "number of targets" indicates how many targets (also out of 22) that each descriptor was asserted. The "accuracy" and "reliability" are the fraction of correct target and response material on an individual descriptor basis. Approved For Release'2000/0$/10S CI~FjOP-996-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) BIT-DEPENDENT FIGURES OF MERIT Bit No. Number of Responses Number of Targets Ace- uracy Reli- ability Figure of Merit 1 8 14 0.500 0.8750 0.438 2 1 10 0.000 0.000 0.000 3 9 13 0.539 0.778 0.419 4 3 6 0.167 0.333 0.056 5 0 6 0.000 0.000 0.000 6 13 12 0.500 0.462 0.231 7 1 2 0.000 0.000 0.000 8 14 12 0.750 0.643 0.482 9 3 4 0.750 1.000 0.750 10 0 8 0.000 0.000 0.000 11 0 2 0.000 0.000 0.000 12 2 7 0.143 0.500 0.071 13 17 15 0.733 0.647 0.475 14 6 8 0.250 0.333 0.083 15 5 6 0.000 0.000 0.000 16 1 6 0.000 0.000 0.000 17 1 2 0.000 0.000 0.000 18 3 9 0.111 0.333 0.037 19 9 8 0.500 0.444 0.222 20 0 2 0.000 0.000 0.000 UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) Finally, the "FM" is the figure of merit for each bit, For example, Bit Number 9 (city descriptor) was in the targets 4 out of 22 times. This viewer responded with "city" 3 out of 22 times. Of the 4 times a city was present in the target, the viewer correctly identified the city 3 times (thus an accuracy of 0.75). Of the 3 times the viewer responded with city, he/she was correct all the time (thus a reliability of 1.00). Therefore, the figure of merit for the city descriptor is 0.75. From the FMs of all the bits, we see that this viewer is particularly adept at remote viewing cities. Considering a large number of remote viewings, it is possible by this technique to build "viewing signatures" or track records for each viewer. When applied in the application environment, the bit-dependent figure of merit can be used as a guideline for task-specific viewer selection. c. (U) Absolute Figure of Merit (FM) (U) We have obtained an estimate of the meaning of FM on an absolute basis. Suppose ten viewers have contributed 50 sessions each to a training series. Each session has a figure of merit associated with it that has been calculated by the above techniques. If we add the number of responses for all viewers for each of the descriptor bits, we can obtain an estimate as to "response/analysis" bias that may have occurred across the training session. For example, suppose, of the 500 sessions, Bit Number 1 was asserted 40 times. On the average, we can assume for this training series the probability of Bit 1 being in a response is 40/500 or 0.08. By repeating this calculation for each of the descriptor bits, we can determine the probability of occurrence for all bits under the same conditions used in RV training. (U) To determine the absolute FM distribution, a random number generator is used to create pseudo responses that are assumed to be free of psychoenergetic functioning. Each bit in a given pseudo response is generated from the emperical "bias" described above. Once the response is generated, simple descriptor-bit logical consistency is applied to finalize the pseudo response. By this technique, 10 sets of 50 pseudo responses containing no RV information can be generated. The next step is to select, on a random basis, targets from the set that were used during an actual training period to complete the pseudo sessions. The standard target-pool-independent analysis is applied to the pseudo sessions to calculate figures of merit that have, by definition, no psychoenergetic content. The histogram of FMs is fit with a gaussian distribution to provide an estimate of the mean (?) and standard deviation (a) UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) FM for random data. Since this gaussian distribution is truncated at zero FM, we must use the following procedure to determine the p-value for a given figure of merit, f. Calculate an observed z-score, z = (f - g)/or- Determine an intermediate p-value, p', in the usual way given z. Calculate a normalization z-score, zo = -g/v. Determine a normalization p-value, po, as in Step (2). Calculate the correct p-value, p = p' /po. For example, during the Alternate Training Task for FY'84, the mean and standard deviation calculated as described above was 0.132 and 0.163 respectively. Therefore, using the above procedure in reverse, we find that any FM greater than 0.417 can be considered as significantly above chance. 4. (U) Testing (U) We used the baseline data from the FY'84 Alternate Training Task to test the PEP and the SEP scoring procedures. Three analysts were asked to apply a number of techniques to the set of 6 sessions from 6 RVers each. The procedures and analysis technology that was used are summarized in Table 6. Using the descriptor list shown in Table 1, the three analysts independently scored the target pool, which consisted of 112 National Geographic Magazine photographs, and the set of 36 RV responses. After the scoring was completed, the three analysts met with two experienced RV judges and reached a consensus of RV quality for all 36 responses, using the 0 to 7 point assessment scale shown in Table 7. (U) Linear correlation coefficients were calculated (using the target-dependent Z scores as the dependent variable) for Procedures 1 through 5 correlated against Procedure 7 (0 to 7 point assessment) in Table 6. From the results of these correlations, we were able to assess the effectiveness of each of the RV evaluation procedures, then determine the relative judging ability of the three analysts. UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) RV EVALUATION PROCEDURES UNDER TEST No. Procedure Technology 1 Concept Analysis Target/Response Concepts (equal weights) 2 PEP--Full Scoring* Descriptor List Analysis (computer scored) 3 PEP--Selective Scoringt Descriptor List Analysis (computer scored) 4 SEP--Full Scoringt Descriptor List Analysis (computer scored) 5 SEP--Selective Scoringt Descriptor List Analysis (computer scored) 6 Post Hoc Assessmentt 0 to 7 Point Scale * Scoring includes all response bits, asserted or not. t Scoring includes only asserted response bits. t Assessment scoring done after all others. UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED (U) 0 TO 7 POINT ASSESSMENT SCALE Score Assessment Criteria Excellent correspondence, including good analytical detail 7 (e.g., naming the site), with essentially no incorrect information. Good correspondence with good analytical information 6 (e.g., naming the function), and relatively little incorrect information. 5 Good correspondence with unambiguous, unique matchable elements, but some incorrect information. 4 Good correspondence with several matchable elements intermixed with incorrect information. 3 Mixture of correct and incorrect elements, but enough of the former to indicate viewer has made contact with the site. 2 Some correct elements, but not sufficient to suggest results beyond chance expectation. 1 Little correspondence. 0 No correspondence. UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 UNCLASSIFIED V RESULTS AND DISCUSSION (U) (U) The first, and most striking, result was the necessity for the RV response coder and the target coder to be the same individual. Correlations between all scoring methods and the 0-to-7-point assessments were calculated for all possible cross-coder combinations. Only those correlations corresponding to the case where the coder of responses and targets was the same analyst, were statistically significant. This was the expected result because an analyst might be willing to adopt a liberal scoring attitude (i.e., find most descriptors present) in both the responses and targets, whereas a second analyst might adopt a conservative scoring procedure and assign few descriptors as present. As long as a particular analyst's "bias" is consistant for the targets and responses, a good assessment of RViability can be made. Thus, in the results described below, no cross-coder data are considered. (U) Table 8 shows the linear correlation coefficients (which were calculated for all procedures listed in Table 6 against the 0-to-7-point assessment scale) for each of the three analysts. Because all the correlations are statistically significant, any analyst/procedure (U) Z SCORES CORRELATED AGAINST THE 0-TO-7-POINT SCALE Princeton SRI Analyst Full Selective Full Selective 374 0.462 0.410 0.566 0.523 642 0.388 0.364 0.385 0.339 802 0.530 0.433 0.503 0.453 UNCLASSIFIED Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 C I! (U) combination would provide good RV assessments. The correlation coefficients averaged over all analysts were 0.431 and 0.462 for the Princeton and the SRI procedures respectively. While this difference is not significant, there is a bias in favor of the SRI procedure. Within the SRI procedure, No. 642 was the least consistent analyst. There were no significant differences between the full and the selective scoring. (S/NF) In summary, we have developed a computer-based RV analysis tool that is applicable for both the training and operational environment. The figure-of-merit analysis allows target-pool-independent assessment of the relative progress of RV trainees. Within a given training program absolute probabilities (against chance) can be assigned for a single training session. (S/NF) By carefully creating an appropriate operational descriptor list, and by tracking figures of merit on a bit-by-bit basis, the techniques described above are applicable to the operational environment. The figure-of-merit analysis requires that complete descriptor information of the site be known. In the operational setting, as feedback information is available, descriptor track records (figure-of-merit analysis) can be kept over many sessions to provide accuracy and reliability data on a viewer-by-viewer basis. Thus, viewers can be selected on the basis of their a priori probabilites on the operational descriptors of interest, and a priori assessments of their responses can be made by using the same track record. Teew Approved For Release 2000/08/10 : CIA-RDP96-00789R003800310001-6 Approved For Relq t0A 9I FWEUP96-00789ROO3800310001-6 VI REFERENCES (U) 1. H. E. Puthoff and R. Targ, "A Perceptual Channel for Information Transfer Over Kilometer Distances: Historical Perspective and Recent Research," Proc. IEEE, Vol. 64, No. 3 (March 1976) UNCLASSIFIED. 2. R. Targ, H. E. Puthoff, and E. C. May, "1977 Proceedings of the International Conference of Cybernetics and Society," pp. 519-529, UNCLASSIFIED. 3. E. C. May, "A Remote Viewing Evaluation Protocol (U)," Final Report (Revised), SRI Project 4028, SRI International, Menlo Park, California (July 1983) SECRET. 4. R. G. Jahn, B. J. Dunne, and E. G. Jahn, "Analytical Judging Procedure for Remote Perception Experiments," The Journal of Parapsychology, Vol. 44, No. 3, pp. 207-231 (September 1980) UNCLASSIFIED. 5. E. C. May, "Data Base Management (U)," Final Report, SRI Project 7048, SRI International, Menlo Park, California (October 1984) SECRET. 6. E. C. May and B. S. Humphrey "Alternate Training Task," Final Report, SRI Project 7048, SRI International, Menlo Park, California (October 1984) SECRET. Approved For ReleaseU t(A S$AIR -00789R003800310001-6