(SUBMISSION FOR COINS SEMIANNUAL REPORT) COINS EVALUATION PANEL

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80B01139A000100110014-6
• Release Decision: RIPPUB
• Original Classification: C
• Document Page Count: 5
• Document Creation Date: December 20, 2016
• Document Release Date: July 27, 2006
• Sequence Number: 14
• Publication Date: March 14, 1968
• Content Type: REQ

Approved For Release 2006/07/27: CIA-RDP80B01139A000100110014-6
1:11\r. 'Tt
Mm -.
(Submission for COINS semiannual report)
COINS Evaluation Panel
The Evaluation Panel was reconstituted by memorandum
dated 30 January 1968 of the Chairman, COINS Committee. Its
members are:
The tasks and responsibilities assigned to the panel are as
follows:
1. Develop and present by 1 June 1968 to the COINS Committee
for approval a plan for acquiring, analyzing and presenting
data on:
a. System utilization,
b. System performance, and
c. User reaction to the system.
2. Oversee and coordinate execution of the approved plan.
3. Recommend such other data acquisition and analysis as in
the Panel's judgment is necessary or desirable in connec-
tionwith system evaluation.
CONFtDrnmTmAi_
MORI/CDF Pages _1 thru
Approved For Release 2006/07/27: CIA-RDP80B01139A000100110014-6
14 March 1968
 Approved For Release Efrp4OBO 1139A000 10011.0014-6
V40V M',MDFMT1r4r 1%W
The panel has met four times in February and March 1968
and is now organized into subgroups, each of which.is developing a
draft of one of the following portions of a plan for gathering and
analyzing data on COINS:
Hardware/ software system performance
User log form for recording certain needed data
User survey to gather additional data on circumstances
of use and user reactions
Environmental conditions in participating agencies
that may affect use
The framework of the plan that is being developed is as
follows:
1. Statistical axial eis of performance and of minimal data
on utilization and user reaction.
Data on the performance of the hardware/ software system will
be gathered systematically for a year beginning in July 1968 in
two ways -- by the system itself in a log maintained by the switch
on all traffic through the switch and by means of user log forms
completed by users at each terminal. The switch log is already
functioning and maintaining a record of the present DIA/NSA
activity. The user log form will serve to gather data on the identity
of the user and his office, the times at which queries are placed
and responses received at the terminal, the user's satisfaction
or dissatisfaction with the response, the principal reason(s) for
dissatisfaction, and an indication as to whether the same information
could have been obtained in some other way in time to serve the
user's purposes. The data from the log forms will be coded and
together with the data from the switch log will be analyzed at the
end of each two-month period throughout the year, in order to'
obtain statistics on use of each. file, respppnsetimes, J7t hLZ-4
Approved For Release 2006/07/27: CIA-RDP80BO1139AO00100110014-6
 Approved For Release 2006/07127: CIA-RDP80BO1139A000100110014-6
CONFIDENTIAL
The user log form will be tried out on some present users
before a final version of the form is recommended for use
throughout COINS.
2. More intensive analysis of utilization and user reaction.
An interview guide is being developed for in-depth interviews
of selected users in order to learn more about the circumstances
of use, with attention to both favorable and unfavorable experiences
with the system, and to obtain users' views on deficiencies.
needed improvements, training, potential use of such a system, and
alternatives to COINS. Selection of the sample of users to be inter-
viewed will be based largely on the bimonthly statistical data on
use. and user reaction.
3. Description of environmental conditions affecting use.
In order that the data on use and user reaction may be
properly understood, participating agencies will'be asked to describe
the environmental conditions in each agency that may affect use
of COINS, such as training, assistance provided, steps taken to
encourage or urge potential users to try the system, locations of
terminals in relation to potential users, and previous experience
with on-line querying of computer-based files.
On March 13, as a result of the CODIB discussion of the report
of the ad hoc Panel on Files for Use in COINS, the Evaluation Panel
was asked iif its plan for gathering and analyzing data could be
restricted to the set of ten files recommended for inclusion in
the COINS "experiment" by the CODIB Panel, even though other
files will be available for use in the system. The Evaluation Panel's
response is that in'.order to make statistical analyses and estimates
of performance characteristics of he COINS hardware/ software
system it is essential to have data on all the activity in the system.
Data on the restricted set of files can be presented separately
C0NFIrF~ Im ij
Approved For Release 2006/07/27: CIA-RDP80BO1139AO00100110014-6
 Approved For Release 2006/07/27: CIA-RDP80B01139A000100110014-6
V ` 5 J', ?
N i H 1' T 6"kL
C E
if desired, but data on the other files in the system will also
be available. The in-depth study of system utilization and user
reaction could be focussed primarily on users of the restricted
set of files, but it would seem to be preferable to defer decisions
on the composition of the interview samples until some data on
actual use of files in the system are available.
Problems
Since data on use and performance are to be gathered
from 1 July 1968, prompt attention to the training of potential
users is crucial. Otherwise we may be in the position of gathering
data on use when, in fact, very few persons know how to use
the system and query the files.
It is essential that participating agencies see to it that
a unique reference number is assigned to each query as it is
forwarded to the switch and make provision for relating the user log
form in each instance to that query reference number. It is also
essential that the agencies provide certain data on their retrieval
systems, the number and size of queues in each system, and the
numbers of local queries to each file competing with COINS queries
for processing. These requirements are being discussed informally
with those responsible for the operation of the agency systems.
Recommendations
The Panel is not yet prepared to make formal recommendations.
CONFIDENTIAL
Approved For Release 2006/07/27: CIA-RDP80B01139A000100110014-6
 Approved For Release 2006/07/27: CIA-RDP80BO1139A000100110014-6 25x1
a
*Moe
Outline of Panel's Report
A, Directive to the Panel
B. Panel's awareness of the problems of evaluating a network such
as COINS and of the limited nature of the Panel's assignment
C. Scope of the Panel's plan
What it includes
Certain things it does not include and why
Security
Cost data
]File composition and organization,, query languages,
and their impact on performance (we will have users'
reactions)
D. Factors considered in deciding upon the schedule of data
gathering, analysis,, and reporting
II. Plan for data gathering., analysis' and presentation
A, Statistical analysis of hardware/software performance and
of minimal data on utilization and user reaction
Sources of data
Switch log,, user log forms, info. on agency retrieval systems
Detailed plans for statistical analysis
Including the how and the who
Periodic reporting
B. More intensive analysis of utilization and user reaction
(Unresolved question here on non-users)
Sampling procedures
Interview guide )
Analysis of findings how and who
Reporting
0. Description of environmental conditions that may affect use
Information required of participating agencies
Preparation of Panel's comments on their possible effects
on use of COINS
III. Recommendations on other data-gathering and analysis efforts
relevant to an evaluation of experience with COINS I
SUP p. u'
AA- 6
Approved For Release 2006/07/27: CIA-RDP80BO1139AO00100110014-6