NetNews Usenet Archive 1993 #1/NN_1993_1.iso/spool/bit/listserv/vpiejl/327

Text File | 1993-01-04 | 11.1 KB | 201 lines

Comments: Gated by [e-mail]
Path: sparky!uunet!europa.asd.contel.com!paladin.american.edu!auvm!PRINCETON.EDU!HARNAD
Approved-By:  James Powell <[e-mail]>
Message-ID: <[e-mail]>
Newsgroups: bit.listserv.vpiej-l
Date:         Mon, 4 Jan 1993 08:47:40 EST
Sender:       "Publishing E-Journals : Publishing, Archiving,
              and Access" <[e-mail]>
From:         Stevan Harnad <[e-mail]>
Subject:      Electronic Peer Review: PSYC Call for Commentators
Lines: 188

The following article has just been published in PSYCOLOQUY, a
refereed electronic journal of peer commentary.
Commentary is now invited. Instructions for Commentators appear
after the article. Please submit commentaries to [e-mail]
----------------------------------------------------------------------
psycoloquy.92.3.67.peer-review.1.stodolsky   Thursday, 31 December 1992
ISSN 1055-0143                   (9 paragraphs, 1 reference, 131 lines)
PSYCOLOQUY is sponsored by the American Psychological Association (APA)
                Copyright 1992 David Stodolosky

        INVITATIONAL JOURNALS BASED UPON EDITORIAL CONSENSUS:
        A NEW EDITORIAL ROLE IN ELECTRONIC JOURNAL PUBLICATION

                David Stodolsky
                Dept. of Computer Science, Bldg. 20.1
                Roskilde University
                DK-4000 Roskilde, Denmark
                [e-mail] (or [e-mail])
                ([phone]

    0.0  ABSTRACT: Objectivity has long been both an ideal in
    scientific communication and the basis for accreditation claims by
    authors of journal articles, yet the evaluation of articles for
    publication proceeds on a mostly subjective basis. This has been
    particularly true in the case of "invitational journals," where an
    unassisted editorial judgement may determine whether or not
    something is published. A "consensus journal" has been proposed,
    that uses a rigorous measurement model based on multidimensional
    peer judgements to generate invitations, but this model assumes an
    infrastructure that is not currently available. An invitational
    system based on similarity judgements could also yield objective
    performance measurements. Such a system could be a transitional one
    between currently existing editorial mechanisms and those of a
    consensus journal.

    0.1  KEYWORDS: invitational journal, editorial role, electronic
    publication, accreditation, measurement, peer review, consensus

1.1  A major factor in the transition from paper to electronic journals
concerns the effective use of judgmental resources. The intensity of
evaluation is much greater with electronic journals: Each manuscript
may be reviewed by dozens or even hundreds of persons before it is seen
by an average reader. The typical process of review by two or three
referees prior to publication, common with paper journals, is
inappropriate in the computer-network environment.

1.2  Stodolsky (1990) presented a role-free model for invitational
journals based upon peer consensus. This model is limited in that it
assumes preexisting dimensions for evaluative judgments and the
availability of a robust statistical procedure for calculating
consensus positions among reviewers. The objective of issuing
invitations to the most knowledgeable proponents of different consensus
positions can also be achieved by creating an editorial role. This may
require a redefinition of what is meant by a "consensus" position and a
"most knowledgeable" proponent. It is not clear that the same
performance can be achieved with this new model. However, the new model
can be used to build the set of preexisting dimensions needed to
operate the "fully automatic" invitational journal. The approach
discussed here is "manual," because editorial intervention is required
before invitations can be issued. However, as with the earlier model,
statistical measures of performance are available. We can call this new
model an "invitational journal based upon editorial consensus."

1.3  For the purpose of this discussion, we assume that articles
persist in a database until they are withdrawn. Editorial expertise is
measured by analyzing, in the first instance, the agreement of editors
in issuing invitations. The final criteria depend on what articles are
actually submitted, whether any are determined to be redundant, and how
long articles persist before they are withdrawn.

1.4  After reading a published article, potential authors submit short
(e.g., one-paragraph) reactions to the target article. These reactions
are treated as proposals for new articles and directed to a jury of
editors (this could be done by a set of corresponding editors if more
then one jury was available). The editors then independently select a
subset of the proposals judged to be mutually exclusive and
representing the most competent opinion. The judgment of mutual
exclusivity can be performed by sorting proposals into groups. Within
each group, the potential authors would be addressing the same
question. Between groups there would be a significant difference in
what question was being treated (or how it was to be treated).
Discrepancies among groupings by editors would require a "consensus"
set of groupings to be calculated. Editors' performance could be
calculated from their judgements using rater reliability statistics.
Later articles and responses to them would be the final criterion as to
whether two articles were, in the final analysis, distinctive
(addressing different questions or the same question in a different
way).

1.5  When two or more proposals were sorted into the same group, an
author would be selected on the basis of expertise. In the simplest
case, the editorial choice could also include a judgement of author
expertise, based upon the proposal submitted. A more objective
mechanism would base choice on previously demonstrated performance. For
example, an author might be issued a credential for each week a
submitted article was maintained in the database. Then, in the case of
nonexclusive proposals, the author with the largest number of
credentials (in that subject area) would automatically be issued an
invitation. A further consideration could be the past performance of
the author in responding to invitations. For example, an author who
failed to respond to invitations 10% of the time might lose future
invitations one of ten times to less competent peers.

1.6  Similar objective mechanisms could be applied to editorial
responses, allowing editors' judgments to be weighted according to
previous performance. A system of credentials and selection mechanisms
could also be developed, as suggested above for authors. The final
criterion for editorial performance could be the number of
article-weeks generated by invitations.

1.7  Given a history of operation for the journal structure described,
the groupings made by editors could be used as a basis for generating
dimensions for evaluative judgments. A reliable set of such dimensions
would permit more efficient sorting of proposals within the framework
of an invitational journal based upon editorial consensus; it would
also permit the testing of the more powerful invitational structure
based upon peer consensus.

1.8  Thus, the invitational journal based upon editorial consensus can
be viewed as a stepping stone between current invitational journals
such as sci.psychology.digest (PSYCOLOQUY's Usenet Edition) and the
more advanced invitational structures proposed by Stodolsky (1990). The
transition from current practice requires employing more people in the
editorial process. However, it also permits greater automation in the
administration of a journal. A very important benefit is the
quantitative measurement of editorial performance that could be used to
support claims of journal quality.

REFERENCE

Stodolsky, D. S. (1990). Consensus Journals: Invitational journals
based upon peer consensus. PSYCOLOQUY 1(15) psyc.arch.1.15.90.
Also appeared in: Datalogiske Skrifter (Writings on Computer Science)
No. 29 1990. Roskilde University Centre, Institute of Geography,
Socioeconomic Analysis, and Computer Science (ISSN 0109-9779-29).
------------------------------------------------------------------
   INSTRUCTIONS FOR PSYCOLOQUY AUTHORS AND COMMENTATORS

PSYCOLOQUY is a refereed electronic journal (ISSN 1055-0143) sponsored
on an experimental basis by the American Psychological Association
and currently estimated to reach a readership of 20,000. PSYCOLOQUY
publishes brief reports of new ideas and findings on which the author
wishes to solicit rapid peer feedback, international and
interdisciplinary ("Scholarly Skywriting"), in all areas of psychology
and its related fields (biobehavioral, cognitive, neural, social, etc.)
All contributions are refereed by members of PSYCOLOQUY's Editorial Board.

Target articles should normally not exceed 500 lines in length
(commentaries and responses should not exceed 200 lines). All target
articles must have (1) a short abstract (<100 words), (2) an indexable
title, (3) 6-8 indexable keywords, and the (4) author's full name and
institutional address. The submission should be accompanied by (5) a
rationale for soliciting commentary (e.g., why would commentary be
useful and of interest to the field? what kind of commentary do you
expect to elicit?) and (6) a list of potential commentators (with their
email addresses). Commentaries must have indexable titles and the
commentator's full name and institutional address (abstract is
optional). All paragraphs should be numbered in articles, commentaries
and responses (see format of already articles articles in PSYCOLOQUY).

It is strongly recommended that all figures be designed so as to be
screen-readable ascii. If this is not possible, the provisional
solution is the less desirable hybrid one of submitting them as
postscript files (or in some other universally available format) to be
printed out locally by readers to supplement the screen-readable text
of the article.

PSYCOLOQUY also publishes multiple reviews of books in any of the above
fields; these should normally be the same length as commentaries, but
longer reviews will be considered as well. Book authors should submit a
500-line self-contained Precis of their book, in the format of a target
article; if accepted, this will be published in PSYCOLOQUY together
with a formal Call for Reviews (of the book, not the Precis). The
author's publisher must agree in advance to furnish review copies to the
reviewers selected.

Authors of accepted manuscripts assign to PSYCOLOQUY the right to
publish and distribute their text electronically and to archive and
make it permanently retrievable electronically, but they retain the
copyright, and after it has appeared in PSYCOLOQUY authors may
republish their text in any way they wish -- electronic or print -- as
long as they clearly acknowledge PSYCOLOQUY as its original locus of
publication. However, except in very special cases, agreed upon in
advance, contributions that have already been published or are being
considered for publication elsewhere are not eligible to be considered
for publication in PSYCOLOQUY,

Please submit all material to [e-mail] or [e-mail]