Science 2.0 - Peer-reviewed publications

By alterednate from Flickr

Robert Boynton gives us a cautionary tale of then-University of Chicago political science junior faculty Daniel Drezner. He was denied tenure because part of his academic output had been directed into the blogosphere.

The current antipathy toward blogging may have something to do with the fact that universities have no tools for judging blogs. And most people agree that blogs would need to be evaluated through some kind of peer-review mechanism if they are to be taken into account. "It is utterly absurd to propose giving someone credit for activity with no barriers to entry," [John Holbo, the Editor of The Valve] says.

So, how might a blog be peer-reviewed? The market provides a number of viable models. eBay, for one, has established an efficient rating system for buyers and sellers, based on the number and quality of transactions they execute. In a noncommercial medium, Slashdot uses a "Moderation and Meta Moderation System," in which moderators are awarded higher or lower "karma" according to how well they police the discussions on the site. (The "Meta Moderation System" judges the moderators' moderators.)

An even better model is Kuro5hin.org, which I think has successfully adapted the peer-review process to the Web. The manuscript submission process should sound familiar:

1. The author contribue articles by formatting the manuscript (or "stories" in the world of Kuro5hin) with respect to a standard recommendation.


2. The author submits the manuscript to the edit queue, along with plans for editing the manuscript


3. A subset of Kuro5hin users are selected to be editors of this particular mansucript. They see a link to the manuscript in their Kuro5hin "moderation" pages. At this point, the text is not visible to the rest of the Kuro5hin community.


4. Editorial commments are visible to the author, who can respond to the comments, make changes to the manuscript, and even rate the usefulness of each individual editorial comment


5. Editors vote on acceptance of this manuscript


6. Assume it is accepted, the manuscript becomes an article. Another subset of Kuro5hin users are select to vote on the placement of the article: on the Frontpage or in one of the Sections (e.g., Politics, Technology, Culture, etc).


7. Now the entire Kuro5hin community can read the article and comment.


8. Each reader can rate other readers' comments. The author can likewise respond to the questions and comments.

Possible improvements:

• Allow editors to tack on metadata to the text. This seems a natural task for editors, who will be doing fact-checking as a part of the editing process.


• Allow for multiple authors per manuscript for collaborative efforts.


• Anonymous editorial process to encourage frank discussions.


• Giving some structure to the comment system, making it easier to track questions that have been answered by the author and those that have not.


• As noted by many, a credit system for rewarding editorial work and constructive comments.

John Holbo's post on this on The Valve.


Technorati Tags:
Science 2.0 peer-review