NDLI: Quantifying the quality of peer reviewers through Zipf’s law

Content Provider	Springer Nature Link
Author	Ausloos, Marcel Nedic, Olgica Fronczak, Agata Fronczak, Piotr
Copyright Year	2015
Abstract	This paper introduces a statistical and other analysis of peer reviewers in order to approach their “quality” through some quantification measure, thereby leading to some quality metrics. Peer reviewer reports for the Journal of the Serbian Chemical Society are examined. The text of each report has first to be adapted to word counting software in order to avoid jargon inducing confusion when searching for the word frequency: e.g. C must be distinguished, depending if it means Carbon or Celsius, etc. Thus, every report has to be carefully “rewritten”. Thereafter, the quantity, variety and distribution of words are examined in each report and compared to the whole set. Two separate months, according when reports came in, are distinguished to observe any possible hidden spurious effects. Coherence is found. An empirical distribution is searched for through a Zipf–Pareto rank-size law. It is observed that peer review reports are very far from usual texts in this respect. Deviations from the usual (first) Zipf’s law are discussed. A theoretical suggestion for the “best (or worst) report” and by extension “good (or bad) reviewer”, within this context, is provided from an entropy argument, through the concept of “distance to average” behavior. Another entropy-based measure also allows to measure the journal reviews (whence reviewers) for further comparison with other journals through their own reviewer reports.
Starting Page	347
Ending Page	368
Page Count	22
File Format	PDF
ISSN	01389130
Journal	Scientometrics
Volume Number	106
Issue Number	1
e-ISSN	15882861
Language	English
Publisher	Springer Netherlands
Publisher Date	2015-09-05
Publisher Place	Dordrecht
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Peer review Zipf’s law Rank-size rule Entropic distance Information Storage and Retrieval Library Science
Content Type	Text
Subject	Library and Information Sciences Computer Science Applications

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A web application for aggregating conflicting reviewers’ preferences

Authors and reviewers who suffer from confirmatory bias

Modified Benford’s law for two-exponent distributions

A new rank-size distribution of Zipf"s Law and its applications

Persistent nepotism in peer-review

Citation cycles and peer review cycles

On peer review in computer science: analysis of its effectiveness and suggestions for improvement

Influence of the principle of least effort across disciplines

Peer review— Experience at national and European level

Quantifying the quality of peer reviewers through Zipf’s law

Similar Documents

A web application for aggregating conflicting reviewers’ preferences

Authors and reviewers who suffer from confirmatory bias

Modified Benford’s law for two-exponent distributions

A new rank-size distribution of Zipf"s Law and its applications

Persistent nepotism in peer-review

Citation cycles and peer review cycles

On peer review in computer science: analysis of its effectiveness and suggestions for improvement

Influence of the principle of least effort across disciplines

Peer review— Experience at national and European level

Quantifying the quality of peer reviewers through Zipf’s law