A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
arXiv:1804.09635
Abstract
Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
arXiv commentNAACL 2018
References in corpus (3)
Cited by in corpus (9)
- A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks
- Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction
- Using Embeddings to Correct for Unobserved Confounding in Networks
- A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
- Longitudinal Citation Prediction using Temporal Graph Neural Networks
- S2ORC: The Semantic Scholar Open Research Corpus
- Which Review Aspect Has a Greater Impact on the Duration of Open Peer Review in Multiple Rounds? -- Evidence from Nature Communications
- Back to the Future -- Sequential Alignment of Text Representations
- Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation