The original post from EMNLP 2025
Criteria for Determining Irresponsible Reviewers
This post accompanies the ARR post Changes to reviewer volunteering requirement and incentives, and defines the specific criteria for what we will deem as “Highly Irresponsible” reviewers. While the focus here is reviewers, we will use similar criteria for determining irresponsible Area chairs (ACs).
Non-submitted reviews
If a reviewer fails to submit their reviews by the official deadline and has not submitted a personal emergency declaration (note: declaring a personal emergency after the review deadline will not be considered) will automatically be flagged as “Highly Irresponsible”.
Extremely terse or unprofessional reviews
Where the submissions are good-faith work that merits a serious review (otherwise a short review can suffice, assuming it clearly explains the fundamental problems with that work), reviews that only contain a single argument (1-2 sentences) and no constructive feedback should be flagged. We may also penalize reviews that are extremely unprofessional in tone (e.g., rude, racist, sexist, ableist, etc. content; I4 in the list of 12 commonly reported issues), even if they are otherwise detailed.
Here are some guidelines for determining whether to consider a submission to be in good faith: At minimum, a good faith article states the contribution up front and provides an evaluation that supports that. If the writing is so poor that the intended contribution can’t be identified or the article is missing an evaluation positioned as supporting that, then the article does not warrant a serious review. If the issue is just that the stated contribution is not clear, or the evaluation is not sufficient or rigorous enough, that does warrant a serious review. Furthermore, if the paper shows a naivete about the state of the art, the paper still warrants a serious review, but if the paper shows a complete lack of awareness of work in the field (for example, if virtually all of the citations are from another field), then the paper is not a good faith submission. Even interdisciplinary papers should show an awareness of the audience they are submitting their work to.
LLM-generated reviews
As per the ACL Policy on Publication Ethics, it is acceptable to use LLMs for paraphrasing, grammatical checks and proof-reading, but not for the content of the (meta-)reviews. Furthermore, the content of both the submission and (meta-)review is confidential. Therefore, even for acceptable purposes such as proofreading, it must not be passed on to non-privacy-preserving third parties, such as commercial LLM services, which may store it.
Authors will be able to flag such cases and present any evidence they have to support their allegation. While there is no definitive way of determining whether a review was (entirely) generated by an LLM, the Program Chairs will review the evidence and only proceed in cases where there is no reasonable doubt.
Flagging review process
The process is specified in the ARR post. Ultimately, all decisions will be made by the Program Chairs after a careful review of all evidence. Reviewers/ACs will be able to appeal to the publication ethics committee1 if they want to dispute the Program Chairs decisions.
https://www.aclweb.org/adminwiki/index.php/Process_for_ACL_Publication_Ethics_Review
Updated: May 06, 2025
