How to Review an Academic Paper?
Reviewing a paper is a challenging task. Usually only a senior Ph.D. or above has the opportunity to review papers for academic conferences or journals. But in recent years, some top Artificial Intelligence conferences started inviting undergrads as reviewers due to the increasing number of paper submissions. Reviewing experience is rewarding as it helps improve your writing skills and sometimes you get recognized for high quality reviews. But how to review a paper?
- Read carefully the abstract and the introduction.
You may refer to my previous post for how to read the abstract and introduction efficiently and check if the authors answered the questions I mentioned in the post: what is the problem? is it important? what does the authors propose? what are the results?
Most importantly, you want to compare the work against previous works and judge if the contributions are significant. Yes, it is subjective. While you can try to be as objective as possible by comparing the contribution of this paper with those from other papers. They are not comparable, and that is why it has to be subjective. It is still too early to decide whether to accept or reject the paper, but you may have something in mind.
2. Read the rest of the paper.
Unfortunately, many papers under review are not well-written. You will find abbreviations without full forms, notations without definitions, wrong equation references, hard-to-understand sentences, etc. If you found the paper hard to follow, say it in your review. Authors are responsible to express their ideas well. But still, you have to try your best to understand the paper even if you want to reject it now.
Read the theorems carefully if there are. Do they make sense? I did see theorems that are obviously wrong in my reviewing experience. These are valid reasons to reject the paper, and these issues are sometimes easy to detect. I like reviewing such papers because I can make an easy decision:) Reviewers are supposed to check the proofs of theorems, too. But most reviewers do not do that because it is time-consuming and errors in proofs are usually hard to detect.
There are papers that do not have theoretical results at all. You have to evaluate the paper based on its methodology and experiments. Is the method well explained? Do you think the experiments reproducible based on the descriptions in the paper? Does the proposed method provide a significant improvement against previous methods? Are the experiments and comparisons comprehensive? Are there any flaws?
3. Write the review.
Usually, there is a review instruction asking you to write a summary and list your positive and negative comments in different areas. Do as you are asked.
If you like the paper, say it. Good efforts should always be encouraged. When I was reviewing my first few papers, I felt embarrassing if no other reviewers agree with me. It is fine. If other reviewers provide some perspective you never think of, learn from it. This is how everyone grows.
