How to properly review AI papers?
3 likes656 views
The document provides guidelines and best practices for reviewing AI papers, emphasizing the importance of constructive criticism and timely submissions. It highlights challenges faced by new reviewers and stresses the need to evaluate papers based on their novelty and clarity. Key principles include promoting a respectful community, being responsible in the review process, and ensuring good writing to aid reviewers.
![UNIVERSITY of ROCHESTER
Outline
● 他山之石 What other experienced researchers have said
○ Rick Szeliski (Facebook) [slides]
○ Jordi Pont-Tuset (Google) [slides]
○ Fatma Güney (Koç University) [slides]
○ Konrad Schindler (ETH Zürich) [slides]
○ Michael Goesele (Facebook) [slides]
● Main Principles
○ 己所不欲， 勿施于人
○ It takes a community 社区是由每个人组成的
○ Look for reasons to accept
○ Writing is important
○ The No. 1 goal of top conferences/journals
○ What if you are not familiar with the area?
4](https://image.slidesharecdn.com/howtoproperlyreviewaipapers-200710022751/85/How-to-properly-review-AI-papers-4-320.jpg)
How to properly review AI papers?
- 1. UNIVERSITY of ROCHESTER How to Properly Review AI Papers? Jiebo Luo Editor-in-Chief, IEEE Transactions on Multimedia Program Chair, CVPR 2012 07/10/2020 Valse Webinar 1
- 2. UNIVERSITY of ROCHESTER Adversarial Tasks ● How to write an AI paper? ● How to review an AI paper? 2
- 3. UNIVERSITY of ROCHESTER Motivations ● The explosive growth in AI related fields ○ Need I say more? (7000 submissions x 3 reviews) / 7 = 3000 reviewers ● The rise of new and inexperienced reviewers ○ PhD students ○ MS students ○ Undergraduate students ○ Researchers from other fields ● Lack of systematic training for reviewers ○ Unsupervised learning from personal (often bad) experiences ● A well recognized problem for the communities 冤冤相报何时了 ○ The health and growth of a field depends crucially on the quality of peer reviews 3
- 4. UNIVERSITY of ROCHESTER Outline ● 他山之石 What other experienced researchers have said ○ Rick Szeliski (Facebook) [slides] ○ Jordi Pont-Tuset (Google) [slides] ○ Fatma Güney (Koç University) [slides] ○ Konrad Schindler (ETH Zürich) [slides] ○ Michael Goesele (Facebook) [slides] ● Main Principles ○ 己所不欲， 勿施于人 ○ It takes a community 社区是由每个人组成的 ○ Look for reasons to accept ○ Writing is important ○ The No. 1 goal of top conferences/journals ○ What if you are not familiar with the area? 4
- 5. UNIVERSITY of ROCHESTER 社区是由每个人组成的 ● Protect, not pollute the playing ground ○ Everything could come back and haunt you ● Be responsive ○ Late/missing reviews hurt everyone involved (and you too) ● Be responsible ○ Senior researchers should review their students’ reviews! ● Do not force authors to cite your own (irrelevant) papers ○ Do suggest missing important references (不夹带私货，但要堵绝恶意不引用） ○ Do point out plagiarism / double-submission ● Do not force authors to cite (unpublished) arxiv papers ○ Not peer reviewed ○ May contain errors and false information (some could be malicious!) 5
- 6. UNIVERSITY of ROCHESTER 己所不欲， 勿施于人 ● Submit the reviews on time ○ Aren't you tired of waiting forever for the first round decisions from TPAMI/TMM/etc.? ● Spend enough time understanding the paper ○ Aren’t you mad at the reviewers who clearly did not read your paper adequately? ● Be constructive ○ One good deed deserves another ● Do not use small excuses to reject ○ Look for reasons to accept ● Do not simply wave the two big sticks ○ “No/little novelty” ○ “Poor/insufficient experiments” ● Do not force a gazillion of new experiments ● Do read other reviews and rebuttal (be open-minded) 6
- 7. UNIVERSITY of ROCHESTER Look for Reasons to Accept ● What is novelty? ○ Novel problem (most exciting) or generalizable method or deep insight ○ Novel solution to an existing problem ○ Is “combination of known components” by definition trivial? ● How important is higher performance? 不以百分点论英雄 ○ Statistically significant? sufficient data size? ○ Pure brute-force? ● A balancing act ○ Highly novel/exciting but with clear flaws (has enough been demonstrated?) ○ Flawless but not exciting (what could the community learn from such work?) ● Terminology game ○ Does the same terminology mean the same thing? ○ Does different terminologies mean something new? 7
- 11. UNIVERSITY of ROCHESTER Writing is important! ● Paper is written for others (HELP REVIEWERS HELP YOU!) ○ Clarity ○ Proper English ○ Neat tables, illustrative figures, and effective examples ○ Is “paper gestalt” justified? (can you train yourself?) ■ Deep Paper Gestalt claims it can safely reject 50% of the bad papers while wrongly reject only 0.4% of the good papers ● Innocent mistakes ○ Apparent errors (symbols, equations, annotations, references) ● Intentional/malicious acts ○ Ignore important/well-known/recent/classic references (reinventing the wheel) ○ Smoke and mirror (for GPU abuse) ● Could poor writing be the reason to reject ○ Did the authors respect the reviewers and their time, and act professional? 11
- 12. UNIVERSITY of ROCHESTER What is the Goal of Top Conferences/Journals? ● Making sure good work is published ○ Very important，做伯乐 ● Making sure bad work is not published ○ No so important, 水货会自然流失 ● Test of time ○ 时间， 才是检验good work的唯一标准 ■ Oral vs. Poster (our own study shows…) ■ Best papers (test-of-time award papers?) ● Different standards for different tiers of venues? ○ Yes 12
- 13. UNIVERSITY of ROCHESTER What if you are not familiar with the area? ● Keep an open mind ○ It’s a learning opportunity (and get inspired accidentally) ● Do you best to follow the paper ○ Check the main references if necessary ● Do not get affected by personal taste (e.g., data-driven vs model-driven) ○ Consider if the paper is worth seeing by the community ● You are doing an important job in this case！ ○ 当局者迷， 旁观者清 (conflict of interest in either way) ○ You are the fairest of all (esp. given the wide use of paper matching algorithms) 13
- 14. UNIVERSITY of ROCHESTER The Review Flow Diagram 14
- 15. UNIVERSITY of ROCHESTER Let’s Make Reviews Great Again! Thank you! 15