This post is dedicated to provide detailed guidelines to Area Chairs of CVPR 2012 as to how the process will unfold and what are their responsibilities.
Role of the Area Chairs
The ACs help identify suitable reviewers, assign reviewers to papers, and evaluate the reviews as well as the authors' responses. Our goal is to provide the highest quality reviews to authors and for this we hope to limit the number of papers assigned per AC. This will allow each AC to play a more active role in ensuring higher quality and more timely reviews, and in initiating discussions amongst reviewers (anonymously), and in looking at authors' responses to the reviews via the online rebuttal process. At the AC meeting in February 2012, ACs will work with other ACs (in pairs and then in panels) to decide which papers to accept and will write consolidation reports to explain the decisions. Needless to say this is a very important part of the Academic process of Peer Reviewing and has a direct impact on the quality of work in Computer Vision. All of the Area Chairs for CVPR 2012 are committed to doing this task in a timely, professional and ethical manner.
Best Practices of Being an Area Chair
We just want to remind all ACs of some of the basic (and very obvious) practices of being an AC.You have a strong influence on a decision on a paper. Take it seriously and be fair. DO NOT talk to any other AC about papers assigned to you until you are told about who is on your panel and who is your buddy. There may be several other ACs conflicted with the paper for which you are the Area Chair. We will provide details on how to do this as we get closer to the AC meeting. The Program Co-Chairs did not submit any papers, so they are not in direct conflict with any papers and they know all authors of all papers and also all reviewers/ACs. They will NOT be involved with any decisions of papers from institutions with which they are in conflict. You can send them a question about any paper and if one of them is conflicted, another one will respond to you without including the conflicted PC Chair. DO NOT talk to any other AC about your own paper (the paper you are an author on) or a paper you have some conflict with, during this whole process, unless that AC and you are already collaborators and both of you have an already defined conflict. Do not lobby for your paper, hand out copies to other ACs and/or show results or discuss your paper during the review process. Be professional and willing to listen to other reviewers and ACs. Do not give in to undue influence from anyone. Do remember, it is not fair to dismiss any review without looking at other reviews and reading the paper yourself to make an evaluation. Outright rejecting a review is not acceptable without a clearly articulated supporting argument. Area chairs who wish to make a decision that is not clearly supported by the reviews will have to present the paper to their panel at the area chairs meeting for discussion and confirmation of the decision; any such decisions that do not have the consensus support of the panel will be later reviewed by the program chairs who will make the final decision. Senior members of the committee should guide and advise the younger and first-time members. First-time members, please feel free to ask for help as needed.
Consolidation Reports
Area Chair Consolidation Reports are the MOST CRUCIAL aspect of the review process of CVPR. This is where the Area Chair justifies his/her recommendation to accept/reject a paper. The Program Chairs will read all Consolidation Reports and we have committed ourselves to ensure that these reports are representative of a good quality review process, which we are guaranteeing to the Vision community resting on the Area Chair's shoulders. These reports should highlight why the decision was reached. If all reviewers agree on a paper, then this consolidation report can be simple, but feel free to encourage authors and provide constructive feedback. If there is even a slight disagreement on the reviews, it is your job to clarify why and how the disagreement was resolved. Again, just saying "reviewer 1 was wrong" without any reasons is not acceptable. Explain why. For the most divergent reviews, explain the process you went through to reach a decision and if you and others read the paper, try to add some details from your and others reviews with the consolidation reports. Overruling all three reviewers to make a decision on a paper is not acceptable. If you strongly feel that the paper deserves that, make sure to discuss it in detail with your AC Buddy and also in your AC Panel. In such cases, (a) you should read the paper completely, (b) provide a detailed consolidation report, and (c) solicit confirmation of the decision from your panel. The program chairs will review and confirm or reverse any such decisions that do not have the consensus support of the panel.
During the reviewing phase, please periodically log in to the CMT site and monitor the review progress.
Phase 1 - Assigning Reviewers
The Program Chairs are going to assign papers to you based on (a) papers from authors not in conflict with you (see conflict of interest in the Reviewer Instructions), (b) subject areas you chose when you registered on the review site, and (c) subject areas, the authors chose, when they registered the paper. As can be imagined, this will NOT be perfect as some authors have differing takes on some subject areas then we do and also, sometimes the Program Chairs have to do some load-balancing. Area Chairs should look through all the papers assigned to them and ensure that they are (1) not conflicted and (2) are knowledgeable in the subject area.For each paper you are assigned you must initially select 5 or more candidate reviewers during the reviewer assignment step. After all ACs have completed this step, the CMT system will then optimize over all papers and attempt select 3 from each candidate list, subject to necessary constraints on reviewer load. If it is not able to assign 3 from your list, it will assign fewer than 3, and you will be required to iterate again suggesting additional reviewers. CMT will not assign a reviewer to a paper that you have not suggested. But several iterations may be necessary if you only suggest the minimal number of candidate reviewers, and those will occur in a relatively short cycle (24 or 48 hours). If you plan to only assign the minimal number of reviewers, please expect to be available on short notice to add a few more to the papers which are not satisfied on the first round. If you will be unavailable for several days, it will be wise to assign 10 or more candidate reviewers. Please assign as many candidate reviewers as possible to each paper, as long as you are comfortable getting any subset of 3 actually assigned. You must assign at least 5, and the more you assign the greater the chances you will not be required to suggest additional reviewer names for that paper. While CMT will not automatically do so, the program chairs may personally assign reviewers based on preference keyword match or any other scheme necessary to papers of ACs who do not respond within a day to email requests for additional candidate reviewer suggestions for papers that are not satisfied on the initial assignment optimization.Here are some suggestions on how to proceed. It can take anywhere from 4-6 minutes a paper do this, if you look at the paper and other info to help make a decision. Considering a max load of 35 papers, you can see that this may take some time. See Reviewer Instructions for details of how to use the system to assign reviewers.You have been chosen as an Area Chair because of your specific expertise in certain areas of Vision and also your general expertise in Computer Vision. When you look at a paper, you may just come up with a name or three of ideal reviewers. Consider these people as the reviewers for this, but do also consider things like, (a) are they current in the field and (b) will they do a good job in a timely manner. Look at the paper, especially the Introduction, Related Work, and Citations at the end. See who they refer to and whose work they are building on. Often the cited people would be ideal reviewers. Do remember that the system will give you some recommendations, based on simple heuristics by matching subject areas. This will provide a good starting point, but will not be perfect as some folks are not that good at choosing keywords (authors and reviewers). Usually, we suggest that you choose a reviewer you know and trust. But the vision community is large (how else would we get 1450+ submissions). If the system recommends someone you do not know, look for their webpage or look for their papers on on ACM or IEEE digital libraries or Google Scholar and the like. Again, spend good time doing this.
Phase 2 - Check Incoming Reviews
After you have ranked reviewers, the Program Chairs, with the review system's help will distribute papers to the reviewers. During this phase, we recommend that you log into the system to check if any reviews are being entered. Do remember that the reviewers DO NOT KNOW your identity, so it is best if you do not contact them and/or identify yourself. This is yet another form of "anonymity" in the review process, as we don't want the reviewers to be unduly influenced by anyone. If reviews are showing up, look at them and see if (a) the reviewer has added sufficient detail and (b) has used appropriate tone in the review, and most importantly (c) provided a knowledgeable review. Contact the reviewers via the review system to get them to clarify issues from the review. Doing this during the process may prevent you from doing it all at the deadline when reviews are due.After the review deadline has passed, check if all reviews are in. If not, send them a note and nicely DEMAND that they finish the reviews soon. The system will also send reminders to them.In "Area Chair" console, select "Consolidation Reports". You should be able to see all the reviewers' names and the review status right next to each. For this incarnation of CMT, we just highlight cases where reviews have not been submitted (in red). For each paper, you can: * View all the reviews ("View All"). * Grade the reviews ("Rate Reviewers"). * Email the reviewers anonymously via CMT ("Email Reviewers"). You can select all the assigned reviewers for that paper (default), or just those not done with reviewing, or specific reviewer(s). For the last item, you will be able to edit the recipient list. Remember: Don't identify yourself or the other reviewers in the email! You can also choose to bulk email all reviewers or all reviewers who have not completed their reviews (for all the papers in your stack). See below the title "Consolidation Reports" ("Email Reviewer"). Mouse over the "Email Reviewer" box and you will see the two options (mentioned earlier) from which to choose. You can view all reviews for all papers in a single webpage. Just select "View All Reviews" in the line right below the title "Consolidation Reports". Important note on emailing via CMT: Since any email sent on your behalf is anonymously sent through CMT, the communication is one way only. Please tell your email recipients to not respond to the email, because there is no automatic routing of email to you; responses go directly to the CMT admin people. The CMT folks are handling a number of conferences at any given time and will not have the resources to forward such emails. So, be very specific about what you would like the reviewer(s) to do. It is the area chair's responsibility to contact and urge the reviewers to finish in a timely manner.
Phase 3 - Discussion Phase
Right after the review deadline, please look through ALL the reviews. Please let us know who have yet to finish their reviews, and we will contact them directly. If any review appears unreasonable (e.g., one-liner comments or overly harsh), please contact the reviewer and ask for clarification. Remember that communication is only one-way, so be specific about what you need from the reviewer. Check again in one or two days to see if there is any update to that review. Since we will not be keeping track of your emails (sorry!), please do the book-keeping yourself (best to make a note of what you've done).
3.1. When to Initiate a Discussion
We urge you to start a discussion any time you feel that the overall scores and/or the comments are insufficient to allow you to make a decision (e.g., all "Borderline"s). At the very least, initiate a discussion if the difference between the maximum and minimum scores is greater than 2 levels.
3.2. During the Discussion
You play the role of a moderator in the discussion. While it is ideal to reach a consensus, don't feel obligated to force one - the discussions are mainly to allow you to make more informed decisions. It is ok if the reviewers end up not changing their overall scores as long as you've gained a better understanding of how they arrived at their scores. Stick to the facts, and do not influence the outcome of the discussion by imposing your views on the paper. However, if you disagree with the majority view, say so, provide compelling reasons, and solicit feedback from the reviewers. If the discussion looks like it is getting out of hand (e.g., when a reviewer makes inappropriate remarks or a battle of words erupts), please contact us (cvpr2012-pc-chairs@googlegroups.com) immediately. Don't forget to let us know the paper number. For cases of "Borderline"s, if the discussion appears to not head towards any decision, we suggest you ask the question "If you had to choose now, would you accept or reject this paper?"
3.3. Anonymous Discussion Feature
This feature can be accessed from the "Detailed Meta-Reviews and Discussions" page:You must do the following steps to enable the reviewers to participate in the discussion. Emailing the reviewers via CMT is NOT the right step to initiate the discussion. To initiate a discussion for a specific paper, first enable the discussion and then select "View/Post Message" under the "Discussion" column for that paper. In the "Paper Discussion" page, click on "Start A New Topic" to initiate the discussion (unless a reviewer took the initiative of starting one, in which case you reply; see next bullet item). We suggest that your first post starts with a very brief summary of the reviews, a request for them to look at the other reviews for details, followed by specific things you want the reviewers to address. Once a discussion has been initiated, click on "Reply" on the far right to continue posting on the same discussion thread. Please sign your posts as "Area Chair", and identify the reviewers by the review number.Never identify yourself or other reviewers by name. Anytime a post is made (either by the area chair or a reviewer), the area chair and reviewers will receive an email notification from CMT with the subject that looks like "CVPR2012: New reviewer discussion posted for Paper ID XXX". There is a link in the email you can use to join the discussion (after logging in, you will be routed directly to the discussion page). Alternatively, you can just log in to CMT as usual. After you've posted, DO NOT REFRESH PAGE (e.g., by hitting F5)! This will generate another post with the exact same message! In the discussion, you (as AC) can see the identities of the posters. However, each reviewer will NOT know the identities of the other posters. Once the author rebuttal period is over, you and the reviewers will be able to see the author rebuttal (but not before). We will be enabling discussions for a week past the rebuttal deadline, just in case reviewers have any reactions to the author rebuttals. Because of the frank nature of the discussions, the authors will not see them at any time. The consolidation reports will be visible to authors only after the AC meeting, when decisions are made known. Reviewer instructions for the discussion feature are given here. Note that reviewers can revise their reviews until the review deadline.
Phase 4 - Rebuttal Period
During the author rebuttal period reviewers are no longer allowed to modify their reviews.
Grading Reviews
ACs are expected to grade each review. This information will be used in part to help us identify exceptional reviewers for recognition during the main CVPR12 conference. Some guidelines:"Highly Relevant": Reviewer is on the ball on practically everything. The score is well justified, references are given to support claims that references are deficient and/or work is not original, and the reviewer appears very knowledgeable on the topic. You feel you can totally rely on this assessment. "Sufficient": The review quality is mixed, but overall reliable. There are parts of the review that appear perfectly reasonable, but there are other parts that seem a bit deficient in some ways. You feel you are satisfied with the review and the reviewer has given you an evaluation you can use to make judgement on the paper. "Below Average": Reviewer does not justify scores well, makes overly general unhelpful statements, states incorrect or misleading claims, provides very terse or irrelevant remarks, or appears to be overly biased. Basically, you feel you can ignore most of this review.
Phase 5. Review Author Feedback and Discuss with Reviewers
You should be able to see the Author Rebuttal/Feedback to the reviews.You can start a discussion using the discussion board (see Phase 3: Discussion Phase) with the reviewers of the papers, especially if the rebuttal is responding to a specific review. While, the reviews cannot be changed, you can take the input from the reviewers at this phase to help in your proposed decision and the consolidation report that you have drafted (or will!). Make sure to be clear about how the rebuttal was taken into account in the consolidation report. Remember, all consolidation reports are considered a draft and all decisions are just proposed and will be finalized at the AC Meeting.
To See Consolidation Scores on the AC Website
To allow you to see the reviewer scores on the consolidation page, in the Area Chair "Consolidation Report" page, you can customize the page by selecting "Customize The View"->"Edit Review Columns". Select Q2 ("Overall Rating") and click on "OK" to see the updated table.
Phase 6 - Prepare for AC Meeting Discuss with AC Buddies, Propose Decisions on Paper
Area Chair Meeting will occur February 25-26, 2012.
Conflicts with Papers Assigned to Area Chairs
I recognize a piece of work (ie I know the authors) should I recuse myself from the paper? They are not collaborators and I have no financial ties or other connections with them. My feeling is that I can be impartial and handle the review but it is your call. If you think you can do the review impartially and fairly, and there is no direct conflict, we think you should review this paper, as we feel you are best qualified for it. Thanks for letting us know. I have been assigned a paper that might be pretty close to things I am working on at present (I don't know for sure since I didn't want to look at the full paper). I would like to decline handling that paper, if possible, on grounds of potential conflict of interest. Thanks for letting us know. Yes, this does qualify as a conflict of interest so we will assign to some other Area Chair. I see a paper that I know is authored by a recent collaborator of mine (collaborated in last 3 years). I am reasonably confident I can still render a fair and an impartial review. But wanted to let you know. This appears to be a conflict. While we do trust you to be fair and impartial, it is best to have someone else take over this paper. Thanks for letting us know. We will swap it. I have a paper assigned to me that appears to be very similar to one that I have just co-authored for CVPR 2010. I can be still objective in review of this paper, irrespective of my submission. Thanks for telling us. We feel you are the expert in the area and trust you will do a good job. If you feel you cannot, then we will re-assign to another area chair.
Assigning Reviewers
What should I do with a paper with the names of the authors explicitly mentioned? Papers with authors names explicitly violate the anonymity requirement of CVPR and will be rejected. Let the Program Chairs know the paper id. Thanks. What should I do with a paper that did not exactly follow the required format? If the paper is still 2 column style CVPR format (missing the line numbers, paper id on every page, etc.) and not longer then 8 pages (font not too small, margins ok, etc.), then it is fine to assign reviewers to (We don't want to be too rigid). What about papers that over 8 pages long? Overall, our hard-line policy is (as the authors were warned!) that the paper will be rejected. If you see a paper with just a citation or 1-2 lines on page 9, then we are letting it go. But if the paper is grossly over onto page 9, then it is a "administrative" reject. Is there a race to assign reviewers? If I do not assign reviewers by Dec 4, will I lose all the good reviewers for papers assigned to me? No, there is NO race. You are choosing (recommending) 5+ reviewers for each paper and on Dec 14, we will optimize the assignments based on (a) reviewer load, (b) your choice/rank of a reviewer, and (c) subject area match to reviewer. We hope this way, good reviewers per your choice will be assigned the paper for which you are an AC. However, we do not want to see all of you waiting until Dec 8 and after to assign papers. I have scanned a paper and it seems very weak and will be rejected, what should I do? Unfortunately, if the paper has been submitted and does not meet a direct criteria for an administrative reject (non-anonymous, too long, dual-submission, etc.) then the paper has to be reviewed. Give the paper a fair chance and have it reviewed. We agree that sometimes this is a waste of reviewing resources, but we feel the authors deserve the best from us.
Assigning New Reviewers
When should I decide to assign a new reviewer that is not in the database of reviewers? After you scan a paper to see what is the content, some obvious names of reviewers will come to your mind. Look for them in the reviewer lists. All reviewers should be visible to you (there are over 700 of them, so it is a bit slow). They are sorted by "subject area relevance" (matching their chosen subject areas to authors' subject areas for their paper). If you do not see the person you wish to review the paper, then consider adding them as an additional reviewer. Consider however that (a) this person may already have been asked to review for CVPR12 and may have declined and (b) this person may not have the time to review this time. So DO NOT assume that the new reviewer will agree to do this review. Add an extra reviewer as a back-up for this one.
Area Chairs Meeting When and where is the Area Chairs' Meeting? AC Meeting will be held on Feb 25-26. All AC are expected to arrive on Feb 24 (Fri), 2012, and be there until the session ends on the evening of Feb 26 (Sun), which will then be followed by a dinner. All ACs are invited to stay an extra day, Feb 27 (Mon), 2012 to participate in an informal workshop on computer vision and related topics. When should we make reservations for the AC meeting? Soon you will be getting instructions from us about how to plan for your travel and how you will be reimbursed.
|