AC Instructions
Phases:
- Dec 1 - Dec 15, 2008: Phase 1: Assigning Reviewers.
- Jan 1 - Jan 29, 2009: Phase 2: Check Incoming reviews, follow up with reviewers.
- Jan 29 - Feb 5, 2009: Phase 3: Initiate discussions with reviewers, Write Draft Consolidation Reports.
- Feb 6 - Feb 13, 2009: Phase 4: Rebuttal Period. Grade Reviewers
- Feb 13 - Feb 17, 2009: Phase 5: Review Author Feedback and Discuss with Reviewers
- Feb 13 - Feb 20, 2009: Phase 6: Prepare for AC Meeting. Discuss with AC buddies, Propose decisions on paper.
- Feb 21 - Fen 22, 2009: Phase 7: AC Meeting
- NEED HELP
Phase 2. Jan 1 to review deadline (5PM PST Jan. 29, 2009, Thursday): Check incoming reviews
- In “Area Chair” console, select “Consolidation Reports”.
- You should be able to see all the reviewers’ names and the review status right next to each. For this incarnation of CMT, we just highlight cases where reviews have not been submitted (in red).
- For each paper, you can:
* View all the reviews (”View All”).
* Grade the reviews (”Rate Reviewers”).
* Email the reviewers anonymously via CMT (”Email Reviewers”). You can select all the assigned reviewers for that paper (default), or just those not done with reviewing, or specific reviewer(s). For the last item, you will be able to edit the recipient list. Remember: Don’t identify yourself or the other reviewers in the email! - You can also choose to bulk email all reviewers or all reviewers who have not completed their reviews (for all the papers in your stack). See below the title “Consolidation Reports” (”Email Reviewer”). Mouse over the “Email Reviewer” box and you will see the two options (mentioned earlier) from which to choose.
- You can view all reviews for all papers in a single webpage. Just select “View All Reviews” in the line right below the title “Consolidation Reports”.
- Important note on emailing via CMT: Since any email sent on your behalf is anonymously sent through CMT, the communication is one way only. Please tell your email recipients to not respond to the email, because there is no automatic routing of email to you; responses go directly to the CMT admin people. The CMT folks are handling a number of conferences at any given time and will not have the resources to forward such emails. So, be very specific about what you would like the reviewer(s) to do.
- It is the area chair’s responsibility to contact and urge the reviewers to finish in a timely manner.
Phase 3. Between review deadline and 5PM PST Feb. 5, 2009, Thursday: Initiate Discussions with Reviewers.
Right after the review deadline, please look through ALL the reviews. Please let us know who have yet to finish their reviews, and we will contact them directly.
If any review appears unreasonable (e.g., one-liner comments or overly harsh), please contact the reviewer and ask for clarification. Remember that communication is only one-way, so be specific about what you need from the reviewer. Check again in one or two days to see if there is any update to that review. Since we will not be keeping track of your emails (sorry!), please do the book-keeping yourself (best to make a note of what you’ve done).
3.1 When to Initiate a Discussion
We urge you to start a discussion any time you feel that the overall scores and/or the comments are insufficient to allow you to make a decision (e.g., all “Borderline”s). At the very least, initiate a discussion if the difference between the maximum and minimum scores is greater than 2 levels.
3.2 During the Discussion
You play the role of a moderator in the discussion. While it is ideal to reach a consensus, don’t feel obligated to force one—the discussions are mainly to allow you to make more informed decisions. It is ok if the reviewers end up not changing their overall scores as long as you’ve gained a better understanding of how they arrived at their scores.
Stick to the facts, and do not influence the outcome of the discussion by imposing your views on the paper. However, if you disagree with the majority view, say so, provide compelling reasons, and solicit feedback from the reviewers.
If the discussion looks like it is getting out of hand (e.g., when a reviewer makes inappropriate remarks or a battle of words erupts), please contact us (cvpr09-pc-chairs@googlegroups.com) immediately. Don’t forget to let us know the paper number.
For cases of “Borderline”s, if the discussion appears to not head towards any decision, we suggest you ask the question “If you had to choose now, would you accept or reject this paper?”
3.3 Anonymous Discussion Feature
This feature can be accessed from the “Consolidation Reports” page:
- You must do the following steps to enable the reviewers to participate in the discussion. Emailing the reviewers via CMT is NOT the right step to initiate the discussion. Reviewers can already see the other reviews for their papers.
- To initiate a discussion for a specific paper, select “View/Post Message” under the “Consolidation Report” column for that paper.
- In the “Paper Discussion” page, click on “Start A New Topic” to initiate the discussion (unless a reviewer took the initiative of starting one, in which case you reply; see next bullet item). We suggest that your first post starts with a very brief summary of the reviews, a request for them to look at the other reviews for details, followed by specific things you want the reviewers to address.
- Once a discussion has been initiated, click on “Reply” on the far right to continue posting on the same discussion thread.
- Please sign your posts as “Area Chair”, and identify the reviewers by the review number. Never identify yourself or other reviewers by name.
- Anytime a post is made (either by the area chair or a reviewer), the area chair and reviewers will receive an email notification from CMT with the subject that looks like “CVPR2009: New reviewer discussion posted for Paper ID XXX”. There is a link in the email you can use to join the discussion (after logging in, you will routed directly to the discussion page). Alternatively, you can just log in to CMT as usual.
- After you’ve posted, DO NOT REFRESH PAGE (e.g., by hitting F5)! This will generate another post with the exact same message!
- In the discussion, you (as AC) can see the identities of the posters. However, each reviewer will NOT know the identities of the other posters.
- NEW: The author rebuttal period is Feb 6, 2009 5pm PST - Feb 13, 2009 5pm PST. Once the author rebuttal period is over, you and the reviewers will be able to see the author rebuttal (but not before). We will be enabling discussions for a week past the rebuttal deadline, just in case reviewers have any reactions to the author rebuttals.
- Because of the frank nature of the discussions, the authors will not see them at any time. The consolidation reports will be visible to authors only after the AC meeting, when decisions are made known.
- Reviewer instructions for the discussion feature are given in Section 7 here. Note that reviewers can revise their reviews, up til the 5PM PST Feb 5 deadline.
Phase 4. Feb. 6, 2009 - Feb. 13, 2009: Rebuttal Period
Remember: Feb. 6-13 is the author rebuttal period. Reviewers are no longer allowed to modify their reviews after 5PM PST Feb. 5. Authors’ reviews will be visible to both reviewers and AC after Feb 13, 5pm EST.
4.1 Grading Reviews
ACs are expected to grade each review. This information will be used in part to help us identify exceptional reviewers for recognition during the main CVPR09 conference in Miami. Some guidelines:
- “Highly Relevant”: Reviewer is on the ball on practically everything. The score is well justified, references are given to support claims that references are deficient and/or work is not original, and the reviewer appears very knowledgeable on the topic. You feel you can totally rely on this assessment.
- “Sufficient”: The review quality is mixed, but overall reliable. There are parts of the review that appear perfectly reasonable, but there are other parts that seem a bit deficient in some ways. You feel you are satisfied with the review and the reviewer has given you an evaluation you can use to make judgement on the paper.
- “Below Average”: Reviewer does not justify scores well, makes overly general unhelpful statements, states incorrect or misleading claims, provides very terse or irrelevant remarks, or appears to be overly biased. Basically, you feel you can ignore most of this review.
Phase 5. Feb 13 - Feb 17, 2009: Review Author Feedback and Discuss with Reviewers
You should be able to see the Author Rebuttal/Feedback to the reviews.You can start a discussion using the discussion board (see Phase 3: Discussion Phase) with the reviewers of the papers, especially if the rebuttal is responding to a specific review. While, the reviews cannot be changed, you can take the input from the reviewers at this phase to help in your proposed decision and the consolidation report that you have drafted (or will!). Make sure to be clear about how the rebuttal was taken into account in the consolidation report.
Remember, all consolidation reports are considered a draft and all decisions are just proposed and will be finalized at the AC Meeting.
To See Consolidates Scores on the AC website:
To allow you to see the reviewer scores on the consolidation page, in the Area Chair “Consolidation Report” page, you can customize the page by selecting “Customize The View”->”Edit Review Columns”. Select Q2 (“Overall Rating”) and click on “OK” to see the updated table.
Phase 6. Feb 13 - Feb 20, 2009: Phase 6: Prepare for AC Meeting. Discuss with AC buddies, Propose decisions on paper.
6.1 Feb. 13-20: Working with Buddies
We have enabled the system so you can now see the papers of your buddies (your buddy is listed in the table at the end of this email). Go to “Consolidation Report” from your console and at top you will see
Show Assigned Papers | Show Buddy Papers | ..
Click on “Show Buddy Papers” and it will show all the papers that your buddy was assigned. There should not be any conflict with any of the papers your buddy has. In the buddy paper list, if you don’t see your buddy AC’s name in the page, you can verify by clicking on any of the “View” link next to “Consolidation Report”.
Now here is a bit about the process.
Our Area Chair Committee is divided into Four Panels and within each panel, each AC is paired up with another AC. This has been done in the past, so it is NOT new for the old-timers in this group. The purpose of this is to have ACs work as a pair to make sure all decisions are made with some discussion with another person and to ensure all issues are dealt with fairly and appropriately. Consider this as an opportunity for someone else to look over all your decisions and look at the reviews and the author rebuttal to ensure all was fair.
Each buddy pair will be required to
- Make Accept/Reject Decision Recommendations for each of their paper
- Of the accepted papers, make a Recommendation for at least 6 papers as an ORAL. These will be discussed in PANELs at the AC Meeting.
- Of the accepted papers, make a Recommendation for at least 1 paper as potential AWARD papers (if more than 1, we need a rank-ordered list). These will be forwarded to the Awards Committee, after some discussion within a panel.
Do note that buddies have ONLY READ access to the buddy papers. The Assigned AC is the only one who can edit consolidation report and make decision recommendation and therefore it is important that both the AC discuss these and the lead AC consolidate.
So here is what to do.
- Look at the reviews/rebuttals for each paper you have and then send a note to your AC buddy asking them to look at the papers where you would appreciate some help or at-least a look over. If you ask a specific question, like “can you see paper #XXX’s review #2 and see if you agree” OR “based on reviews of paper #YYY, I am considering a recommendation of Poster” or “paper #ZZZ is the best paper in my stack, do you agree?”
- Discuss these papers with your buddy and write your consolidation report.
- Make a recommendation of a decision on this paper and when consolidation report is done, “Check box if report is FINAL” and the PC Chairs will be informed on our consoles.
Remember, you should try to do as much of this before the AC meeting and the first part of the AC meeting will also be dedicated to finishing this phase.
Here are the buddy pairs with their EMAILs (in PDF)
6.2 Feb. 13-20: Writing Preliminary Consolidation Reports
Here is what a consolidation reports should contain.
- a summary (or highlights) of what the reviewers found in the paper that they liked/disliked
- a summary of discussions with the reviewers and your buddy (without names of course!) as the discussions influenced your evaluation.
- if a variance in reviews, explain you came to your evaluation.
- if decision is to overrule one of the reviewers, please provide as much details and also discuss with your buddy. The reviewers have looked at the paper longer then you (hopefully) so their evaluation should count and if dismissed, we need to have reasons for it.