From msmit@cs.ualberta.ca Thu Nov 8 01:47:27 2007 Date: Thu, 8 Nov 2007 01:47:27 -0700 (MST) From: Michael Smit <msmit@cs.ualberta.ca> To: c603-announce@ugrad.cs.ualberta.ca Subject: p1 reviews & rebuttal marks (comments) The marking for paper 1 reviews and rebuttals is now complete. The marks will soon be uploaded to docsdb; you will be able to view them there. Another email will be sent out when the marks are available. However, before you start worrying about marks, I wanted to send you this email giving some general comments. Assume you got a 1/10, and that you should read every one of my comments carefully to figure out where you went wrong. :) You know how I think what you learn in a class is more important than the marks, so let's get through this part quickly so I can talk about some lessons to learn. :) For the most part, the reviews and rebuttals were well done; I am very pleased. The class average for the reviews was 8/10. As you can see in the marking guide, 8/10 means "all 3 reviews were good or excellent"; that this was the class average is definitely good news. I think in some ways reviewing paper 1 is harder than reviewing a normal conference paper, so give yourselves a pat on the back. The class average for the rebuttals was 4/5, which again means "good" according to the marking guide. Again, well done. My marking methodology was semi-blind, in that I did not know who wrote the review/rebuttal until after I had written down a mark. Each review was compared to the class as a whole, and to the other reviews for that paper. In cases of disagreement among the reviewers, I reviewed the paper myself and compared my own observations to those of the reviewers. In case of disagreements between the reviewer and the author, I checked the paper and formed my own opinion. It was not necessary to agree with me exactly; a pretty generous threshold was allowed. To interpret your mark, I recommend relying on the marking guides linked from the paper 1 web page, http://www.cs.ualberta.ca/~c603/paper1.shtml *In addition* to the guides, to the sample review, and to the class reading, you should consider the following comments: Reviews ------- - The guiding principle here is "be useful to the program committee and to the author". - When I said "Take as much space as you need", that was not a license to write pages and pages. What that means is be succinct - communicate your message in as few words as possible. The number one conference in my field, ICSE, receives approximately 650 submissions each year. Each submission is assigned three reviewers; each reviewer writes 20-25 reviews. Writing 3-4 pages is simply not feasible for the reviewers or for the program committee - brevity is an important skill to learn. - On the other hand, the more details you can include the more it helps the authors, so I don't want to discourage that. In general, you were lightly penalized if I found you used 50 words when 5 would have done the job, but this time most of you got away with it. For paper 2, the review form will include two sections: 1) comments for program committee, 2) advice / recommendations for author. The idea is to encourage splitting the major strengths/weaknesses that affected your decision from the minor points that you include as a favour to help the authors with their revisions. The advice part is not a strict requirement, and will not be carefully read and marked like the first part will be. It's just there to allow you space to offer additional comments to the author, so you can be helpful without worrying about length or brevity. - Organization is very important. Most PC members will be skimming your review; it should be easy to look at a review and say "Hmmm, 2/10 for related work. Where do they talk about related work?" How you organize it is up to you, I won't dictate that. But set it up in such a way it can be read and understood quickly. Some useful organizations your peers used were: a) by section of the paper, b) by the organization of the sample preview, c) by major comments, minor comments, etc. These are just three approaches; anything readable is fine by me. - Related to organization is how important the summary is. I can't stress that enough. Some of you simply said "And then the authors proposed some research." That tells me nothing. A better sentence is "The authors then proposed to parallelize the algorithm and decrease running time by a factor of 100." - Also related to helping the PC skim the report is to state your overall opinion, preferably right up front, but anywhere will do. - You should be explaining your scores. For example, too many people gave 'topic relevance' a score below 10 without ever mentioning why. (For this course, topic relevance should mean "did the author try convince you the topic is relevant") If you said "related work" was only a 6, tell us why it was a 6. If it was a 10, give me a quick explanation of why it was so great. You don't have to do this directly; I don't need to see "I gave a 6 for related work because ____". But if I read your review, it should be clear to me why you gave that mark. For the most part, people did well at this. - As promised, reviewers who were too lenient were penalized. Reviewers who were too harsh were also penalized. I accepted a relatively broad range of scores for the same paper, provided the scores were well justified. If you missed a very important point that other reviewers caught, that was not good (on the other hand, if you caught important points that the other reviewers did not, that was good). In general, I found that reviewers caught a variety of different things, meant everyone was catching important points that others did not and it all kind of evened out. - Proof-reading, spell-checking... these things matter. The most common mistake was using the word "summery", which means "like summer". The word you mean is "summary". Although a spell checker won't catch that problem, in general there is no excuse for not spell-checking your work. - A common comment was "section X was too long", or "section Y is too short". By now, you should know I'm not a fan of describing things by length. What you want to say "Section X includes details that are not needed to make the point, such as ____ ", or "Section Y needs more details, for example ____" - Don't make assumptions. Don't guess at the gender of the author, don't guess at how well they know english, don't assume anything. - For the purposes of THIS course, the word "plagiarism" should not appear in your review if you have not emailed me or Nelson about it. We want to know about it ASAP so we can deal with it. You were not penalized for this; this is just something to note for paper 2. In normal papers, you can stop reviewing once you find plagiarism and just reject the paper, and we'd rather you not do that for the course. Rebuttals --------- - If you didn't actually make a change, don't say you did - say that you will. - If you want to disagree with something all three of your reviewers said, make sure beyond any doubt you are right. - again, spelling, proof-reading, etc. - I was serious about the word limit - I only did a word count if the review looked long, and accepted anything up to about 225 words. Most of you stayed with in that. Speaking of the limit... - Well done on being succinct. Obviously you can't respond to everything in 200 words, so I looked at how many comments you managed to address, whether you correctly identified the most important points to address, and how well you addressed them. For example, "Rev 2 is wrong about my research proposal" is not at all helpful. If a reviewer said "The research proposal is not original", but instead of addressing that you decided to debate the grammar mistake they pointed out on page 3, paragraph two, that would be bad. If you spent 200 words debating a single point and ignored other important points entirely, again - bad news. In general, the 200 word limit was a tough challenge and you guys rose to it. - Remember, it is important to agree to make at least some changes proposed by the reviewers. - You should never, ever, ever, ever make snide remarks about the talent of the reviewers. "Maybe if the reviewer paid attention..." is a bad phrase. "I thank the reviewers for their comments" is a good phrase (though not required, as there are better things to say using 7 words). - Opening sentences like "You should accept this paper because..." are not necessary. You are the author of the paper. We know you want it accepted. I hope that you find these comments helpful, and as always I stand ready for your questions. Cheers, -Mike