Subject: bidding deadline & review

Dear 603 students:

This email gets long, but contains important information about the review.
Not knowing something that was communicated to you in an email is no excuse,
so be sure to read it all.

1. Anyone who hasn't submitted a bid by 6pm on Friday, Oct 12 will be
assigned 4 reviews instead of 3.  I'll be picking the 4 for you at 6:05pm.
If you have a problem logging in to record bids, now is the time to mention
it.

2. Due to the odd number of people in the class, it may be necessary for at
least three people to do four reviews instead of three.  If everyone submits
their bid on time, three people will be chosen randomly.

3. Speaking of bidding, if you intentionally picked a friend's paper to
review so that you could give them a better mark, don't do it.  If you know
which papers your friends wrote, identify them as a conflict of interest.
(it's not too late to change your bid).  Your review will count toward 2% of
their mark.  The mark assigned by the TA who catches you doing this is worth
10% of YOUR mark.  Think about it. :)  Beyond just the marks, this
experience will be the most useful to you if the people reviewing your paper
are honest, and if you get to write honest reviews.


4. Speaking of marking.  I normally much prefer to talk about LEARNING as
opposed to however marks are assigned.  I've always viewed what you learn as
more important than what mark you get, but I do understand that marks are
still important at places like NSERC.  In response to some concerns, I want
to talk for a minute about the difference between the review you write, and
the mark and grade the author of the paper receives.

You will be asked to rate the paper in a variety of categories (the full
list was given to you in the paper 1 spec).  You will be asked to give an
overall score (1-10), and to place the paper in one of 5 categories - strong
accept, weak accept, neutral, weak reject, strong reject.

When writing reviews, all of the resources we have shown you encourage you
to take a stand - to give excellent papers a 10, and weak papers a 1, and
the other papers somewhere in between.  Giving a paper a "neutral" requires
very strong justification.  It is sometimes necessary and appropriate, but
in general a review that makes no accept/reject recommendation is not very
useful to the program committee.  Remember that your peer reviews are a
small component of the mark the author will receive; the main point is for
YOU to get practice reviewing and providing useful feedback on technical
papers.

Some people worry about the implication the overall score will have on the
author's mark, and so only give out scores in the 7-10 range - I've seen
(weakly) rejected papers with scores of 7 in past years.  Don't do this. The
writing TA will mark papers on a scale that is normal for grad students, but
your peer reviews should use the full 1-10 range.

Why?  Well, you might think that a score in the 1-5 range out of a possible
10 points is a "failure"; 10-50%.  Try not to think of it in this way.
Remember from one of the first UTS sessions - the numeric mark you receive
is not a "grade".  There is no set conversion from numbers to letter grades.
The numbers are simply to allow us to create a partially ordered list of
students based on performance; the grades are then assigned on some type of
curve.  Nelson tells me that last year's grading curve matched exactly the
average grade curve across graduate courses at the University of Alberta.

In summary, we want a fair and honest peer review of the papers.  You should
follow the advice of the various reviewing guides and differentiate between
strong papers and weak papers (without being unduly harsh or lenient).  It
requires judgment, which is why we have this first practice paper.  Also,
rememember that just as important as the actual score is the justification
you provide for that score.

You should note that one component of our evaluation of your reviews will be
how appropriate the score you give is based on a) your comments, b) the
quality of the paper, and c) the scores given by the other reviewers. There
is no "right answer", but we do want to ensure you write a solid, useful
review.

That was a long discussion, but I hope it helps you better understand this
aspect of the reviewing process.  If you have any questions, now is the time
to ask - email me or see me in class next week.


5. Nelson mentioned this in class, but I wanted to emphasize it - it might
be hard, but do not take the peer reviews personally if you get a lower
score than you expected.  This is the academic process.  If you get a low
score, you will also get a list of things to improve in the future. Trust
me, this will be more valuable to your academic career than an extra few
marks on the peer review would ever be.  The key here - as in most aspects
of the course - is to learn whatever you can from the experience.


6. I've jumped ahead a bit here to talk about the reviews, and your marks,
and your grades - in the next few days, I will send out another email with
more reviewing advice, including a sample review or two.


I hope it took you less time to read that than it took me to write it. :)
Have a great night!

Cheers,

-Mike