This example/introduction uses the organization described in Reading Assignment #2. Other organizations are acceptable if you are more familiar with them, but this organization is very common.
You should read Task of the Referee. Seriously, do it now. This sample review requires that you read and understand that paper first.
The section headings (in bold) are for readability and organization in this sample and should not appear in your review. You should separate each section with a paragraph break. Note also that the examples in each section are not related.
[Brief statement of recommendation and reasoning]
Two examples for reviews of different tone:- I am pleased to recommend the acceptance of this paper. The author provides an ambitious but practical research proposal that would be a useful contribution to the field. The related work section is thorough and provides useful comparisons to the paper under study.
- I must recommend that this paper be rejected. The summary of the paper being studied is incomplete, and the related work section does not provide adequate comparisons with the papers summarized. The experimental section is a single paragraph long and clearly inadequate.
(Notice that in example 1, I refer to the author, but in example 2 I refer to the paper. It is a matter of preference; in general, it is easier to write a negative review using impersonal references to a paper. Whichever you choose, remember you are critiquing the PAPER, but that a PERSON wrote it. You'll find that the examples below include a mix of comments that refer to the author and to the paper.)
[BRIEF summary; 1-5 sentences]
(We all know what every paper 1 will have in common. A good summary - for Paper 1, at least - will describe where it deviates from the norm. In other reviews, including Paper 2, you should summarize the paper more generally.)
The paper studied in this paper describes a method of translating natural language into XML. After describing the current state of the art, the author proposes to extend the algorithm with a simple user interface that allows the user to annotate automatically generated XML. These annotations are used as input to a slightly modified second pass of the algorithm. An unrelated experimental section compares the BayesNet, NaiveBayes, ZeroR, and StackingC classifiers when run on a variety of data sets.
[Evaluate the validity and significance of the research goal.]
(For the purposes of Paper 1, this will be an evaluation of the research proposal and its potential. This type of evaluation is difficult to generalize. In this example, the reviewer merged comments on the research proposal with the rest of his or her comments in the next section; this is acceptable for paper one, as long as it is obvious what you are doing.)
[Evaluate the quality of the work (methodology, techniques, accuracy, and presentation).]
(Example from one of last year's reviews (modified). The reviewer voluntarily chose to split his or her review into sections, starting with the major points and then addressing organizational and presentation issues. You may do this if you wish, but other organization is acceptable (e.g., address issues in the order in which they appear in the original paper). This review is also on the 'wordy' side; the same messages could have been communicated with fewer words. This section should provide enough detail to justify your final score, and if your recommendation is to accept, should include the things you would want corrected before publishing it in your journal.)
I like the idea of addressing how to model the changing nature of network data over long periods of time. However, I would like to see a more specific research proposal. What do you plan to do, how would you approach it? Simply identifying a few areas and broadly waving in that direction didn't meet my expectations. Food for thought - network data may change over weeks, months, years, hours, or minutes. The automated backups that run at 3am will have different characteristics than the web browsing activity of 3pm. The check-your-email-in-the-morning at 9am will differ from the CVS-checkins when your employees start working at 10am. And if your network spans time zones, it gets even more complex. I like the initial thought, but don't be afraid to extend it and give some details.
I am disappointed that nothing was said about the data needed for training. Where does one find realistic network data that is completely devoid of malicious activity? Any amount of real-world data will contain port scans, DOS attacks, DOS spray back, DDOS attacks, and more. But data generated artificially in a lab is well-known to be useless, and of course will not represent the real traffic on your network. So the training part of intelligent algorithms is very, very difficult, and I had hoped to hear more about this.
The experimental section is well done, particularly the description of the algorithms and the presentation of the experimental data.
The related works are copied verbatim from the original paper, with 4 additions. 1 is the original paper, and 2 of the remaining three 3 were written by the same person. Since the original paper was published in 2000, the "related work" is out of date. The opening statement of "there hasn't been a significant research done directly related to this field" is incorrect. In the last 6 years, intrusion detection has become immensely popular. Consider some of the following sources: NSPW, IEEE Security, John McHugh and his (former) team at Carnegie Melon, CERT, Symantec white papers, Carlton University security group, etc. Adding only that single 2005 paper is not sufficient.
Figure 1 probably is not necessary. It also does not match the remainder of the paper. Figure 2 is never explicity referenced, and I have no idea what point is being made by it.
Organization
------------
- 1.1 through 1.6 would be better suited to the summary section than to the introduction section
- data sets 1 and 2 are described in separate paragraphs; data sets 3 and 4 are merged into a single sentence.
Presentation
------------
- The document should match the template provided in the specification.
- The references should be properly and consistently formatted, preferably according to the standard set in the specification. For references [2,8,11,12], the claim is that web pages were viewed in early 1999, 7.5 years ago. I doubt the truth of that claim.
- Vague words and phrases, like "very", "many", "to a certain extent", and all similar words should be avoided.
- Figure 2 is illegible at that size.
- A common error in this paper is subject-verb agreement. If the subject of a sentence is singular, present-tense verbs should be "plural". For plural subjects, use "singular" verbs. That is, "He sees the error" versus "they see the error".
- In general, the grammar and spelling of this paper is not at a level appropriate for academic writing.
[Provide an overall recommendation for or against publication.]
At this point, repeat your decision (to accept, reject, etc.) and summarize the motivation for this recommendation. If you wish, you may also summarize the motivation for your overall score. You should also make the strength of your opinion clear (strong, weak, indifferent, etc.). Note that by the end of the report, the reasons for your decision should be very clear - this is only a summary.
