Bad Reviews — Part I

Yesterday, a Facebook friend posted a link to a great talk by Jeffrey Naughton about the depressing state of affairs in CS research communities. I won’t do justice to that talk by trying to summarize it; follow the link and see it for yourself. His slides reminded me of a few review war stories I’ve been through — luckily not that many (for the record, a very small minority), but enough to make me very mad at a system that allows these bad reviews to reach authors with complete impunity for the reviewers. Two of them stand out in my memory. I’ll start with the most recent one.

Here is a verbatim review I received on a paper I submitted to the Working Conference on the Future of Software Engineering Research at FSE 2010. As the name says, this wasn’t a technical conference; it was, I assume (as I ended up not attending), about the future of Software Engineering research, meant, among other things, to inform research funding agencies on matters of what it is that the community values.

This is not so much a position paper as a shriek of pain and exultation by someone who perhaps has recently emerged from the cocoon of being a junior faculty member and realized that academic software engineering research is often divorced from software engineering practice. It contains some good points about experimentation and NSF funding. It contains opinions generally unsupported by evidence and often questionable (example: “Great software … is rarely produced by large organized teams with deliverables and a pre-existing maser plan.”). It seems to be motivated by the idea that software engineering research is focused on producing innovative new software. In fact, engineering is rarely mentioned, although science is occasionally invoked. The primary point seems to be that NSF should encourage open source software development.

In a future post, I will publish the paper in question, as I believe quite strongly on what I wrote there and I don’t think I’ll submit it anywhere else. But this post is about bad reviews — this one in particular. This is a bad review, not because it expresses a negative opinion about the paper, but because of how the reviewer chose to do it, and the points s/he decided to focus on. Here is the rebuttal I wrote back to the Program Committee chair:

This is a highly discriminatory statement (rank discrimination) from a seemingly senior colleague. Sounds exactly like the kind of condescending, personable, biased comments one often gets at software engineering conferences — and rarely at serious scientific venues.

It also shows the reviewer’s matter-of-factedly acceptance of this sad state of research in software engineering — a divorce from practice that doesn’t even seem to be worth discussing, because everyone knows about it (“the poor little thing is coming out of her cocoon!”), so we just keep doing our thing and hope that no one (e.g. the rest of NSF) notices how irrelevant we are.

Maybe I misunderstood the goal, but if this is to be an NSF-sponsored  workshop for establishing the next 10 years of research funding in software engineering welcoming thought-provoking positions given in essay-like papers (that’s what I wanted to attend), and not a research workshop in itself for publishing more-of-the-same technical papers (not interested), then maybe there is some ground to the pervasively-spread idea among colleagues that software engineering research is nothing but pseudo-scientific snake-oil that is completely disconnected from the reality of both science and software engineering practice: when reviewers can’t distinguish between an event for brainstorming/discussing difficult issues affecting the community and events for presenting technical work, there’s no hope.

I don’t know if this rebuttal ever made it to the anonymous reviewer’s Inbox. Probably not. The PC Chair and I continued to debate the matter for a few days, until it dried out — that’s what PC Chairs are supposed to do, buffer complaints from disgruntled authors of rejected papers. As he said, “However, it was not my role to prevail on senior members of the SE community to edit their reviews.” I understand this position, but I happen to disagree with it. The sad thing is that this reviewer will likely continue to produce bad reviews, unaware that there’s something very wrong with them — because nobody calls him/her out on it.

So, for the person who wrote it and for every reviewer who feels tempted to succumb to feelings of superiority, condescension and sloppiness under the cloak of anonymity — here is a concrete example of a bad review. This tone is unacceptable, please mind what you write, and do take a look at Naughton’s talk.