Conferences vs. Journals: The Hidden Assumptions

Some conversations I’ve been having over the past year led me to a deeper exploration of the issue of conferences vs. journals in Computer Science. The debate, so far, seems to be missing a few critical observations regarding scientific journals and our own ACM, and therefore it is somewhat incomplete. This essay lays out my thoughts on it.

Warning: No one except academics cares about this!

I’ll start with the historical context. CS is this oddball academic discipline where people publish their most important work in conferences instead of in journals. For several years now, CS faculty have been fighting an uphill battle with promotion committees regarding the workings of our field. Back in 1999, the Computing Research Association (CRA) put out a “best practices memo” called Evaluating Computer Scientists and Engineers for Promotion and Tenure written by three heavyweight computer scientists. The starting point of that document is this:

[…] Relying on journal publications as the sole demonstration of scholarly achievement, especially counting such publications to determine whether they exceed a prescribed threshold, ignores significant evidence of accomplishment in computer science and engineering. For example, conference publication is preferred in the field, and computational artifacts —software, chips, etc. —are a tangible means of conveying ideas and insight. Obligating faculty to be evaluated by this traditional standard handicaps their careers, and indirectly harms the field.

This memo gave ammunition to many CS departments over the past several years, at least in the US. In our School of ICS, for example, tenure and promotion cases are often attached with a memo written by the Chair or the Dean conveying essentially the message quoted above, and many letter-writers reinforce this message in their letters.

In spite of this ammunition, the battle has continued to be an uphill one, because, well, we’re the only ones doing this odd thing. About four years ago, Moshe Vardi brought the issue into the public arena in his CACM letter called Conferences vs. Journals in Computing Research, where he wonders if CS is driving on the wrong side of the road:

[…] the prevailing academic standard of “publish” is “publish in archival journals.” Why are we the only discipline driving on the conference side of the “publication road?”

His letter exposes the Achilles Heel of the CS publication process:

My concern is our system has compromised one of the cornerstones of scientific publication—peer review. Some call computing-research conferences “refereed conferences,” but we all know this is just an attempt to mollify promotion and tenure committees. The reviewing process performed by program committees is done under extreme time and workload pressures, and it does not rise to the level of careful refereeing. There is some expectation that conference papers will be followed up by journal papers, where careful refereeing will ultimately take place. In truth, only a small fraction of conference papers are followed up by journal papers.

He’s right, sort of, and we all know this, of course. In the top CS conferences, reviewers are assigned something like 15 to 28 papers to review in about 2 months. I’ve heard about reviewers who basically suspend their jobs for 5 weeks in order to read the papers and produce in-depth reviews. Most reviewers, however, are sane people who don’t, or can’t afford to, do this. They apply all sorts of heuristics like spending more time with the best papers in their pile and less time with the ones that aren’t so good. Or they give some papers out to colleagues and students who they know are in a better position to make in-depth reviews. Or, plain and simply, they just spend an hour or so with each paper and produce the best review they can produce within that time. Given the enormous task we ask ourselves to do, at no pay, we can be unhappy about this practice, but we can’t really condemn it!

The limited time that reviewers allocate to papers shows up in the quality of their reviews. It’s not unusual to see reviews consisting of one short paragraph. At that point, these aren’t reviews anymore, they’re just opinions on whether the paper should or should not be accepted; the review is missing.

But Vardi’s observation doesn’t tell the complete story. Journals often reject papers off the bat without even sending them to review. Top journals do a significant percentage of desk rejects. This is equivalent to 1-paragraph reviews. Many papers submitted to conferences aren’t mature enough to be given a thorough review, and that’s ok. The conference review process is more democratic than the journal review one, in the sense that all papers in scope of the conference are given a chance, and it’s up to the reviewers to decide whether the papers are worth their time or not.

There is no question, however, that when good reviews are provided, their integration into the papers is extremely beneficial for everyone, even if the authors need to do some more work. And this is where journals really have an edge over conferences. In most of our conferences, so far, we produce reviews, but their proper integration into the papers may or may not happen, and no one is there to check, because papers are accepted immediately.

So the debate of monsters vs. zombies continues.

Recently, parts of the CS community has started to come up with interesting hybrids. Grudin, Mark and Riedl described them well in a CACM Viewpoint called Conference-Journal Hybrids that I encourage everyone to read. They boil the hybrids down to three:

• Journal acceptance precedes conference presentation. In this category: PVLDB, TACO-HiPEAC and a few other conference-journal pairs, including TOPLAS-PLDI, that are starting to experiment with this model.
  
• Shepherded conference papers become journal articles. In this category: SIGGRAPH and Infovis and their journal counterparts ACM TOG and IEEE TVCG.
• Conferences without a journal affiliation that incorporate a revision cycle. In this category: AOSD, CSCW, and now also OOPSLA, of which I’m the PC Chair this year.

This is all very interesting, but here is a fundamental question: does the CS community really know how our colleagues’ journals work? After all, we don’t usually publish in them, and have, for the most part, stayed away from our own journals for 50 years…

In the quest to answer this question, I searched the web, talked to people, and have come across what seems to be the core of the CS community’s understanding of journals, something that is prominently stated in Grudin et al’s Viewpoint:

Journals encourage more revision and are less deadline driven

Where does this idea come from?

While it’s true that many journals engage in very long, and many, review cycles — the CS journals being in that category –, others don’t. Many excellent journals in other fields have very short cycles and at most 2 rounds of reviews. Here are some well known examples: Science, PNAS and Physics Review D. Not all journals are like that, but many are. There isn’t just one single journal model, there are many! (and I don’t even want to touch on the fascinating history of peer review, which puts a lot of what we take for granted into question)

So, again, where does our monochromatic CS journal model come from?

The ACM Publications Board — the powerful board that controls all journals and transactions published by the ACM — felt the need to write down a policy concerning the publication of conference proceedings in ACM journals. I suspect, but I’m not sure, that this explicit policy has something to do with several conference leaders approaching them recently in order to just publish their conference proceedings as journals. And there it is, in that policy, the core of the community’s understanding of journals:

ACM journals and transactions are designed to publish research results which are the gold standard for the profession, i.e., they are of high novelty and interest, technically sound, and well presented. Achieving this level of quality requires a review process that provides the time necessary for careful review by acknowledged experts in the field. In particular, this means selection of reviewers from the widest possible pool, and open- ended review cycles that ensure the most sound and polished result. Such a standard is largely incompatible with conference review procedures which are sharply constrained by deadline.

So this is where the crux of the issue lives! The ACM, which pretty much institutes academic computer science, believes that its journals should have open-ended review cycles — that means, prolonged ping-pong between authors and reviewers that can last for a year or more. It also believes that the reviewer pool should be as wide as possible, as opposed to being the 25-30 people as the conferences’ program committees. But this last point is largely mute, because, as stated above, conference reviewers have naturally found sane ways of coping with reviewing overload which includes requesting reviews from others. This makes Program Committee members be essentially equivalent to journals’ Associate Editors (some conferences already switched their terminology, e.g. CHI and now ICSE). So the real difference here is the open-ended nature of the reviewing/revision process: the ACM thinks it’s critical for quality; many excellent journals disagree.

In fact, our best conferences aren’t that far from those excellent, short-cycle journals. This is particularly true for the conferences that are following the 3rd hybrid model described by Grudin et al. such as CSCW and OOPSLA, with the extra check at the end. The main difference is that we do it in batch, whereas those journals do it continuously. Let me explain: we recruit the community to perform actions en-masse under strict deadlines — there’s deadlines for submission, so our conferences get 200-400 papers all at once; there’s deadlines for reviews, so we get 600-1200 reviews all at once; then we meet and discuss the most promising papers; the entire program committee / associate editors participate in the final decisions, not just the PC Chair / Editor In Chief; we select a subset of papers to go to the second round all together; the authors all have the same amount of time to review their papers and resubmit on a strict deadline; we review a second time and submit the final assessment on a strict deadline; and we finally send the final notification at the same time for the papers in the second round. This is a communal effort, everyone is in it together!

So, yes, it is different from the continuous pipeline that our colleagues’ journals use, but is that bad? I like deadlines. Deadlines make my world go around! If it weren’t for deadlines I would procrastinate like a snail. I have a feeling that the CS community, in general, likes deadlines. The only problem with the batch model is that we have just one cycle per year, and that slows things down. But wait. We already have only one cycle per year, because our conferences happen once a year, so we can’t really point to that as a disadvantage of this model over our current conferences. The way we cope with that is by having many conferences over the year to which we can submit our work. But, granted, more than one cycle per year would be better than just one! — we just need to find a way of making that work in a way that doesn’t loose the community engagement that comes with the batch process.

Coda

The very same reasons that made conferences the loci of research work in our field also made journals in our field seriously lag behind in many respects. The community is not used to submitting original research work to journals; most of the journals in our field publish extended versions of conference papers, and it is not unusual that reviewers expect that as a condition for acceptance of papers in journals. If we disturb the prestige of our conferences, it is unclear where the good, original work in this field will be submitted to.

Many of our conferences have brilliant histories, having been the forum for some of the most important innovations over the last several decades. Devaluing that by transforming them in conferences-as-other-fields-have-it would waste all that history. OOPSLA, for example, has been naturally following the evolution and maturation of the software field; it has evolved as we, the community, have evolved. While I see the value of journals in the pursuit of solid knowledge that is backed up by evidence, I don’t necessarily agree that we should simply abandon the conferences in our field that have brought us to where we are today and start publishing in the existing open-ended review journals. Instead, I am inclined to accept that the next step in this evolution is for our best conferences to get more serious about argumentation and validation of the published work by adding a second round of reviews, bringing them to level with many excellent journals in other fields. Once that is in place, we can then have the discussion of  what to call them: conference proceedings or journals?

From what I understand of academic history, the Humanities, Arts and Social Sciences have traditionally had open-ended review cycles. The “hard sciences” not so much, they want speed (this is an oversimplification). But all this is messy and interesting: the field of scientific publication is wide and mixed, it’s not monochromatic. The ACM, so far, has decided that it wants only one type of journal, and does not accept new journal proposals unless they account for open-ended review cycles. The CS community, on the other hand, by and large prefers short cycles with strict deadlines.

So here’s the departing thought: how much of this debate of journals vs. conferences in CS exists because of ACM’s current open-ended review cycles policy for its journals?