The Story of a Journal Proposal

This is the story of Jan Vitek and I approaching the ACM Publications Board with a proposal for publishing the OOPSLA papers in a journal. Even though we did this together, the analysis and opinions in this post are mine alone.

TL;DR: if there’s any lesson to be learned from the ACM Publications Board’s policies on scientific publication is this: pay attention to their policies, then do the exact opposite.

[WARNING: this post is irrelevant for all but academics]

Over the past year or so, I have been involved in making the OOPSLA conference become a venue for the publication of papers that are as good as they can be. This was achieved with the introduction of a two-phase review process, where the set of papers selected in the first phase were given concrete editorial feedback for improvement, which was acted upon by the authors and was checked 2 months later by the reviewers. You know, journal like. I am very proud of the results this year: the papers presented people’s work in their full potential. And this added no more time between submission and final acceptance! — 4 months.

Love or hate the work that has been published at OOPSLA this year, it is there in its best Sunday dress, as opposed to covered in rags with missing pieces of cloth. Only time will tell if the community values these papers by reading them and citing them (or not). I’ll keep an eye on the citation metrics! But that’s not the purpose of this post.

In a previous post, I talked about this conundrum of conferences vs. journals in Computer Science. For the longest time in Academic promotion cases, Computer Science department chairs have made the case that, unlike everyone else, Computer Science researchers publish their best work in conferences. It’s a fact. Well, sort of. 10 years ago, it used to be a pervasive practice. Now, certain parts of Computer Science have finally given up the uphill battle, and have started to publish in journals [too]. They don’t do it all in the same manner: machine learning people do truly publish in journals, in the sense that they send their original work to journals; but others (e.g. software engineering and programming languages people, for example) are unwilling to let go of their conferences, so they publish their original work at their best conferences and then patch that work up with 5 or 6 more pages of shinning cloth and bug fixes, and send that to a journal. While a lot of people do this last coat of veneer, [almost] everybody despises doing it. It’s a waste of time and it dilutes citations. If we are going to publish our work, why not publish it in a form that’s as best as it can be in the first place? I tell you why: internally within the community, what matters are the conference publications that describe original contributions and are selected by peers; externally, the journal articles give the necessary veneer for promotions. That’s why people do it this way.

It’s a perverse practice. It wastes people time. It’s a sham. It should stop.

The truth of the matter is that in many parts of Computer Science, the publications selected for conference presentation are, indeed, the original research work. So why not just acknowledge that by (1) making it the best that it can be right there and then, and (2) tagging these collections of papers with the word “journal”? — because that’s exactly what they are, especially when going to 2 phases of review, like OOPSLA and CSCW.

While giving this extra kick of quality to the OOPSLA papers, the OOPSLA Steering Committee gave some of us the blessing to go ahead and try to publish the OOPSLA papers directly in a journal. And hence the real story of this post begins…

From the beginning, I suspected this was not going to end well. I had read the information regarding the creation of new journals within the ACM. Early on, I also came across very strong wording put out by the Pubs Board regarding the publication of conference proceedings in ACM journals. This policy basically says: no way in hell we are going to trust conference program committees to select the right papers for our journals (*SIGGRAPH not included). OK!, I thought. We’ll give it a try anyway, and hope to learn something along the way. And learn we did.

We wrote a short proposal as a means to start the conversation. Read it, if you have time! That proposal is so bluntly at odds with the Pubs Board’s policies that they must have thought we came from Mars! But we didn’t. That was an honest-to-good proposal for correcting the double standards that academics in the ACM community regularly have to resort to: internally valuing the peer-selected publications presented at conferences while externally presenting journal-versions of those articles that add very little or no value to the original conference publication; or getting away with publishing a semi-decent version of their work in a conference and moving on to the next project without bothering to apply veneer.

The conversation with the Pubs Board dragged for several months. First, we were encouraged to talk to the ACM TACO editors, which we did. The ACM Pubs Board likes that model very much. The model is a bit contrived, as it is trying to achieve two conflicting goals at the same time: the speed of conference reviewing and the cover story of ACM journals. I won’t try to explain it here. But the bottom line is this: an existing, conference (HiPEAC) was devoid of editorial decisions. Instead, that responsibility was moved to a journal, TACO, which started in 2004 and was having a hard time getting its feet off the ground (meaning: no submissions). TACO, like any ACM journal, has a fixed, 3-year term, editorial board, headed by the 2 founders of the journal, who, as far as I can tell, are still the heart and soul of the resulting publication 10 years down the line (they are the ones we talked to, not the Editor-in-Chief).

So, here’s my interpretation of the TACO/HiPEAC hybrid: an existing, perfectly healthy conference was devoid of all editorial decisions — basically the Program Chairs have no role in selecting papers and supervising the review process anymore, and there is no Program Committee. Those tasks were moved to a non-starter ACM journal with a fixed editorial board — essentially, a 3-year-term Program Chair called Editor-in-Chief (EiC), a 3-year-term Program Committee called Associate Editors (ACs), and the critically indefinite-term 2 Senior Editors, the 2 founders of the journal, who keep the ball rolling.

Cynical as I may sound, I believe the community around the HiPEAC conference likes this model! I was just left wondering what will happen to TACO when the original founders don’t want to do the pushing anymore… That is, indeed, the real risk of letting outstanding individuals do exceptional things, instead of coming up with a model that scales beyond exceptions!

Our proposal to create a new journal for the OOPSLA papers was almost identical to the TACO model, except for a few details… and a whole different governance attitude: (1) editors would rotate on a yearly basis; (2) editors would be selected by the conferences’ Steering Committees, which would also be part of the editorial board; (3) the journal would not accept ad-hoc submissions, only submissions intended to be presented at the conferences; (4) there would be no dragging of papers for the next year: they would either be accepted in 4 months, or they would be rejected.

These details, however, make the whole difference between accepting the top-down governance imposed by the Pubs Board, where the editorial board is under the control of a very small group of people who don’t necessarily know anything about the topic, and maintaining the bottom-up governance that we have in conferences, where members of the research community who stand out over the years are asked to serve increasingly more important roles within the organization of the conferences and the selection of papers. This model has proven to work very well for the past 40 years — and it works without exceptional individuals!

The conversation with Pubs Board also took us to TOPLAS, but that was a dead-end on arrival, since the current EiC’s vision for TOPLAS is that it publishes “the best work of the year in programming languages“, resulting in… less than 20 papers per year (!). OOPSLA alone publishes 50+ papers per year! Unlike TACO, TOPLAS has been around for a long time, so it has its own history and goals; that was also a strong factor in making a possible merge a non-starter.

OK, so our proposal was eventually shot down, as I suspected it would. But here’s what I found along the way.

The governance surrounding scientific publications is really important, and the top-down governance model used by the Pubs Board and so many other publishers is not only outdated, but it may very well explain the dismal performance of ACM journals in citation-based quality metrics. At least the commercial publishers have a strong incentive to make their journals perform well on those metrics, so chances are they choose editorial boards that can deliver. Not-for-profit publishers like the ACM have no such incentives, so if a journal is poorly cited over many years, no one really cares, and the whole issue is swept under the rug of the tagline/wishful thinking/self delusion “highest quality research in X”.

Here is some interesting data related to citation-based metrics of ACM publications.

From Thomson Web of Science’s Journal Citation Reports (Impact Factor, JCR 2012):

• The high IF of TOG should send a strong signal to the Pubs Board about the effects of publishing the best conferences in ACM journals using the conferences’ governance instead of the top down governance that it favors.
• CACM vs. JACM is an eye opener. Finally, after soooo many years!, the ACM acknowledged that CACM, not JACM, is, indeed, its “flagship” publication.
• TOPLAS (an elite publication) vs. SIGPLAN Notices (a huge patchwork of PL papers) is another interesting comparison: best as they are supposed to be, TOPLAS papers aren’t that much more cited than the entire collection of 300+ papers carried by SIGPLAN Notices.

Not many people like Thomson’s IF. Indeed, the representativeness of Computer Science material on that index is questionable, because, apart from SIGPLAN Notices, JCR doesn’t contain any conference papers. So, let’s look at other bibliometrics.

Google Scholar has this rank for CS/Software publications. Here is a snapshot of what it looks like today:

This ranking has it even worse for ACM journals: they are nowhere to be seen. The top 20 publications are dominated by ACM and IEEE conferences, along with some non-ACM journals.

Here is Microsoft Academic’s ranking of all CS journals. Snapshot below:

There’s CACM topping the list. SIGPLAN Notices (listed as a journal) comes in #10 (!), almost tied with JACM. No other ACM journal makes the top 20. ACM Computing Surveys comes at #19 [Thanks for pointing that out, Jonathan!] No other ACM journal makes the top 20.

Finally, let’s zoom in on Programming Languages in Microsoft Academic:

SIGPLAN Notices comes on top.

In others words, the data from both Google and Microsoft Academic is showing, in different ways, that the large patchwork of papers that are presented in ACM software conferences has higher citation-based rankings than the carefully selected collections published in the ACM journals focusing on software.

And here is where the Guardians of Quality come in and say: “But these rankings are meaningless! That is a popularity contest! Everyone knows that IF and the like is a horrible way to measure research quality! The ACM journals are carefully refereed and the conferences aren’t!” Perhaps, but many people disagree on that refereeing statement (and clearly SIGGRAPH’s papers are refereed as poorly or as well as any other top conference’s papers). In my book, after data like this, that kind of defense sounds like elitism for the sake of elitism, especially given that when impact factors come out, the ACM checks how it’s doing. In  my book, data like this gives a strong hint about how ACM journals are simply doing it wrong — and how the Pubs Board is missing its opportunity to improve the situation by taking a more community-oriented approach to running its publications: taking the best conferences, with their governance unchanged, their hits and misses, and publishing their papers as journals, which is exactly what we proposed to do for OOPSLA. Perhaps, just perhaps, if the ACM does that, we will start seeing ACM journals routinely getting IFs above 3, like TOG.

Community-oriented, bottom-up, vote-with-your-citations-and-your-downloads scientific publications is in the spirit of how other fields are now starting to do it. We’ve been doing it for a while… in conferences.