Longer or shorter conference papers?

The past few days have seen the review deadline for IJCAI papers, SIGIR papers, and SIGIR posters, of which I had 11 (mostly farmed out), 10, and 4 (all done myself), respectively. IJCAI had a six-page limit, whereas this year, SIGIR expanded their papers from the usual eight pages to ten.

I don't know what the reason for SIGIR's expanded page limit was, but I don't think it was a good idea. It's arguably kind to authors, but it's unkind to readers, and rather cruel to reviewers. For the papers I reviewed, the extra pages seemed to add padding to weak papers and remove concision from strong ones. If conferences provide anything worthwhile as a publication venue, it is early exposure for novel and interesting ideas; and there are few interesting ideas that need more than six double-column pages to convey.

One argument I have heard put forward for increasing the page limit on conference papers is that the shift to soft-copy distribution removes the physical constraint on proceedings size. The cost of paper, however, is hardly the main concern. A far rarer resource is attention: the volume of literature that a researcher must keep up with is constantly growing, and our time to read it is generally shrinking. SIGIR itself has expanded from 28 papers two decades ago, to 47 papers ten years back, to 87 papers last year. In the face of literary inflation, the importance of concision increases. Conference organizers should be tightening, not loosening, their allowances.

6 Responses to “Longer or shorter conference papers?”

  1. paul says:

    After crawling out from my own pile I couldn't agree more; at least the original 14-paper load for SIGIR was quickly fixed.

    I'm not sure that a ten-page paper is any kinder to authors, except in that it lets us think less about what our message really is (and that's not kind, not really). Given that each paper, even the worst, is read by at least four people before being decided upon, there are twice as many reviewers suffering as there are authors rejoicing. Who exactly wins here?

    But maybe I'm just grumpy from review-induced lack of sleep.

  2. Itman says:

    I would ask whether 8 pages is enough to deliver the main idea? Isn't it hard to judge and read the paper if it does not have enough details?

  3. FD says:

    I think I heard a complaint that 8 pages was too short at the last SIGIR Business meetings. The argument was that authors cannot fit results in that amount of space. The several grunts of agreement must have been enough to run the 10 page experiment (?) this year. Other venues (e.g. WWW, WSDM, CHI) have 10 page limits so I'm not clear on the problem. It's just another way that papers can be bad. I'd be surprised if the few papers that are accepted do not use that space well.

    Now, it's a different argument when considered in the context of reviewer load. Part of me is sympathetic. On the other hand, improvements to reviewing in general may come at the cost of increased load for those blessed few strong reviewers. I don't accept this argument from anyone who complained about poor reviewing in previous years. We need to step up if we want these things done right.

    Finally, I think there were several tweaks to submitting/reviewing this year. Some may be experimental. Feedback is appreciated. So is patience. The organizers are definitely responding to the (recent?) criticisms with the process.

  4. Ian says:

    The question isn't whether 8 pages is too short, it's whether the paper you're trying to write is in fact an 8-page paper. Think about a 2-page poster. Some ideas really fit that medium well... I think there's an art to the 2-page paper. 8 pages is the same.

    Of course, the flip side is that there are no decent venues for long-form papers. Journals are glacial, unread, and offer no opportunity for community.

  5. william says:

    Thanks all for your comments.

    I think Ian has it right. Conference submissions should be kept as short as possible. More complex and finished work should go to journal. The IR community should be moving towards a stronger journal culture, not further loading a conference system that is on the verge of collapse.

  6. I think peer review is highly overrated. Even if there were some absolute notion of goodness in a paper, reviewing is so noisy and contentious that I don't see it even approximating such a notion in practice. I think there's also a strong conservative incremental results bias (at least in NLP and machine learning) that isn't helpful in a conference setting aimed at exploring new ideas rather than gatekeeping tenure and promotion decisions.

    One solution would be to change the publishing paradigm for computer science. As a first step, I like the idea of reverting to a more traditional arrangement, where there are

    1. short conference abstracts (say 1 page), high acceptance rates, and

    2. quicker journal reviewing turnaround.

    I benefit from seeing a quick blurb about stuff as it appears and I get readable papers in a timely fashion that are as long as they need to be. I suspect it'd be a huge relief to reviewers; at the very least, it'd spread the load out over the year.

    I really dislike having to learn about things in 6-10 page chunks. In the machine learning and NLP worlds, papers have gotten very dense. I wind up with four papers on the same topic, each of which is so terse that I can't learn about the topic from the papers in the way I could from one 20 page version.

    I favor a more radical approach where everyone publishes everything on something like Arxiv and the whole notion of journals goes away. The main value add of journals to papers is not the gatekeeping, except to the bean counters (and by that, I mean the people who tabulate conference acceptance rates, journal impact ratings, etc. when reading a CV).

    I think the main advantage of reviewing is getting people to read a paper moderately closely, often from different perspectives. What I don't like is that as a reviewer I have to waste so much time on bad papers. I'd rather read papers that I'm interested in and give authors feedback. Or go back into the professoriate and help students on a more extended one-on-one basis.

    I'm putting my time where my mouth is. I try very hard not to review for low-acceptance rate, dense paper conferences any more [I'll make exceptions for friends and colleagues in a pinch]. Nor do I want to submit to them [repeat above qualification]. I won't review for closed access journals or conferences any more, either. 10 years in industry where I couldn't even access papers I reviewed (without paying for them) soured me on that whole ball of IP wax.

Leave a Reply