The recently-launched Microsoft Academic Search, a product of Microsoft Research Asia, has made a bit of a splash as a potential competitor to Google Scholar. Although its coverage does not seem as detailed as Google Scholar quite yet, MS Academic Search has a number of additional features, such as author and conference pages, publication activity graphs, and the like. (It also has a really unwieldly, eight-syllable name; let me abbreviate to MSAS.)

Amongst its features, MSAS provides rankings of “Top Authors” within various fields of computer science. This is, needless to say, not something to be done lightly: academia is built around status and reputation, and to rank academics is to make a statement about their status, which a disclaimer buried in the explanatory notes does little to diminish. And in this case, MSAS should have thought bit more carefully before they decided to publish these rankings.

In MSAS, authors are ranked by the number of in-domain citations they receive within a given time period. This is a dubious enough metric in itself. But even bearing the questionable methodology in mind, MSAS comes up with some slightly surprising top-ten entries in their list of top information retrieval authors of all time , and some even more surprising ones in their ranking of top authors in the last five years.

There are, I suspect, a number of problems with how MSAS is calculating citation counts, but a quick browse through the cited-paper lists of some highly-ranked authors shows that their most egregious aberration within the field of information retrieval is that they are including TREC overview papers in their citation counts.

Now, TREC overview papers attract a lot of citations, because whenever anyone uses the test collection developed at a TREC task, they cite the corresponding overview paper. But TREC is not a peer-reviewed conference. And the overview papers in particular are not research publications at all; they are, rather, a summary of organizational information, participant descriptions, and result statistics. Track organizers put a lot of effort into the tracks, and deserve recognition for this effort in other ways; but counting citations to their overview papers is like ascribing conference citations to the person who wrote the preface to the proceedings.

To be fair, Google Scholar also includes TREC papers in their list of academic publications. But then Google Scholar does not attempt to rank academics. This may simply be laziness on Google’s part, but it might also be a recognition that this is tricky ground to be stepping on. Providing a slight specialization of a search interface is one thing; taking it upon yourself to summarize and rank the publication career of researchers is another. If you’re going to do the latter, you had better pay some attention to doing it correctly.

MSAS has some interesting and useful features. But, as it stands, the author ranking is not one of them. It should be taken down until they get their data and methodology right.

This year was my first at NTCIR. It’s a reasonably sized community, drawn largely but not exclusively from East Asia, engaged in an interesting and diverse range of tasks. It is, therefore, a good opportunity to meet people in this geographic and scientific area. There are also some interesting cultural differences. The audience was very reluctant to ask questions of speakers, and once Charlie Clarke and Ian Soboroff left (on the last day), it was noticeably quieter.

The first day of NTCIR was given over to EVIA (the international workshop for evaluating information access), for which I was co-chair. There were a number of interesting papers, not least from Charlie Clarke and Ian Soboroff, and (repeatedly) from Tetsuya Sakai. We’ve informally renamed it the “Tetsuya Sakai Memorial Colloquium”. The proceedings for EVIA are also available online.

The organization of NTCIR is changing from this year, with a stronger focus upon community-led initiatives. The new Evaluation Co-Chairs are Hideo Joho, of the University of Tsukuba, and the famous Tetsuya Sakai, of Microsoft Research Asia. The Call for NTCIR-9 Task Proposals has been released. If you’re interested in proposing a task for the next iteration of NTCIR, concluding in December 2011, please grab the Call, and contact the organizers. And once the tasks have been decided and announced, please consider taking part in one of the efforts. I recommend NTCIR as an approachable, interesting, and energetic community to get involved with.
]]> http://blog.codalism.com/?feed=rss2&p=1211 0 http://blog.codalism.com/?p=1195 http://blog.codalism.com/?p=1195#comments Mon, 05 Jul 2010 11:58:14 +0000 william http://blog.codalism.com/?p=1195
I’ve just finished my CIKM 2010 reviewing assignments, and have come out of the process with a number of questions:

Is nine papers too many to review?

This year, I was assigned nine papers to review. Last year it was five. For SIGIR (if my own local cache of reviews is to be believed) it was only two this year (so few!) and eight the year before.

Nine papers is quite a heavy reviewing load. It used to take me the best part of a day to give a responsible review. Now, with experience, I seem to have got it down to four hours or so. Even so, a nine-paper stack takes a few days to get through. Another colleague, himself a responsible reviewer, opted out this year because of the load.

The advantage with having a large number to review, though, is that it gives you a sufficient sample to calibrate your own reviewing. This is particularly important for a conference with as low an acceptance rate as CIKM, whose rate has hovered around 16% in recent years. With nine papers, you have a ration of one accept and one maybe. With only, say, four papers, it is more difficult to know whether you are being overly indulgent or overly stingy.

Is 16% of submissions too few to accept?

There was a very interesting article in the most recent issue of the Communications of the ACM, by JiLin Chen and Joseph A. Konstan, entitled “Conference Paper Selectivity and Impact”. The authors surveyed the ACM Digital Library, comparing the acceptance rates and impact factors of ACM conferences and journals. They found that, on average, journal articles achieve a similar citation rate as papers from conferences with an acceptance rate of 35%. This is interesting in itself, as it implies (as people in CS always claim) that top conferences are more prestigious than journals (although part of the cause might be the ACM’s glacial turnover rate in getting journal submissions to print).

More interestingly for our current discussion, though, was the finding (on slightly questionable methodology, it is true) that the most-cited 10% of submissions at conferences with acceptance rates below 15% receive fewer citations than the most-cited 10% of submissions at conferences with between 15% and 25% acceptance rates. I say that the methodology was slightly questionable because it compared all the papers from a conference with 10% acceptance with, say, the most highly cited half of papers from a conference with 20% acceptance, which elides random effects (more citeable papers from the stricter conference may have been rejected by chance). Nevertheless, the finding gives quantitive form to the often-expressed rule-of-thumb that an acceptance rate over 20% means accepting bad papers, whereas one under 20% means rejecting good ones.

Is it the case (as the authors speculate) that conferences with too-tight acceptance rates are squeezing out innovative papers in favour of formulaic ones?

Are big conferences eating small conferences?

The main reason, of course, why CIKM reviewers have such a heavy workload, and CIKM acceptance rates are so low, is that CIKM receives a huge and swelling number of submissions, as the following table shows:

(By the way, the above table also gives quantitive justification to my qualitative assesment that no-one in their right mind wants to go to a conference in Europe.)

Meanwhile, my anecdotal impression is that smaller conferences are struggling to attract submissions; certainly, this has been the case for a number of smaller conferences I have been involved with recently.

There are many reasons why big conferences could be eating small conferences. One of them might be a growing emphasis upon prestige of venue in assessing publication impact. Whatever the reason, it is hardly a desireable trend. The large conferences are sausage factories for academic publications — CIKM churned through 55 presentations on the opening day last year, ignoring posters. There is also both anecdotal (and, given the previous point, now quantitative) evidence that the large conferences suppress innovation in favour of formula. Plus the sheer scale of the larger conferences make them questionable venues for actually meeting and getting to know other people in your area — unless, that is, you are a senior researcher meeting up with your peers.

In contrast, smaller conferences provide more of the opportunities that conferences properly provide (leaving aside their dubious role as high-status publication venues). The lack of parallel tracks means that there is more unity to the proceedings, and more of an opportunity to hear a range of work. The smaller number of attendees means there is more of an opportunity to get to know other researchers. And the higher acceptance rate means there is more room for novel, off-beat, and experimental ideas.

Why do people publish incrementally at conferences?

One of my pet hates is researchers who publish their work incrementally at conferences: the basic idea as an innovation at one conference, then a series of tweaks to it at successive ones. Two or three of the CIKM submissions I reviewed this year were in this category: one group of authors gave additional experiments for a previous publication; another developed a framework for choosing the parameters for their earlier optimization method. (For the third, double-blind reviewing meant I couldn’t be sure whether they were minimally extending their own work, or someone else’s.) This is a real perversion of the purported point of conferences as venues for the fast publication of new ideas. Once you have your SIGIR publication for your research program, surely that is enough validation? Isn’t the next step to develop it over the next year or so into a fully-fledged journal article, rather than dribbling out new ideas as they occur to you? Such incremental submissions should be stamped out with all vigour. These minor evolutions clutter up high-profile conferences like CIKM and SIGIR, squeezing out newer, fresher work.

Why am I repeatedly given my own submission to review?

This is the second CIKM at which I have been assigned my own submission to review. The first time appeared to be a chance occurrence, but the second began to make me suspicious. How many others does this happen to? Let’s assume for simplicity that people review only in the research sub-field they submit to; that there are as many sub-fields at CIKM as there are sessions; and that submissions are evenly divided amongst sessions. There have been around 900 submissions to CIKM for each of the past two years, and 25 or so sessions per conference. This means 36 submissions per sub-field. If you review 8 submissions, then there is a (35/36) * (34/35) * … * (28/29) = 7/9 chance that you will not randomly be given your own paper to review, or a 2/9 chance that you will. Therefore, the chance of it happening two years in a row is 4/81 (although of course there’s a selection bias here: you only take notice of the two-year sequence that it does happen in). So it’s not that improbable an occurrence. But given it must be happening to over 20% of the program committee each year, surely the allocation algorithm would have been fixed to make sure reviewers weren’t assigned their own paper? Or am I the only one who lets the chairs know when it happens?

Keyword search in databases is not what I originally thought it was. That is, it is not full-text search on individual text fields. Rather, the search is done over the content of whole objects in the database, in the form of (joined) tuples. And since there are many ways that tuples can be joined in a database, one of the immediate questions is how to realize these joins, what level of object granularity to consider for retrieval, and what effect tuple structure has upon the similarity score matching queries to results.

The field of keyword search on databases (or, more generally, on structured data) is still relatively new; the initial work in the area was done in 2002. And one of the aspects of the field that is still being developed is its evaluation resources and methodology. As I describe in my article, different research groups develop their own test sets. A few structured collections are commonly used, such as DBLP. But almost invariably, each research group specifies their own retrieval task, creates their own test queries (or selects them from a log), and performs their own evaluation. Perhaps as a result, a common pattern is that each paper reports almost perfect effectiveness, doubling or tripling that of the previous method — which itself had achieved almost perfect effectiveness, doubling or tripling that of its predecessor.

As I suggest in my article, such evaluation results suggest a lack of objectivity in the evaluation process. In particular, it is likely that if a research group has set out to solve one particular problem in keyword search (for instance, matching of keywords to schema terms), then they are likely to formulate test tasks that illustrate the problem they are trying to solve. Such targetted evaluations have their place, but what is needed in addition is a common set of retrieval tasks, intended to be representative of general retrieval needs, which serves as a neutral benchmark against which each group’s methods can be tested. That is to say, what is needed is a public test collection, the core evaluation tool in IR.
]]> http://blog.codalism.com/?feed=rss2&p=1171 0 http://blog.codalism.com/?p=1166 http://blog.codalism.com/?p=1166#comments Mon, 05 Apr 2010 01:20:47 +0000 william http://blog.codalism.com/?p=1166 I’m currently reading Nonsampling Error in Surveys, by Judith Lessler and William Kalsbeek (Wiley, 1992) for a project I’m working on. It is, for the most part, an informative but necessarily rather dry treatment of statistical questions in survey design and interpretation. The segue at the end of Chapter 10 (p276), however, reads:

If the pancake does not decide to enter the Olympics as a disk, causing the athletes to fall far below records set previously by Russian athletes because of the sticky maple syrup, we will tell you next, dear reader, about measurement variability.

This is even referenced in the index under “pancake(s)”. I wonder whether this was intended as an Easter egg for the reader, or to test the attention of the editor…

On an unrelated topic, the submission deadline for EVIA 2010 (of which I am co-chair) is this coming Wednesday, March 31st. We happen to be using the same paper format as SIGIR. If anyone has work on information retrieval evaluation they’re looking for a venue for, we’d be very happy to consider it for publication at EVIA.

If such stories of dizzying excess captivate you, then you’ll not want to miss the EVIA 2010, the Evaluating Information Access workshop, held at NII in Tokyo, just across the expressway from the Imperial Palace itself! This is the third iteration for EVIA, the first international workshop dedicated to information retrieval evaluation issues. I’ve been invited to join Tetsuya Sakai, the eminence grise of Asian information retrieval research, and Mark Sanderson, as co-chair this year. It’s an interesting time for evaluation research, with increasing interest in HCIR issues and the emergence of crowd-sourced evaluation using Mechanical Turk and the like.

Deadline for submission is March 31st; please see the Call For Papers.