How (and why) not to rank academics

The recently-launched Microsoft Academic Search, a product of Microsoft Research Asia, has made a bit of a splash as a potential competitor to Google Scholar. Although its coverage does not seem as detailed as Google Scholar quite yet, MS Academic Search has a number of additional features, such as author and conference pages, publication activity graphs, and the like. (It also has a really unwieldly, eight-syllable name; let me abbreviate to MSAS.)

Amongst its features, MSAS provides rankings of "Top Authors" within various fields of computer science. This is, needless to say, not something to be done lightly: academia is built around status and reputation, and to rank academics is to make a statement about their status, which a disclaimer buried in the explanatory notes does little to diminish. And in this case, MSAS should have thought bit more carefully before they decided to publish these rankings.

In MSAS, authors are ranked by the number of in-domain citations they receive within a given time period. This is a dubious enough metric in itself. But even bearing the questionable methodology in mind, MSAS comes up with some slightly surprising top-ten entries in their list of top information retrieval authors of all time , and some even more surprising ones in their ranking of top authors in the last five years.

There are, I suspect, a number of problems with how MSAS is calculating citation counts, but a quick browse through the cited-paper lists of some highly-ranked authors shows that their most egregious aberration within the field of information retrieval is that they are including TREC overview papers in their citation counts.

Now, TREC overview papers attract a lot of citations, because whenever anyone uses the test collection developed at a TREC task, they cite the corresponding overview paper. But TREC is not a peer-reviewed conference. And the overview papers in particular are not research publications at all; they are, rather, a summary of organizational information, participant descriptions, and result statistics. Track organizers put a lot of effort into the tracks, and deserve recognition for this effort in other ways; but counting citations to their overview papers is like ascribing conference citations to the person who wrote the preface to the proceedings.

To be fair, Google Scholar also includes TREC papers in their list of academic publications. But then Google Scholar does not attempt to rank academics. This may simply be laziness on Google's part, but it might also be a recognition that this is tricky ground to be stepping on. Providing a slight specialization of a search interface is one thing; taking it upon yourself to summarize and rank the publication career of researchers is another. If you're going to do the latter, you had better pay some attention to doing it correctly.

MSAS has some interesting and useful features. But, as it stands, the author ranking is not one of them. It should be taken down until they get their data and methodology right.

10 Responses to “How (and why) not to rank academics”

  1. Irrelevant says:

    I imagine academics are quite capable of assessing the merits and limitations of this (or any) rank. Microsoft's effort has merit per se.

  2. william says:

    It took me a good half hour of poking around to figure out why MSAS was giving the rankings it was. How many people are going to make that effort? And I have the domain experience to understand why counting TREC citations is invalid. How many people have this expertise? By publishing these rankings, MS is implicitly giving their credit to them. When academics start writing "ranked 4th in Information Retrieval by MSAS" in their promotion applications, how many promotion committee members are going to take the time or have the expertise to investigate and (correctly) discount those claims?

    Microsoft Academic Search overall has merit, but the rankings don't.

  3. required says:

    layman's question - if a TREC paper has no value, why would people cite it? if very few people cite it, then why do we care?

    Hope to see some more explanations.


  4. william says:

    TREC participant reports are very rarely cited, with a small number of exceptions, such as the BM25 report from (I think) TREC 3. Overview papers are widely cited so that there is a reference for a data set that is being used. This is somewhat similar to the practice of citing a paper that describes a tool you're using in your research; for instance, the Terrier team gets a lot of cites for a SIGIR workshop paper describing the Terrier retrieval system by people who use that retrieval system. Such cites perform three chief purposes. First, they point the reader to a location where they can get more information on a dataset or tool. Second, they serve as an acknowledgment and sort of "citation-tip" to the people who developed the dataset or tool. And third, it's simply good form to have references in academic papers. What such cites do not do, though, in the great majority of cases, is refer to the research contribution of the paper being cited.

    There is merit in counting such citations; they reward groups that make resource contributions to the research community. But resource contributions are not the same as research contributions. It is therefore misleading to describe researchers as "top authors" in a research field because of citations to description documents for resources they've released. Where, then, to draw the line? Well, any line is going to be somewhat arbitrary; but the line traditionally draw in the research community is at peer-reviewed venues. And TREC is not.

  5. required says:

    Thanks for the explanation, it seems some people are "tool makers" - they produce TREC and other reports; some people are "trail blazers" - they push the state of the art with geniune research ideas, published in peer-reviewed venues; and yet there are others, just playing the game of citation.

    hopefully there are more metrics to reveal the true contribution of scholars.

  6. fabseb60 says:

    The argument according to which publications that are not peer reviewed (such as the TREC proceedings) should not be considered in citation counting is IMHO a bit weak (for a completely opposite position -- that I do not subscribe to -- see

    The very notion of citation counting is based on the fact that the number of incoming citations of a given work may be taken, by and large, to measure the impact that this work has had on research. Many research BOOKS have had a large impact on research, but books are not peer reviewed according to the usual standards; so, should a book by author X not be considered when measuring the impact of author X on science and research? My feeling is that if a paper which has not been peer reviewed is frequently quoted, in general this means that it is an important paper; if this were not an important paper, why would many papers, many of them published in top conferences and journals, go to the trouble of quoting it? My feeling is that, if the distinction between what is peer reviewed and what is not should be taken into account in citation counting at all, then it should concern the quoting text, and not the quoted text: citations outgoing from a peer reviewed source are counted, those that are outgoing from a non-peer-reviewed one are not.

    Concerning the author rankings that MSAS displays on its page, I agree that they are clumsy and that they certainly have little agreement with the consideration that authors enjoy in the scientific community. If I had to “rank” authors in a given discipline by citation counting means, I would #at the very least#

    (a) not rank them according to the quotations they have received #from within the discipline#; this penalizes authors who are cross-disciplinary in nature, and who fertilize other disciplines with the findings from their own discipline; the citations that an author receives from the entire scientific community should be counted;

    (b) I would certainly not rank them according to the #raw number# of quotations they have received; h-index and g-index are certainly better indicators.

    (c) I would apply normalizations; everyone working in IR knows that raw numbers lead nowhere without appropriate normalizations. For instance, a much better alternative to the h-index is the h_m index, that normalizes by the number of co-authors of a given paper. And normalization by the number of outgoing citations, a la PageRank, should also be used, so that a citation from a paper that contains 250 citations counts less that one outgoing from a paper that has 25.

  7. Nonsense says:

    I think that any ranking of academics will have winners and losers. The poster and the previous commenter seem to imply that the MSR rankings are not representative, perhaps because they are both not listed high enough in these rankings. In fact both rankings are extremely representative and reflect the extent to which researchers are contributing to the field of IR. The truth of matter is that even using their own proposed approach to rank authors, both the poster and the previous commenter will still not feature high enough in such rankings.

    So what's more important or useful to research, a peer-reviewed paper with no citation whatsoever, that nobody clearly cared to read or endorse (like several of the poster's papers), or a non peer-reviewed paper that has over 300 citations? The answer is obvious, still the poster seems to suggest that we should discard papers that are not peer-reviewed. The poster also implies that only peer-reviewed papers are "research publications". This is laughable. The PageRank paper is an obvious counter-example (as well as the corresponding true story that it was (ridiculously) rejected from a major CS conference). I'm sure that the poster will happily exchange all his peer-reviewed papers for the PageRank one. The PageRank paper has had much more impact and influence on the research community than any IR paper published in top peer-reviewed conferences. I'm sure that the poster will also happily exchange all his "research contributions" for the creation of a useful tool such as Google. The latter has much more impact on research than any other peer-reviewed "research publication". It is also de facto a research outcome. In fact, Google pretty much made the IR community completely irrelevant!

    So again, what's more important to the advance of research, a "resource" that facilitates research, or a peer-reviewed paper with no or very little citations? What gives the poster the right or confidence to claim that what people cite in their hundreds is less valuable or research-driven than the papers published in peer-reviewed venues? What about books (textbooks, research book, etc.), which are the main, if not sometimes the only outcome for which several 'big' names in IR are known: e.g. Salton, Van Rijbergen, Baeza-Yates, etc. I guess that these authors are probably the kind of people the poster would like to see listed first? Still, if we apply his nonsense reasoning by analogy, then these books, which were not peer reviewed, are generally speaking outdated should not be considered to be a "research publication" (after all, they are basically a resource). However, like the TREC overview papers, they are certainly valuable and useful resources for IR research. Unless the poster is advocating that Salton, the father of IR, should also not feature in the top-author ranking?

    Any ranking has flaws. It is funny that the poster seems to claim that he knows how the MSR Academic search engine ranks authors. It is a ludicrous, non-scientific and non-backed up claim (apart from cherry picking misleading, incorrect, biased, incomplete and anecdotal examples). In fact, the top-ten entries in both linked to lists are clearly very active, well-known and well regarded IR researchers, with a significant track record (not only of "Overview" technical reports, but also peer-reviewed papers published in highly reputable conferences and journals). Perhaps, MSR Academic search is simply rewarding real and active researchers, those who are building useful resources and tools for the IR community (like Google), over the non-active and useless ones. I think this is a rather good thing.

  8. Itman says:

    Hi William,
    The top 10-50 do not look unreasonable to me. Whom do you think are we missing there?

  9. william says:

    Notable researchers are not entirely missing, but the relative ranking is a little surprising, if one interprets it as a ranking by "impact of research ideas". That said, after discussing the matter with other researchers, and reading the comments here, I'm less unhappy with the ranking as a measure of "impact of contribution to research".

  10. xin zou says:

    i'm a member of the MAS team, I learned a lot from the post and the discussion.

    @fabseb60 -

    your proposal a) was already implemented in recent release of MAS.

    we're considering using h-index and providing other sorting methods as well.

    we release every 3 months, please check back often and let us know your comments.

    Microsoft Academic Search Team

Leave a Reply