Hmm, good point. I’ll take a step back on this one. The authors could, and probably should, have conducted more rigorous evaluation than the rather anecdotal variant they did (search for “university” on page titles only). SIGIR is not entirely at fault here. Still, in terms of publications, it is an interesting miss.

William

The original paper did not fit into the SIGIR mold in that it lacked experimental validation. Now of course part of the problem is that the existing test collections of the time did not provide the data needed to adequately validate (or falsify) the method.

This is true. However, look at what the authors write in the original Google paper:

However, most of the research on information retrieval systems is on small well controlled homogeneous collections such as collections of scientific papers or news stories on a related topic. Indeed, the primary benchmark for information retrieval, the Text Retrieval Conference [TREC 96], uses a fairly small, well controlled collection for their benchmarks. The “Very Large Corpus” benchmark is only 20GB compared to the 147GB from our crawl of 24 million web pages. Things that work well on TREC often do not produce good results on the web.

In other words, they already had amassed their own large (147 GB), real life (web-crawled) collection. All they need to do to perform experiments is two things: (1) Come up with 50 queries, (2) Evaluate 2 x 10 x 50 = 1000 pages for relevance.

At the time, 50 queries was enough to establish statistical significance at a believable level in the SIGIR community. It wouldn’t have been that hard to come up with 50 queries. Grab 5 friends and everyone could do 10 each. You could do that in half an hour, if not less.

Second, why do I say 2 x 10 page relevance evaluations? Because you have to have a baseline. Run your search algorithm without pagerank for each query, and then run it with pagerank. Look at the top 10 results from each (2 x 10). Do that for all 50 queries. (You might even be able to get away with less than 1000 total judgments if there is significant overlap between the sets of documents returned by both systems.)

In the worst case, and with 5 friends, that’s only 200 pages each that you have to look at. That’s really not a lot of work, overall. It could have been done. Heck, I wrote my own first search engine around that time (in early 1998, for the IR course at UMass, where I was a grad student) and Bruce Croft and Jamie Callan (who was at UMass at the time) had us do exactly what I’m proposing.. evaluate relevance ourselves for the top 10 docs returned from each query. I think I remember doing about 25 queries x 10 = 250 judgments. Not hard. Took me an afternoon.

Then, with judgments for PageRank and a non-PageRank baseline, they can compute things like rank of the first relevant hit, Precision @3, @5, @10, etc. It’s true that they couldn’t have computed recall. But so what? Web engines today still don’t compute recall Precision at the top is what matters to most web searchers.

So they easily could have done all that. Would the experiment have been on a “standardized” test collection? No. Does that matter to the SIGIR community? My experience is no. I see novel ideas all the time for which there are no test collections, and still folks manage to come up with reasonable evaluations. And SIGIR accepts those papers.

Just my $0.02.

It is true that there is some scepticism about the real value of PageRank in the IR academic community. Thanks for the reference to your blog post; I had not been aware of some of the papers cited in it and the comments; I’m not convinced that they demonstrate that PageRank was inferior to other solutions available at the time of its publication, but that’s a topic for another time.

However, PageRank was certainly an interesting and, at the time, novel idea, one that has inspired a lot of other work in the area (such as on HITS). It has prima facie attraction to it. The original paper did not fit into the SIGIR mold in that it lacked experimental validation. Now of course part of the problem is that the existing test collections of the time did not provide the data needed to adequately validate (or falsify) the method. I’ve discussed this with SIGIR luminaries, and their response has been “well, you can always perform some sort of experimental analysis”. But I think this is quite wrong-headed: if a proper experimental validation is not possible, an improper one should not be cobbled together to meet some publication hurdle. And the nature of innovative research is that test collections to validate a new idea do not become publically available until the new idea has already been established.

As it happens, our CIKM paper is on the topic of going through the motions of empirical validation in SIGIR and CIKM papers without the experiments actually meaning anything.

Please see one of my older blog post on this topic:

Is PageRank just good marketing?
http://www.daniel-lemire.com/blog/archives/2007/11/28/is-pagerank-just-good-marketing/

Disclaimer: I teach PageRank to all students in my Information Retrieval course. I’m no PageRank basher. But was it really rejected because of how innovative it was?

Disclaimer 2: I submitted a paper once to SIGIR and, of course, it was rejected. I submitted it to Information Retrieval (the journal) and it was accepted with raving reviews. To this day, I think it was decent work, though I would now rewrite the paper differently. It was certainly an original take on the topic.

http://irgupf.com/2009/03/09/exploration-and-explanation/comment-page-1/#comment-4480

And I suspect a lot of other people have sighted Belkin 1982 as well, given that it was republished in the 1997 “Readings in Information Retrieval” book (Sparck Jones and Willett, eds.)

I’m sure that 1982 contains much the exact same theoretical groundings as 1980, so it’s not like I/we/the community doesn’t understand the ASK hypothesis and are only going on hearsay. But you raise a good point: Why is it that 1980 gets cited so often, rather than the more widely-read 1982? I’ll bet it has something to do with researchers wanting to cite the earliest works, lest they get called out for it.