Reliable Feedback from Clicking Behavior in Adaptive WWW Search

A central goal of information retrieval is the design of functions that rank documents according to their relevance to a query. In this talk, we present an approach to automatically learning such ranking functions. We show that clicking behavior, which unlike hyperlink structure reflects the entire user population, can provide abundant and accurate training data for this learning task.

To establish the relationship between clicking behavior and the relevance of a page, we conducted an eye-tracking study. The study shows that a particular interpretation of clickthrough data provides reliable feedback. In particular, clicks accurately indicate relative feedback of the kind “for query Q, document A should be ranked higher than document B”.

For this type of relative training data, we propose a Support Vector algorithm, called a Ranking SVM. Experiments show that the method can effectively adapt the retrieval function of a search engine to a particular group of users and to a particular document collection. For a focused group of users, the learned retrieval function outperformed Google in terms of retrieval quality after training on only a few hundred queries.