Selective Weak Supervision for Neural Information Retrieval

This paper democratizes neural information retrieval to scenarios where large scale relevance training signals are not available. We revisit the classic IR intuition that anchor-document relation approximates query-document relevance and propose a reinforcement weak supervision selection method, ReInfoSelect, which learns to select anchor-document pairs that best train neural ranking models, guided by only a handful of human relevance labels. ReInfoSelect uses the NDCG on the target relevance benchmark as the reward and learns to classify whether each anchor-document pair should be used as a training signal (action). It iterates through anchor-document pairs and converges when the neural ranker’s performance peaks on target relevance benchmarks. Our experiments on ClueWeb09-B and Robust04 demonstrate the necessity and effectiveness of ReInfoSelect in leveraging anchor data as weak supervision. On these TREC benchmarks, the neural rankers trained with our ReInfoSelect significantly outperform feature-based learning to rank and match the training effectiveness of Bing User Clicks, while ReInfoSelect only uses publicly available anchor data. Our human evaluation confirms that ReInfoSelect effectively leverages the reward from neural rankers to select anchors that are more similar to search queries and linked documents that are more relevant to the anchor.