{"id":273699,"date":"2017-03-01T14:17:56","date_gmt":"2017-03-01T22:17:56","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=273699"},"modified":"2017-11-28T22:40:04","modified_gmt":"2017-11-29T06:40:04","slug":"letor-learning-rank-information-retrieval","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/letor-learning-rank-information-retrieval\/","title":{"rendered":"LETOR: Learning to Rank for Information Retrieval"},"content":{"rendered":"<div>\n<p>LETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July 2009. Very different from previous versions (V3.0 is an update based on V2.0 and V2.0 is an update based on V1.0), LETOR4.0 is a totally new release. It uses the Gov2 web page collection (~25M pages) and two query sets from <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/ir.cis.udel.edu\/million\/index.html\"> Million Query track<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of TREC 2007 and TREC 2008. We call the two query sets MQ2007 and MQ2008 for short. There are about 1700 queries in MQ2007 with labeled documents and about 800 queries in MQ2008 with labeled documents.<\/p>\n<ul>\n<li>LETOR 3.0 can be cited as<br \/>\nTao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval, Information Retrieval Journal, 2010. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/159.226.40.238\/~junxu\/publications\/IRJ2010_LETOR.pdf\">pdf<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>]<\/li>\n<li>LETOR 4.0 can be cited as<br \/>\nTao Qin and Tie-Yan Liu. Introducing LETOR 4.0 Datasets, arXiv preprint arXiv:1306.2597. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/arxiv.org\/ftp\/arxiv\/papers\/1306\/1306.2597.pdf\" target=\"_self\">pdf<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>]<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<\/div>\n<div><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>LETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-273699","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2009-01-01","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[{"id":0,"name":"LETOR 4.0","content":"[accordion] [panel header=\"Datasets\"]\r\n<div>\r\n\r\nLETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July 2009. Very different from previous versions (V3.0 is an update based on V2.0 and V2.0 is an update based on V1.0), LETOR4.0 is a totally new release. It uses the Gov2 web page collection (~25M pages) and two query sets from <a href=\"http:\/\/ir.cis.udel.edu\/million\/index.html\"> Million Query track<\/a> of TREC 2007 and TREC 2008. We call the two query sets MQ2007 and MQ2008 for short. There are about 1700 queries in MQ2007 with labeled documents and about 800 queries in MQ2008 with labeled documents.\r\n\r\n<\/div>\r\n<div>\r\n<h3>Datasets<\/h3>\r\nLETOR4.0 contains 8 datasets for four ranking settings derived from the two query sets and the Gov2 web page collection. The 5-fold cross validation strategy is adopted and the 5-fold partitions are included in the package. In each fold, there are three subsets for learning: training set, validation set and testing set.\r\n<table style=\"width: 50%\" border=\"1\">\r\n<tbody>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Setting<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Datasets<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Supervised ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Semi-supervised ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-semi<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-semi<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Rank aggregation<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-agg<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-agg<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Listwise ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-list<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-list<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<b>Descriptions<\/b>\r\n<ul>\r\n \t<li>Supervised rankingThere are three versions for each dataset in this setting: NULL, MIN, QueryLevelNorm.\r\n<ul>\r\n \t<li>NULL verion: Since some document may do not contain query terms, we use ``NULL'' to indicate language model features, for which would be minus infinity values. This version of the data cannot be directly be used for learning; the ``NULL'' should be processed first.<\/li>\r\n \t<li>MIN version: Replace the ``NULL'' value in NULL version with the minimal vale of this feature under a same query. This data can be directly used for learning.<\/li>\r\n \t<li>QueryLevelNorm version: Conduct query level normalization based on data in MIN version. This data can be directly used for learning. We further provide 5 fold partitions of this version for cross fold validation.<\/li>\r\n<\/ul>\r\nEach row is a query-document pair. The first column is relevance label of this pair, the second column is query id, the following columns are features, and the end of the row is comment about the pair, including id of the document. The larger the relevance label, the more relevant the query-document pair. A query-document pair is represented by a 46-dimensional feature vector. Here are several example rows from MQ2007 dataset:\r\n\r\n&nbsp;\r\n\r\n=================================\r\n\r\n2 qid:10032 1:0.056537 2:0.000000 3:0.666667 4:1.000000 5:0.067138 ... 45:0.000000 46:0.076923 #docid = GX029-35-5894638 inc = 0.0119881192468859 prob = 0.139842\r\n\r\n0 qid:10032 1:0.279152 2:0.000000 3:0.000000 4:0.000000 5:0.279152 ... 45:0.250000 46:1.000000 #docid = GX030-77-6315042 inc = 1 prob = 0.341364\r\n\r\n0 qid:10032 1:0.130742 2:0.000000 3:0.333333 4:0.000000 5:0.134276 ... 45:0.750000 46:1.000000 #docid = GX140-98-13566007 inc = 1 prob = 0.0701303\r\n\r\n1 qid:10032 1:0.593640 2:1.000000 3:0.000000 4:0.000000 5:0.600707 ... 45:0.500000 46:0.000000 #docid = GX256-43-0740276 inc = 0.0136292023050293 prob = 0.400738\r\n\r\n=================================<\/li>\r\n \t<li>Semi-supervised rankingThe data format in this setting is the same as that in supervised ranking setting. The only difference is that the datasets in this setting contains both judged and undged query-document pair (in training set but not in validation and testing set) while the datasets in supervised ranking contain only judged query-document pair. The relevance label ``-1'' indicates the query-document pair is not judged. An example is shown as follow.<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n<p style=\"padding-left: 30px\">-1 qid:18219 1:0.022594 2:0.000000 3:0.250000 4:0.166667 ... 45:0.004237 46:0.081600 #docid = GX004-66-12099765 inc = -1 prob = 0.223732<\/p>\r\n<p style=\"padding-left: 30px\">0 qid:18219 1:0.027615 2:0.500000 3:0.750000 4:0.333333 ... 45:0.010291 46:0.046400 #docid = GX004-93-7097963 inc = 0.0428115405134536 prob = 0.860366<\/p>\r\n<p style=\"padding-left: 30px\">-1 qid:18219 1:0.018410 2:0.000000 3:0.250000 4:0.166667 ... 45:0.003632 46:0.033600 #docid = GX005-04-11520874 inc = -1 prob = 0.0980801<\/p>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n\r\n<ul>\r\n \t<li>Rank aggregationIn the setting, a query is associated with a set of input ranked lists. The task of rank aggregation is to output a better final ranked list by aggregating the multiple input lists. A row in the data indicate a query-document pair. Several rows are shown as below.<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n<p style=\"padding-left: 30px\">0 qid:10002 1:1 2:30 3:48 4:133 5:NULL ... 25:NULL #docid = GX008-86-4444840 inc = 1 prob = 0.086622<\/p>\r\n<p style=\"padding-left: 30px\">0 qid:10002 1:NULL 2:NULL 3:NULL 4:NULL 5:NULL ... 25:NULL #docid = GX037-06-11625428 inc = 0.0031586555555558 prob = 0.0897452<\/p>\r\n<p style=\"padding-left: 30px\">2 qid:10032 1:6 2:96 3:88 4:NULL 5:NULL ... 25:NULL #docid = GX029-35-5894638 inc = 0.0119881192468859 prob = 0.139842<\/p>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n<p style=\"padding-left: 30px\">The first column is relevance label of this pair, the second column is query id, the following columns are ranks of the document in the input ranked lists, and the end of the row is comment about the pair, including id of the document.In the above example, 2:30 means that the ranks of the document is 30 in the second input list. Note that large ranks mean top positions in the input ranked list, and ``NULL'' means the document does not appear in a ranked list. The larger the relevance label, the more relevant the query-document pair. There are 21 input lists in MQ2007-agg dataset and 25 input lists in MQ2008-agg dataset.<\/p>\r\n\r\n<ul>\r\n \t<li>Listwise ranking<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\">The data format in the setting is very similar to that in supervised ranking. The difference is that the ground truth of this setting is a permutation for a query instead of multiple level relevance judgements. As shown in the following examples, the first column is the relevance degree of a document in ground truth permutation. Large value of the relevance degree means top position of the document in the permutation. The other columns are the same as that in the setting of supervised ranking.<\/p>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n<p style=\"padding-left: 30px\">1008 qid:10 1:0.004356 2:0.080000 3:0.036364 4:0.000000 ... 46:0.000000 #docid = GX057-59-4044939 inc = 1 prob = 0.698286<\/p>\r\n<p style=\"padding-left: 30px\">1007 qid:10 1:0.004901 2:0.000000 3:0.036364 4:0.333333 ... 46:0.000000 #docid = GX235-84-0891544 inc = 1 prob = 0.567746<\/p>\r\n<p style=\"padding-left: 30px\">1006 qid:10 1:0.019058 2:0.240000 3:0.072727 4:0.500000 ... 46:0.000000 #docid = GX016-48-5543459 inc = 1 prob = 0.775913<\/p>\r\n<p style=\"padding-left: 30px\">1005 qid:10 1:0.004901 2:0.160000 3:0.018182 4:0.666667 ... 46:0.000000 #docid = GX068-48-12934837 inc = 1 prob = 0.659932<\/p>\r\n<p style=\"padding-left: 30px\">==============================<\/p>\r\n\r\n<\/div>\r\n<div>\r\n<h3>More low level information<\/h3>\r\nUpdate: Due to website update, all the datasets are moved to cloud (hosted on OneDrive) and can be downloaded <a href=\"https:\/\/1drv.ms\/f\/s!Aqi9ONgj3OqPaynoZZSZVfHPJd0\">here<\/a>. You can get\u00a0the file name from\u00a0the following link and find the corresponding file in OneDrive. Please contact {taoqin AT microsoft DOT com} if any questions.\r\n<ul>\r\n \t<li>Meta dataMeta data for all queries in the two query sets. The information can be used to reproduce some features like BM25 and LMIR, and can also be used to construct some new features. <a href=\"MQ2007meta.rar\">Meta data for MQ2007 query set<\/a> (~ 60M). <a href=\"MQ2008meta.rar\">Meta data for MQ2008 query set<\/a> (~ 50M). <a href=\"Gov2meta.xml\">Collection info<\/a> (~1 k).<\/li>\r\n \t<li><a href=\"Gov2link.rar\">Link graph<\/a> (~ 480M) of Gov2 collectionEach line contains the inlinks of a web page. The first column is the MSRA doc id of the web page, the second column is the number of inlinks of this page, and the following columns list the MSRA doc ids of all the inlinks of this page. The mapping from MSRA doc id to TREC doc id can be found <a href=\"Gov2idmap.rar\">here<\/a> (~ 140M).<\/li>\r\n \t<li><a href=\"Gov2sitemap.rar\">Sitemap<\/a> (~ 65M) of Gov2 collectionEach line is a web page. The first column is the MSRA doc id of the page, the second column is the depth of the url (number of slashes), the third column is the length of url (without ``http:\/\/\"), the fourth column is the number of its child pages in the sitemap, the fifth column is the MSRA doc id of its parent page (-1 indicates no parent page).<\/li>\r\n \t<li>Similarity relation of Gov2 collection<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\">The data is organized by queries. <a href=\"MQ2007-simi.rar\">Similarity for MQ2007 query set <\/a>(~ 4.3G), similarity for MQ2008 query set(<a href=\"MQ2008-simi.part1.rar\">part1<\/a> and <a href=\"MQ2008-simi.part2.rar\">part2<\/a>, \u00a0~ 4.9G).The order of queries in the two files is the same as that in Large_null.txt in the <a href=\"MQ2007-semi.rar\">MQ2007-semi dataset<\/a> and <a href=\"MQ2008-semi.rar\"> MQ2008-semi dataset<\/a>.<\/p>\r\n<p style=\"padding-left: 30px\">The order of documents of a query in the two files is also the same as that in Large_null.txt in the <a href=\"MQ2007-semi.rar\">MQ2007-semi dataset<\/a> and <a href=\"MQ2008-semi.rar\"> MQ2008-semi dataset<\/a>.<\/p>\r\n<p style=\"padding-left: 30px\">Each row in the similarity files describes the similarity between a page and all the other pages under a same query. Note that i-th row in the similiar files is exactly corresponding to the i-th row in Large_null.txt in <a href=\"MQ2007-semi.rar\"> MQ2007-semi dataset<\/a> or <a href=\"MQ2008-semi.rar\">MQ2008-semi dataset<\/a>. Here is the an example line:<\/p>\r\n<p style=\"padding-left: 30px\">============================<\/p>\r\n<p style=\"padding-left: 30px\"><i>qid:10002 qdid:1 406:0.785623 178:0.785519 481:0.784446 63:0.741556 882:0.512454 ...<\/i><\/p>\r\n<p style=\"padding-left: 30px\"><i> <\/i>============================<\/p>\r\n<p style=\"padding-left: 30px\">The first column shows the query id, and the second column shows the page index under the query. For example, for a query with 1000 web pages, the page index ranges from 1 to 1000. The following columns show the similarity between this page and the other pages. For example, 406:0.785623 indicates that the similarity between this page (with index 1 under the query) and the page (with index 406 under the query) is 0.785623. We sort the pages according to the descending order of similarity. The similarity between two pages is consine similarity between the contents of the two pages.<\/p>\r\n\r\n<\/div>\r\n<div>\r\n<h3>Additional Notes<\/h3>\r\n<ul>\r\n \t<li>The following people contributed to the the construction of the LETOR4.0 dataset: <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/taoqin\/\" target=\"_blank\" rel=\"noopener noreferrer\">Tao Qin<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/tyliu\/\" target=\"_blank\" rel=\"noopener noreferrer\">Tie-Yan Liu<\/a>, Wenkui Ding, Jun Xu, Hang Li, Ben Carterette, Javed Aslam, James Allan, Stephen Robertson, Virgil Pavlu, and Emine Yilmaz .<\/li>\r\n \t<li>We would like to thank the following teams to kindly and generiously share their runs submitted to TREC2007\/2008: NEU team, U. Massachusetts team, I3S_Group_of_ICT team, ARSC team, IBM Haifa team, MPI-d5 team, Sabir.buckley team, HIT team, RMIT team, U. Amsterdam team, U. Melbourne team,<\/li>\r\n \t<li>If you have any questions or suggestions with this version, please kindly <a href=\"mailto:letor@microsoft.com\">let us know<\/a>. Our goal is to make the dataset reliable and useful for the community.<\/li>\r\n<\/ul>\r\n<\/div>\r\n[\/panel]\r\n\r\n[panel header=\"Baselines\"]\r\n<h3>Baselines for supervised ranking<\/h3>\r\nUpdate: Due to website update, all the datasets are moved to cloud (hosted on OneDrive) and can be downloaded <a href=\"https:\/\/1drv.ms\/u\/s!Aqi9ONgj3OqPgSS45WACJ5uKK-ok\">here<\/a>. You can get\u00a0the file name from\u00a0the following link and find the corresponding file in OneDrive. Please contact {taoqin AT microsoft DOT com} if any questions.\r\n<ul>\r\n \t<li>Algorithms using linear ranking function\r\n<table border=\"2\">\r\n<tbody>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Algorithms<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Notes<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Experiments by<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">RankSVM-Struct<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankSVM-Struct.MQ2007.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankSVM-Struct.MQ2008.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankSVM-Struct.html\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Tao Qin<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">ListNet<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/ListNet.MQ2007.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/ListNet.MQ2008.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/ListNet.html\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Di He<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">AdaRank-NDCG<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/AdaRank-NDCG.MQ2007.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/AdaRank-NDCG.MQ2008.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Jun Xu<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">AdaRank-MAP<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/AdaRank-MAP.MQ2007.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/AdaRank-MAP.MQ2008.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Jun Xu<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/li>\r\n \t<li>Algorithms using nonlinear ranking function\r\n<table border=\"2\">\r\n<tbody>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Algorithms<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Notes<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Experiments by<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">RankBoost<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankBoost.MQ2007.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankBoost.MQ2008.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/RankBoost.html\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Jun Xu<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/li>\r\n<\/ul>\r\n<h3>Baselines for rank aggregation<\/h3>\r\n<table border=\"2\">\r\n<tbody>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Algorithms<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-agg<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-agg<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Notes<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Experiments by<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">BordaCount<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/BordaCount.MQ2007-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/BordaCount.MQ2008-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/BordaCount.aspx\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Tao Qin<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">CPS-KendallTau<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-KendallTau.MQ2007-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-KendallTau.MQ2008-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS.aspx\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Xiubo Geng<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">CPS-SpearmanFootrule<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-SpearmanFootrule.MQ2007-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-SpearmanFootrule.MQ2008-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS.aspx\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Xiubo Geng<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">CPS-SpearmanRankCorrelation<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-SpearmanRankCorrelation.MQ2007-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS-SpearmanRankCorrelation.MQ2008-agg.txt\">here<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR4.0\/Baselines\/CPS.aspx\" target=\"_blank\" rel=\"noopener noreferrer\">algorithm details<\/a><\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Xiubo Geng<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n[\/panel]\r\n\r\n[panel header=\"Download\"]\r\n\r\nTo use the datasets, you must read and accept <a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR-agreement.txt\" target=\"_blank\" rel=\"noopener noreferrer\"> the online agreement<\/a>. By using the datasets, you agree to be bound by the terms of its license.\r\n<div>\r\n<h3>Datasets<\/h3>\r\n<em>Update: Due to website update, all the datasets are moved to cloud (hosted on OneDrive) and can be <a href=\"https:\/\/1drv.ms\/f\/s!Aqi9ONgj3OqPaynoZZSZVfHPJd0\">downloaded here<\/a>. You can get the file name from the following table and fetch the corresponding file in OneDrive. Please contact {taoqin AT microsoft DOT com} if any \u00a0questions.<\/em>\r\n\r\n<span class=\"highlight\">Note that the two semi-supervised ranking datasets have been updated on Jan. 7, 2010. Please download the new version if you are using the old ones.<\/span>\r\n<table border=\"2\">\r\n<tbody>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Setting<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Datasets<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">Size<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Supervised ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 65M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 15M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Semi-supervised ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-semi.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 940M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-semi.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 650M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Rank aggregation<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-agg.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 20M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-agg.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 4M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\" rowspan=\"2\">Listwise ranking<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2007-list.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 950M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">MQ2008-list.rar<\/td>\r\n<td style=\"border: #000 solid 1px;padding: 5px\">~ 670M<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nFeature list for supervised ranking, semi-supervised ranking and listwise ranking can be found in <a href=\"http:\/\/arxiv.org\/ftp\/arxiv\/papers\/1306\/1306.2597.pdf\">this document<\/a>.\r\n\r\n<\/div>\r\n<div>\r\n<h3>Evaluation tools<\/h3>\r\nThe evaluation scripts for LETOR4.0 are a little different from those for LETOR3.0.\r\n\r\n<span class=\"highlight\">Please do not use the tools across LETOR3.0 and LETOR4.0.<\/span>\r\n\r\n<a href=\"https:\/\/onedrive.live.com\/?authkey=%21ACnoZZSZVfHPJd0&amp;cid=8FEADC23D838BDA8&amp;id=8FEADC23D838BDA8%21122&amp;parId=8FEADC23D838BDA8%21107&amp;o=OneUp\">Evaluation script<\/a> for supervised ranking, semi-supervised ranking and rank aggregation\r\n\r\n<a href=\"https:\/\/onedrive.live.com\/?authkey=%21ACnoZZSZVfHPJd0&amp;cid=8FEADC23D838BDA8&amp;id=8FEADC23D838BDA8%21124&amp;parId=8FEADC23D838BDA8%21107&amp;o=OneUp\">Evaluation script<\/a> for listwise ranking\r\n\r\n<a href=\"https:\/\/onedrive.live.com\/?authkey=%21ACnoZZSZVfHPJd0&amp;cid=8FEADC23D838BDA8&amp;id=8FEADC23D838BDA8%21123&amp;parId=8FEADC23D838BDA8%21107&amp;o=OneUp\">Significance test script<\/a> for all the four settings\r\n\r\n<span class=\"highlight\">Possible issues<\/span>\r\nIf you are using a linux machine and meet some problems with the scripts, you may try the solution from Sergio Daniel. Thank Sergio for sharing!\r\n\r\n-------------------------\r\n\r\nThe evaluation script (http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/\/LETOR4.0\/Evaluation\/Eval-Score-4.0.pl.txt) isn't working for me on the letor 4.0 MQ2008 dataset. I use perl v5.14.2 on a linux machine. I made a little modification and now it is running =)\r\n\r\nI replaced the line:\r\n\r\nif ($lnFea =~ m\/^(\\d+) qid\\:([^\\s]+).*?\\#docid = ([^\\s]+) inc = ([^\\s]+) prob = ([^\\s]+)$\/)\r\n\r\nwith:\r\n\r\nif ($lnFea =~ m\/^(\\d+) qid\\:([^\\s]+).*?\\#docid = ([^\\s]+) inc = ([^\\s]+) prob = ([^\\s]+).$\/)\r\n\r\nSergio.\r\n\r\n-------------------------\r\n<div>\r\n<h3>Low level information<\/h3>\r\n<\/div>\r\n<table style=\"border-collapse: collapse;border-spacing: inherit\" border=\"2\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">Data<\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">Size<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\" rowspan=\"3\">Meta data<\/td>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"MQ2007meta.rar\">Meta data for MQ2007 query set<\/a><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 60M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"MQ2008meta.rar\">Meta data for MQ2008 query set<\/a><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 50M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"Gov2meta.xml\">Collection info<\/a><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~1 k<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\" rowspan=\"4\">Relation information<\/td>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"Gov2link.rar\">Link graph<\/a> of Gov2 collection<\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 480M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"Gov2sitemap.rar\">Sitemap<\/a> of Gov2 collection<\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 65M<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"https:\/\/1drv.ms\/u\/s!Al2jmW8Z8pnEadLOaVbXdvRbzhI\">Similarity for MQ2007 query set<\/a><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 4.3G<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: inherit;border: 2px solid\"><a href=\"https:\/\/1drv.ms\/u\/s!AlcPdxQyH9pwaeQEZA6UiSzTkDM\">similarity for MQ2008 query set<\/a><\/td>\r\n<td style=\"padding: inherit;border: 2px solid\">~ 4.9G<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n[\/panel][\/accordion]"},{"id":1,"name":"LETOR 3.0","content":"[accordion]\r\n\r\n[panel header=\"Datasets\"]\r\nLETOR is a package of benchmark data sets for research on LEarning TO Rank. LETOR3.0 contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines, for the OHSUMED data collection and the '.gov' data collection. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008.\r\n<h3>Recent Updates<\/h3>\r\n<ul>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/letor3dataset.aspx#simirelation\">Similarity relation<\/a> of OHSUMED collection is released.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/letor3dataset.aspx#sitemap\">Sitemap<\/a> of Gov collection is released.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/letor3dataset.aspx#linkgraph\">Link graph<\/a> of Gov collection is released.<\/li>\r\n<\/ul>\r\n<h3>What's new in LETOR3.0?<\/h3>\r\nLETOR3.0 contains several significant updates comparing with version 2.0:\r\n<ul>\r\n \t<li>Add four new datasets: homepage finding 2003, homepage finding 2004, named page finding 2003 and named page finding 2004. Plus the three datasets (OHSUMED, topic distillation 2003 and topic distillation 2004) in LETOR2.0, there are seven datasets in LETOR3.0.<\/li>\r\n \t<li>New document sampling strategy for each query; and so the three datasets in LETOR3.0 are different from those in LETOR2.0;<\/li>\r\n \t<li>New low level features for learning;<\/li>\r\n \t<li>Meta data is provided for better investigation of ranking features;<\/li>\r\n \t<li>More baselines;<\/li>\r\n<\/ul>\r\n<h3>Introduction to LETOR3.0 datasets<\/h3>\r\nPlease access <a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/letor3download.aspx\">this page<\/a> for download.\r\n\r\nA brief description about the directory tree is as follows:\r\n<table border=\"1\">\r\n<tbody>\r\n<tr>\r\n<td class=\"style4\">Folder or file<\/td>\r\n<td class=\"style3\">Description<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Letor.pdf<\/td>\r\n<td class=\"style3\">An incomplete document about the whole dataset.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">EvaluationTool<\/td>\r\n<td class=\"style3\">The evaluation tools<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Gov<\/td>\r\n<td class=\"style3\">Contain 6 datasets in .Gov<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Gov\\Meta<\/td>\r\n<td class=\"style3\">Meta data for all queries in 6 datasets in .Gov. The information can be used to extract some new features.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Gov\\Feature_null<\/td>\r\n<td class=\"style3\">Original feature files of 6 datasets in .Gov. Since some document may do not contain query terms, we use ``NULL'' to indicate language model features, for which would be minus infinity values.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Gov\\Feature_min<\/td>\r\n<td class=\"style3\">Replace the ``NULL'' value in Gov\\Feature_null with the minimal vale of this feature under a same query. This data can be directly used for learning.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">Gov\\QueryLevelNorm<\/td>\r\n<td class=\"style3\">Conduct query level normalization based on data files in Gov\\Feature_min. This data can be directly used for learning.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">OHSUMED<\/td>\r\n<td class=\"style3\">Contain the OHSUMED dataset<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">OHSUMED\\Meta<\/td>\r\n<td class=\"style3\">Meta data for all queries in 6 datasets in .gov. The information can be used to extract some new features.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">OHSUMED\\Feature_null<\/td>\r\n<td class=\"style3\">Original feature files of OHSUMED. Since some document may do not contain query terms, we use ``NULL'' to indicate language model features, for which would be a minus infinity values.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">OHSUMED\\Feature_min<\/td>\r\n<td class=\"style3\">Replace the ``NULL'' value in OHSUMED \\Feature_null with the minimal vale of this feature under a same query. This data can be directly used for learning.<\/td>\r\n<\/tr>\r\n<tr>\r\n<td class=\"style4\">OHSUMED\\QueryLevelNorm<\/td>\r\n<td class=\"style3\">Conduct query level normalization based on data files in OHSUMED \\Feature_min. This data can be directly used for learning.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n&nbsp;\r\n<h3>More Information<\/h3>\r\nAfter the release of LETOR3.0, we have recieved many valuable suggestions and feedbacks. According to the suggestions, we release more information about the datasets.\r\n<ul>\r\n \t<li>Similarity relation of OHSUMED collection<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR3.0\/ohsumed_simi.zip\">Similarity relation<\/a>. The data is organized by queries. The order of queries in the file is the same as that in OHSUMED\\Feature_null\\ALL\\OHSUMED.txt. The documents of a query in the similarity file are also in the same order as the OHSUMED\\Feature_null\\ALL\\OHSUMED.txt file The similarity graph among documents under a specific query is encoded by a upper triangle matrix. Here is the example for a query:<\/p>\r\n<p style=\"padding-left: 30px\">============================<\/p>\r\n<p style=\"padding-left: 30px\"><i>N<\/i><\/p>\r\n<p style=\"padding-left: 30px\"><i>S(1,2) S(1,3) S(1,4) ... S(1,N)<\/i><\/p>\r\n<p style=\"padding-left: 30px\"><i>S(2,3) S(2,4) ... S(2,N)<\/i><\/p>\r\n<p style=\"padding-left: 30px\">...<\/p>\r\n<p style=\"padding-left: 30px\"><i>S(N-2,N-1) S(N-2,N)<\/i><\/p>\r\n<p style=\"padding-left: 30px\"><i>S(N-1,N)<\/i><\/p>\r\n<p style=\"padding-left: 30px\">============================<\/p>\r\n<p style=\"padding-left: 30px\">in which N is the number of documents under this query, S(i,j) means the similarity between the i-th and j-th documents of the query. We simply use cosine similarity beteen the contents of two documents.<\/p>\r\n\r\n<ul>\r\n \t<li>Sitemap of Gov collection<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR3.0\/sitemap_gov.zip\">Sitemap<\/a>. Each line is a web page. The first column is the MSRA doc id of the page, the second column is the depth of the url (number of slashes), the third column is the lenghth of url (without ``http:\/\/\"), the fourth column is the number of its child pages in the sitemap, the fifth column is the MSRA doc id of its parent page (-1 indicates no parent page).<\/p>\r\n<p style=\"padding-left: 30px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR3.0\/reftable.zip\">Mapping from MSRA doc id to TREC doc id<\/a><\/p>\r\n\r\n<ul>\r\n \t<li>Link graph of Gov collection<\/li>\r\n<\/ul>\r\n<p style=\"padding-left: 30px\"><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR3.0\/links_info.zip\">Link graph<\/a>. Each line is a hyperlink. The first column is the MSRA doc id of the source of the hyperlink, and the second column is the MSRA doc id of the destination of the hyperlink.<a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR3.0\/reftable.zip\">Mapping from MSRA doc id to TREC doc id<\/a><\/p>\r\n\r\n<h3>Additional Notes<\/h3>\r\n<ul>\r\n \t<li>The old version of LETOR can be found <a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/letor-old.aspx\">here<\/a>.<\/li>\r\n \t<li>The following people contributed to the the construction of the LETOR dataset: <a href=\"http:\/\/research.microsoft.com\/en-us\/people\/taoqin\/\">Tao Qin<\/a>, <a href=\"http:\/\/research.microsoft.com\/users\/tyliu\/\">Tie-Yan Liu<\/a>, <a href=\"http:\/\/research.microsoft.com\/en-us\/people\/junxu\/\">Jun Xu<\/a>, Chaoliang Zhong, Kang Ji, and <a href=\"http:\/\/research.microsoft.com\/en-us\/people\/hangli\/\" target=\"_blank\" rel=\"noopener noreferrer\"> Hang Li<\/a>.<\/li>\r\n \t<li>If you have any questions or suggestions with this version, please kindly <a href=\"mailto:letor@microsoft.com\"> let us know<\/a>. Our goal is to make the dataset reliable and useful for the community.<\/li>\r\n<\/ul>\r\n[\/panel]\r\n\r\n[panel header=\"Baselines\"]\r\n<h3>Algorithms with linear ranking function<\/h3>\r\n<table border=\"2\" bgcolor=\"transparent\">\r\n<tbody>\r\n<tr>\r\n<th><\/th>\r\n<th>TD2003<\/th>\r\n<th>TD2004<\/th>\r\n<th>NP2003<\/th>\r\n<th>NP2004<\/th>\r\n<th>HP2003<\/th>\r\n<th>HP2004<\/th>\r\n<th>OHSUMED<\/th>\r\n<th>Prediction files on test set<\/th>\r\n<th>Notes<\/th>\r\n<th>Experiments by<\/th>\r\n<\/tr>\r\n<tr bgcolor=\"#dddddd\">\r\n<th>Regression<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression\/Regression.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Da Kuang<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#ffffff\">\r\n<th>RankSVM<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM\/RankSVM.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Chaoliang Zhong<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#dddddd\">\r\n<th>ListNet<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet\/ListNet.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/ListNet.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Da Kuang<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#ffffff\">\r\n<th>AdaRank-MAP<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-MAP\/AdaRank-MAP.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Chaoliang Zhong<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#dddddd\">\r\n<th>AdaRank-NDCG<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank-NDCG\/AdaRank-NDCG.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/AdaRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Chaoliang Zhong<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#ffffff\">\r\n<th>SVMMAP<\/th>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.td2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.td2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.np2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.np2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.hp2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.hp2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/svmmap.ohsumed.txt\">here<\/a><\/td>\r\n<td>not available<\/td>\r\n<td><a href=\"http:\/\/www.yisongyue.com\/results\/svmmap_letor3\/details.html\" target=\"_blank\" rel=\"noopener noreferrer\"> Algorithm details<\/a><\/td>\r\n<td>Yisong Yue<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n&nbsp;\r\n<h3>Algorithms with nonlinear ranking function<\/h3>\r\n<table border=\"2\" bgcolor=\"transparent\">\r\n<tbody>\r\n<tr>\r\n<th><\/th>\r\n<th>TD2003<\/th>\r\n<th>TD2004<\/th>\r\n<th>NP2003<\/th>\r\n<th>NP2004<\/th>\r\n<th>HP2003<\/th>\r\n<th>HP2004<\/th>\r\n<th>OHSUMED<\/th>\r\n<th>Prediction files on test set<\/th>\r\n<th>Notes<\/th>\r\n<th>Experiments by<\/th>\r\n<\/tr>\r\n<tr>\r\n<th>RankBoost<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost\/RankBoost.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankBoost.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Yong-Deok Kim<\/td>\r\n<\/tr>\r\n<tr>\r\n<th>FRank<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.OHSUMED.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank\/FRank.testscore.zip\">test scores<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/FRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Ming-Feng Tsai<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n&nbsp;\r\n<h3>Recently added algorithms (with linear ranking function)<\/h3>\r\nPlease note that the above experimental results are still primal, since the result of almost every algorithm can be further improved. For example, for regression, we can add regularization item to make it more robust; for RankSVM, we can run more steps of iteration so as to guarantee a better convergence of the optimization; for ListNet, we can also add regularization item to its loss function and make it more generalizable to the test set. Any updates about the above algorithms or new ranking algorithms are welcome. The following table lists the updated results of several algorithms (Regression and RankSVM) and a new algorithm SmoothRank.We would like to thank Dr. Olivier Chapelle and Prof. Thorsten Joachims for kindly contributing the results.\r\n<table border=\"2\" bgcolor=\"transparent\">\r\n<tbody>\r\n<tr>\r\n<th><\/th>\r\n<th>TD2003<\/th>\r\n<th>TD2004<\/th>\r\n<th>NP2003<\/th>\r\n<th>NP2004<\/th>\r\n<th>HP2003<\/th>\r\n<th>HP2004<\/th>\r\n<th>OHSUMED<\/th>\r\n<th>Prediction files on test set<\/th>\r\n<th>Notes<\/th>\r\n<th>Experiments by<\/th>\r\n<\/tr>\r\n<tr bgcolor=\"#dddddd\">\r\n<th>Regression+L2 reg<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-Reg\/Regression-Reg.OHSUMED.txt\">here<\/a><\/td>\r\n<td><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/Regression-reg.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Dr. Olivier Chapelle<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#ffffff\">\r\n<th>RankSVM-Primal<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal\/RankSVM-Primal.OHSUMED.txt\">here<\/a><\/td>\r\n<td><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Primal.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Dr. Olivier Chapelle<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#dddddd\">\r\n<th>RankSVM-Struct<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct\/RankSVM-Struct.OHSUMED.txt\">here<\/a><\/td>\r\n<td><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/RankSVM-Struct.txt\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Prof. Thorsten Joachims<\/td>\r\n<\/tr>\r\n<tr bgcolor=\"#ffffff\">\r\n<th>SmoothRank<\/th>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.TD2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.TD2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.NP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.NP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.HP2003.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.HP2004.txt\">here<\/a><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank\/SmoothRank.OHSUMED.txt\">here<\/a><\/td>\r\n<td><\/td>\r\n<td><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/SmoothRank.html\" target=\"_blank\" rel=\"noopener noreferrer\">Algorithm details<\/a><\/td>\r\n<td>Dr. Olivier Chapelle<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n&nbsp;\r\n<h3>Summary of all algorithms and datasets<\/h3>\r\n<a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/Baselines\/letor3baselines.zip\">Excel file<\/a>\r\n<h3>How to compare with the baselines?<\/h3>\r\nWe note that different setting of experiments may greatly affect the performance of a ranking algorithm. To make fair comparisons, we encourage everyone to follow these common settings while using LETOR; deviations from these defaults must be noted when reporting results.\r\n<ul>\r\n \t<li>All reported algorithms use the \"QueryLevelNorm\" version of the datasets (i.e. query level normalization for feature processing). You are encouraged to use the same version and should indicate if you use a different one.<\/li>\r\n \t<li>The test set cannot be used in any manner to make decisions about the structure or parameters of the model.<\/li>\r\n \t<li>The validation set can only be used for model selection (setting hyper-parameters and model structure), but cannot be used for learning. Most baselines released in LETOR website use MAP on the validation set for model selection; you are encouraged to use the same strategy and should indicate if you use a different one.<\/li>\r\n \t<li>All reported results must use the provided evaluation utility. While using the evaluation script, please use the original dataset. The evaluation tool (Eval-Score-3.0.pl) sorts the documents with same ranking scores according to their input order. That is, it is sensitive to the document order in the input file.<\/li>\r\n \t<li>Please explicitly show the function class of ranking models (e.g. linear model, two layer neural net, or decision trees) in your work.<\/li>\r\n<\/ul>\r\n&nbsp;\r\n<h3>Additional Notes<\/h3>\r\n<ul>\r\n \t<li>The prediction score files on test set can be viewed by any text editor such as notepad.<\/li>\r\n \t<li>More algorithms will be added in future.<\/li>\r\n \t<li>If you would be like to publish the results of your algorithm here, please <a href=\"mailto:letor@microsoft.com\">let us know<\/a><\/li>\r\n<\/ul>\r\n[\/panel]\r\n\r\n[panel header=\"Download\"]\r\n\r\nTo use the datasets, you must read and accept <a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/projects\/letor\/LETOR-agreement.txt\" target=\"_blank\" rel=\"noopener noreferrer\">the online agreement<\/a>. By using the datasets, you agree to be bound by the terms of its license.\r\n\r\n<em>Update: Due to website update, all the datasets are moved to cloud (hosted on OneDrive) and can be downloaded <a href=\"https:\/\/1drv.ms\/f\/s!Aqi9ONgj3OqPaynoZZSZVfHPJd0\">here<\/a>. You can get\u00a0the file name as below and find the corresponding file in OneDrive. Please contact {taoqin AT microsoft DOT com} if any questions.<\/em>\r\n\r\n<strong>Download<\/strong>\r\n\r\n\"<a href=\"https:\/\/1drv.ms\/u\/s!Ao5ICDLaLvvAaW2uoviiYYG5lTE\">Gov.rar<\/a>\", \u00a0the .Gov dataset (about 1G),\r\n\r\n\"OHSUMED.rar\", the OHSUMED dataset (about 30M),\r\n\r\nand \"EvaluationTool.zip\", the evaluation tools (about 400k).\r\n\r\n[\/panel]\r\n\r\n[\/accordion]"},{"id":2,"name":"Resources","content":"[accordion]\r\n\r\n[panel header=\"Papers\"]\r\n\r\nBecause of the fast development of this area, it is difficult to keep the list up-to-date and comprehensive. For a comprehensive list and more recent papers, please refer to\r\n<ul>\r\n \t<li><a href=\"http:\/\/academic.research.microsoft.com\/Search.aspx?query=learning%20to%20rank\">Microsoft Academic Search<\/a><\/li>\r\n \t<li><a href=\"http:\/\/scholar.google.com\/scholar?hl=en&amp;q=learning+to+rank&amp;as_sdt=100000000000000&amp;as_ylo=2010&amp;as_vis=0\">Google Scholar<\/a><\/li>\r\n<\/ul>\r\nIf your paper is not listed, please let us know <a href=\"mailto:taoqin@microsoft.com\">taoqin@microsoft.com<\/a>.\r\n<ol>\r\n \t<li>James Petterson, Tiberio Caetano, Julian McAuley and Jin Yu. Exponential Family Graph Matching and Ranking. In NIPS 2009.<\/li>\r\n \t<li>C. Rudin. The P-Norm Push: A Simple Convex Ranking Algorithm that Concentrates at the Top of the List. Journal of Machine Learning Research, 10 (2009) 2233-2271.<\/li>\r\n \t<li>C. Rudin, and R. Schapire. Margin-Based Ranking and an Equivalence Between AdaBoost and RankBoost. Journal of Machine Learning Research, 10 (2009) 2193-2232.<\/li>\r\n \t<li>C. Rudin, R. Passonneau, A. Radeva, H. Dutta, S. Ierome, and D. Isaac. A Process for Predicting Manhole Events in Manhattan. To appear, Machine Learning, 2010.<\/li>\r\n \t<li>S. Agarwal, T. Graepel, T. Herbrich, S. Har-Peled, and D. Roth. Generalization bounds for the area under the roc curve. Journal of Machine Learning, 6:393-425, 2005.<\/li>\r\n \t<li>S. Agarwal and P. Niyogi. Stability and generalization of bipartite ranking algorithms. In COLT 2005, pages 32-47, 2005.<\/li>\r\n \t<li>E. Agichtein, E. Brill, S. T. Dumais, and R. Ragno. Learning user interaction models for predicting web search result preferences. In SIGIR 2006, pages 3-10, 2006.<\/li>\r\n \t<li>N. Ailon and MehryarMohri. An efficient reduction from ranking to classification. In COLT 2008, 2008.<\/li>\r\n \t<li>H. Almeida, M. Goncalves, M. Cristo, and P. Calado. A combined component approach for finding collection-adapted ranking functions based on genetic programming. In SIGIR 2007, pages 399-406, 2007.<\/li>\r\n \t<li>M.-R. Amini, T.-V. Truong, and C. Goutte. A boosting algorithm for learning bipartite ranking functions with partially labeled data. In SIGIR 2008, pages 99-106, 2008.<\/li>\r\n \t<li>M.-F. Balcan, N. Bansal, A. Beygelzimer, D. Coppersmith, J. Langford, and G. B. Sorkin. Robust reductions from ranking to classification. In COLT 2007, 2007.<\/li>\r\n \t<li>B. Bartell, G. W. Cottrell, and R. Belew. Learning to retrieve information. In SCC 1995, 1995.<\/li>\r\n \t<li>C. J. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. In NIPS 2006, pages 395-402, 2006.<\/li>\r\n \t<li>C. J. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML 2005, pages 89-96, 2005.<\/li>\r\n \t<li>G. Cao, J. Nie, L. Si, J. Bai, Learning to Rank Documents for Ad-Hoc Retrieval with Regularized Models, SIGIR 2007 workshop: Learning to Rank for Information Retrieval, 2007<\/li>\r\n \t<li>Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Huang, and H.-W. Hon. Adapting ranking svm to document retrieval. In SIGIR 2006, pages 186-193, 2006.<\/li>\r\n \t<li>Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. Learning to rank: from pairwise approach to listwise approach. In ICML 2007, pages 129-136, 2007.<\/li>\r\n \t<li>V. R. Carvalho, J. L. Elsas, W. W. Cohen, and J. G. Carbonell. A metalearning\r\napproach for robust rank learning. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>S. Chakrabarti, R. Khanna, U. Sawant, and C. Bhattacharyya. Structured learning for non-smooth ranking losses. In SIGKDD 2008, pages 88-96, 2008.<\/li>\r\n \t<li>O. Chapelle, Q. Le, and A. Smola. Large margin optimization of ranking measures. In NIPS workshop on Machine Learning for Web Search 2007, 2007.<\/li>\r\n \t<li>W. Chu and Z. Ghahramani. Gaussian processes for ordinal regression. Journal of Machine Learning Research, 6:1019-1041, 2005.<\/li>\r\n \t<li>W. Chu and Z. Ghahramani. Preference learning with Gaussian processes. In ICML 2005, pages 137-144, 2005.<\/li>\r\n \t<li>W. Chu and S. S. Keerthi. New approaches to support vector ordinal regression. In ICML 2005, pages 145-152, 2005.<\/li>\r\n \t<li>S. Clemenson, G. Lugosi, and N. Vayatis. Ranking and scoring using empirical risk minimization. In COLT 2005, 2005.<\/li>\r\n \t<li>W. W. Cohen, R. E. Schapire, and Y. Singer. Learning to order things. In NIPS 1998, volume 10, pages 243-270, 1998.<\/li>\r\n \t<li>C. Cortes, M. Mohri, and etc. Magnitude-preserving ranking algorithms. In ICML 2007, pages 169-176, 2007.<\/li>\r\n \t<li>D. Cossock and T. Zhang. Subset ranking using regression. In COLT 2006, pages 605-619, 2006.<\/li>\r\n \t<li>K. Crammer and Y. Singer. Pranking with ranking. In NIPS 2002, pages 641-647, 2002.<\/li>\r\n \t<li>K. Duh and K. Kirchhoff. Learning to rank with partially-labeled data. In SIGIR 2008, pages 251-258, 2008.<\/li>\r\n \t<li>W. Fan, E. A. Fox, P. Pathak, and H. Wu. The effects of fitness functions on genetic programming based ranking discovery for web search. Journal of American Society for Information Science and Technology, 55(7):628-636, 2004.<\/li>\r\n \t<li>W. Fan, M. Gordon, and P. Pathak. Discovery of context-specific ranking functions for effective information retrieval using genetic programming. IEEE Transactions on Knowledge and Data Engineering, 16(4):523-527, 2004.<\/li>\r\n \t<li>W. Fan, M. Gordon, and P. Pathak. A generic ranking function discovery framework by genetic programming for information retrieval. Information Processing and Management, 40(4):587-602, 2004.<\/li>\r\n \t<li>W. Fan, M. Gordon, and P. Pathak. Genetic programming-based discovery of ranking functions for effective web search. Journal of Management of Information Systems, 21(4):37-56, 2005.<\/li>\r\n \t<li>W. Fan, M. Gordon, and P. Pathak. On linear mixture of expert approaches to information retrieval. Decision Support System, 42(2):975-987, 2006.<\/li>\r\n \t<li>W. Fan, M. D. Gordon, W. Xi, and E. A. Fox. Ranking function optimization for effective web search by genetic programming: an empirical study. In HICSS 2004, page 40105, 2004.<\/li>\r\n \t<li>Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933-969, 2003.<\/li>\r\n \t<li>N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Transactions on Information Systems, 7(3):183-204, 1989.<\/li>\r\n \t<li>G. Fung, R. Rosales, and B. Krishnapuram, Learning Rankings via Convex Hull Separation, NIPS 2005 workshop on Learning to Rank, 2005<\/li>\r\n \t<li>J. Gao, H. Qi, X. Xia, and J. Nie. Linear discriminant model for information retrieval. In SIGIR 2005, pages 290-297, 2005.<\/li>\r\n \t<li>X.-B. Geng, T.-Y. Liu, and T. Qin. Feature selection for ranking. In SIGIR 2007, pages 407-414, 2007.<\/li>\r\n \t<li>X.-B. Geng, T.-Y. Liu, T. Qin, H. Li, and H.-Y. Shum. Query-dependent ranking using k-nearest neighbor. In SIGIR 2008, pages 115-122, 2008.<\/li>\r\n \t<li>J. Guiver and E. Snelson. Learning to rank with softrank and gaussian processes. In SIGIR 2008, pages 259-266, 2008.<\/li>\r\n \t<li>E. F. Harrington. Online ranking\/collaborative filtering using the perceptron algorithm. In ICML 2003, pages 250-257, 2003.<\/li>\r\n \t<li>R. Herbrich, K. Obermayer, and T. Graepel. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115-132, 2000.<\/li>\r\n \t<li>R. Jin, H. Valizadegan, and H. Li. Ranking refinement and its application to information retrieval. In WWW 2008, pages 397-406, 2008.<\/li>\r\n \t<li>T. Joachims. Optimizing search engines using clickthrough data. In KDD 2002, pages 133-142, 2002.<\/li>\r\n \t<li>T. Joachims. A support vector method for multivariate performance measures. In ICML 2005, pages 377-384, 2005.<\/li>\r\n \t<li>S. Kramer, G. Widmer, B. Pfahringer, and M. D. Groeve. Prediction of ordinal classes using regression trees. Funfamenta Informaticae, 34:1-15, 2000.<\/li>\r\n \t<li>J. Lafferty and C. Zhai. Document language models, query models and risk minimization for information retrieval. In SIGIR 2001, pages 111-119, 2001.<\/li>\r\n \t<li>Y. Lan, T.-Y. Liu, T. Qin, Z. Ma, and H. Li. Query-level stability and generalization in learning to rank. In ICML 2008, pages 512-519, 2008.<\/li>\r\n \t<li>G. Lebanon and J. Lafferty. Cranking: Combining rankings using conditional probability models on permutations. In ICML 2002, pages 363-370, 2002.<\/li>\r\n \t<li>P. Li, C. Burges, and Q. Wu. Mcrank: Learning to rank using multiple classification and gradient boosting. In NIPS 2007, 2007.<\/li>\r\n \t<li>T.-Y. Liu, J. Xu, T. Qin, W.-Y. Xiong, and H. Li. LETOR: Benchmark dataset for research on learning to rank for information retrieval. In SIGIR \u201907 Workshop on learning to rank for information retrieval, 2007.<\/li>\r\n \t<li>Y. Liu, T.-Y. Liu, T. Qin, Z. Ma, and H. Li. Supervised rank aggregation. In WWW 2007, pages 481-490, 2007.<\/li>\r\n \t<li>I. Matveeva, C. Burges, T. Burkard, A. Laucius, and L. Wong. High accuracy retrieval with multiple nested ranker. In SIGIR 2006, pages 437-444, 2006.<\/li>\r\n \t<li>D. A. Metzler and W. B. Croft. A Markov random field model for term dependencies. In SIGIR 2005, pages 472-479, 2005.<\/li>\r\n \t<li>D. A. Metzler, W. B. Croft, and A. McCallum. Direct maximization of rank based metrics for information retrieval. In CIIR Technical Report, 2005.<\/li>\r\n \t<li>D. A. Metzler and T. Kanungo. Machine learned sentence selection strategies for query-biased summarization. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>T. Minka and S. Robertson. Selection bias in the LETOR datasets. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>R. Nallapati. Discriminative models for information retrieval. In SIGIR 2004, pages 64-71, 2004.<\/li>\r\n \t<li>T. Pahikkala, E. Tsivtsivadze, A. Airola, J. Boberg, T. Salakoski, Learning to Rank with Pairwise Regularized Least-Squares, SIGIR 2007 workshop: Learning to Rank for Information Retrieval, 2007<\/li>\r\n \t<li>L. Rigutini, T. Papini, M. Maggini, and F. Scarselli. Learning to rank by a neural-based sorting algorithm. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>T. Qin, T.-Y. Liu, W. Lai, X.-D. Zhang, D.-S.Wang, and H. Li. Ranking with multiple hyperplanes. In SIGIR 2007, pages 279-286, 2007.<\/li>\r\n \t<li>T. Qin, T.-Y. Liu, M.-F. Tsai, X.-D. Zhang, and H. Li. Learning to search web pages with query-level loss functions. Technical Report, MSR-TR-2006-156, 2006.<\/li>\r\n \t<li>T. Qin, T.-Y. Liu, X.-D. Zhang, D. Wang, and H. Li. Learning to rank relational objects and its application to web search. In WWW 2008, pages\r\n407-416, 2008.<\/li>\r\n \t<li>T. Qin, T.-Y. Liu, X.-D. Zhang, D.-S. Wang, and H. Li. Global ranking using continuous conditional random fields. In NIPS 2008, 2008.<\/li>\r\n \t<li>T. Qin, T.-Y. L. X.-D. Zhang, M.-F. Tsai, D.-S. Wang, and H. Li. Query-level loss functions for information retrieval. Information Processing &amp; Management, 44(2):838-855, 2007.<\/li>\r\n \t<li>T. Qin, T.-Y. Liu, J. Xu, and H. Li. How to make LETOR more useful and reliable. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>F. Radlinski and T. Joachims. Query chain: Learning to rank from implicit feedback. In KDD 2005, pages 239-248, 2005.<\/li>\r\n \t<li>F. Radlinski and T. Joachims. Active exploration for learning rankings from clickthrough data. In KDD 2007, 2007.<\/li>\r\n \t<li>F. Radlinski, R. Kleinberg, and T. Joachims. Learning diverse rankings with multi-armed bandits. In ICML 2008, pages 784-791, 2008.<\/li>\r\n \t<li>S. Rajaram and S. Agarwal. Generalization bounds for k-partite ranking. In NIPS 2005 WorkShop on Learning to Rank, 2005.<\/li>\r\n \t<li>S. Robertson and H. Zaragoza. On rank-based effectiveness measures and optimization. Information Retrieval, 10(3):321-339, 2007.<\/li>\r\n \t<li>C. Rudin, C. Cortes, M. Mohri, and R. E. Schapire, Margin-Based Ranking Meets Boosting in the Middle, COLT 2005.<\/li>\r\n \t<li>A. Shashua and A. Levin. Ranking with large margin principles: Two approaches. In NIPS 2002, pages 937-944, 2002.<\/li>\r\n \t<li>M. Talyor, J. Guiver, and etc. Softrank: Optimising non-smooth rank metrics. In WSDM 2008, pages 77-86, 2008.<\/li>\r\n \t<li>M. Taylor, H. Zaragoza, N. Craswell, S. Robertson, and C. J. Burges. Optimisation methods for ranking functions with multiple parameters. In CIKM 2006, pages 585-593, 2006.<\/li>\r\n \t<li>A. Trotman. Learning to rank. Information Retrieval, 8(3):359-381, 2005.<\/li>\r\n \t<li>M.-F. Tsai, T.-Y. Liu, T. Qin, H.-H. Chen, and W.-Y. Ma. Frank: a ranking method with fidelity loss. In SIGIR 2007, pages 383-390, 2007.<\/li>\r\n \t<li>A. Veloso, H. M. de Almeida, M. A. Gon?alves, and W. M. Jr. Learning to rank at query-time using association rules. In SIGIR 2008, pages 267-274, 2008.<\/li>\r\n \t<li>N. Usunier, V. Truong, M. R. Amini, and P. Gallinari, Ranking with Unlabeled Data: A First Study, NIPS 2005 workshop:Learning to Rank, 2005.<\/li>\r\n \t<li>W. Xi, J. Lind, and E. Brill, Learning effective ranking functions for newsgroup search, SIGIR 2004.<\/li>\r\n \t<li>F. Xia, T.-Y. Liu, J. Wang, W. Zhang, and H. Li. Listwise approach to learning to rank - theorem and algorithm. In ICML 2008, pages 1192-1199, 2008.<\/li>\r\n \t<li>J. Xu, Y. Cao, H. Li, and Y. Huang. Cost-sensitive learning of SVM for ranking. In ECML 2006, pages 833-840, 2006.<\/li>\r\n \t<li>J. Xu and H. Li. Adarank: a boosting algorithm for information retrieval. In SIGIR 2007, pages 391-398, 2007.<\/li>\r\n \t<li>J. Xu, T.-Y. Liu, M. Lu, H. Li, and W.-Y. Ma. Directly optimizing IR evaluation measures in learning to rank. In SIGIR 2008, pages 107-114, 2008.<\/li>\r\n \t<li>J.-Y. Yeh, J.-Y. Lin, and etc. Learning to rank for information retrieval using genetic programming. In LR4IR 2007, 2007.<\/li>\r\n \t<li>H. Yu. SVM selective sampling for ranking with application to data retrieval. In KDD 2005, pages 354-363, 2005.<\/li>\r\n \t<li>Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In SIGIR 2007, pages 271-278, 2007.<\/li>\r\n \t<li>Y. Yue and T. Joachims. Predicting diverse subsets using structural SVM. In ICML 2008, pages 1224-1231, 2008.<\/li>\r\n \t<li>C. Zhai and J. Lafferty. A risk minimization framework for information retrieval. Information Processing and Management, 42(1):31-55, 2006.<\/li>\r\n \t<li>Z. Zheng, K. Chen, G. Sun, and H. Zha. A regression framework for learning ranking functions using relative relevance judgments. In SIGIR 2007, pages 287-294, 2007.<\/li>\r\n \t<li>Z. Zheng, H. Zha, and G. Sun. Query-level learning to rank using isotonic regression. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>Z. Zheng, H. Zha, and etc. A general boosting method and its application to learning ranking functions for web search. In NIPS 2007, 2007.<\/li>\r\n \t<li>K. Zhou, G.-R. Xue, H. Zha, and Y. Yu. Learning to rank with ties. In SIGIR 2008, pages 275-282, 2008.<\/li>\r\n \t<li>O. Zoeter, M. Taylor, E. Snelson, J. Guiver, N. Craswell, and M. Szummer. A decision theoretic framework for ranking using implicit feedback. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.<\/li>\r\n \t<li>Jonathan L. Elsas, Vitor R. Carvalho, Jaime G. Carbonell. \"Fast Learning of Document Ranking Functions with the Committee Perceptron,\" Proceedings of the First ACM International Conference on Web Search and Data Mining (WSDM 2008), 2008.<\/li>\r\n \t<li>Ronan Cummins and Colm O'Riordan. An axiomatic comparison of learned term-weighting schemes in information retrieval: clarifications and extensions. Artificial Intelligence Review Journal.<\/li>\r\n \t<li>Ronan Cummins and Colm O'Riordan. Evolving local and global weighting schemes in information retrieval. Journal of Information Retrieval.<\/li>\r\n<\/ol>\r\n[\/panel]\r\n\r\n[panel header=\"Tutorials\"]\r\n<h3>Tutorial talks<\/h3>\r\n<ul>\r\n \t<li><a href=\"http:\/\/www.cse.iitb.ac.in\/%7Esoumen\/doc\/www2007\/\">WWW 2007 tutorial on Learning to rank in vector spaces and social networks<\/a>\r\nby Soumen Chakrabarti, Indian Institute of Technology Bombay<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/people\/tyliu\/learning_to_rank_tutorial_-_www_-_2008.pdf\"> WWW 2008 tutorial on learning to rank for information retrieval<\/a>\r\nby Tie-Yan Liu, Microsoft Research Asia.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/people\/tyliu\/letor-tutorial-sigir08.pdf\"> SIGIR 2008 tutorial on learning to rank for information retrieval<\/a>\r\nby Tie-Yan Liu, Microsoft Research Asia.<\/li>\r\n \t<li>NESCAI 2008 tutorial on learning to rank (<a href=\"http:\/\/www.yisongyue.com\/talks\/LearningToRank_NESCAI08_part1.ppt\">I<\/a>,<a href=\"http:\/\/radlinski.org\/paper.php?p=LearningToRank_NESCAI08.pdf\">II<\/a>)\r\nby Yisong Yue and Filip Radlinski, Cornell University.<\/li>\r\n \t<li><a href=\"http:\/\/www2009.org\/pdf\/T7A-LEARNING%20TO%20RANK%20TUTORIAL.pdf\">WWW 2009 tutorial on learning to rank for information retrieval<\/a>\r\nby Tie-Yan Liu, Microsoft Research Asia.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/events\/icml-2009-mlir-tutorial\">ICML 2009 tutorial on Machine Learning in IR: Recent Successes and New Opportunities<\/a>\r\nby Paul Bennett,Misha Bilenko, and Kevyn Collins-Thompson, Microsoft Research.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/people\/hangli\/li-acl-ijcnlp-2009-tutorial.pdf\">ACL-IJCNLP 2009 tutorial on learning to rank <\/a>\r\nby Hang Li, Microsoft Research Asia.<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/people\/hangli\/acml-tutorial.pdf\">ACML 2010 tutorial on learning to rank <\/a>\r\nby Hang Li, Microsoft Research Asia.<\/li>\r\n<\/ul>\r\n&nbsp;\r\n<h3>Tutorial papers\/books<\/h3>\r\n<ul>\r\n \t<li>Tie-Yan Liu. <a href=\"http:\/\/www.nowpublishers.com\/product.aspx?product=INR&amp;doi=1500000016\"> Learning to Rank for Information Retrieval<\/a>\r\nFoundation and Trends on Information Retrieval, Now Publishers, 2009.<\/li>\r\n<\/ul>\r\n[\/panel]\r\n\r\n[panel header=\"Events\"]\r\n<ul>\r\n \t<li><a href=\"http:\/\/learningtorankchallenge.yahoo.com\/index.php\" target=\"_blank\" rel=\"noopener noreferrer\">Learning to Rank Challenge from Yahoo! Labs<\/a>\r\nby Olivier Chapelle and Yi Chang<\/li>\r\n \t<li><a href=\"http:\/\/www.select.cs.cmu.edu\/meetings\/nips09perm\/\" target=\"_blank\" rel=\"noopener noreferrer\"> NIPS 2009 Workshop on Learning with Orderings<\/a>\r\nby Tiberio Caetano, Carlos Guestrin, Jonathan Huang, Guy Lebanon, Risi Kondor and Marina Meila<\/li>\r\n \t<li><a href=\"http:\/\/web.mit.edu\/shivani\/www\/Ranking-NIPS-09\/\" target=\"_blank\" rel=\"noopener noreferrer\"> NIPS 2009 Workshop on Advances in Ranking<\/a>\r\nby Shivani Agarwal, Chris J.C. Burges and Koby Crammer<\/li>\r\n \t<li><a href=\"http:\/\/ir.cis.udel.edu\/IDR-workshop\/\" target=\"_blank\" rel=\"noopener noreferrer\"> SIGIR 2009 Workshop on Redundancy, Diversity, and Interdependent Document Relevance (IDR '09)<\/a>\r\nby Paul N. Bennett, Ben Carterette, Thorsten Joachims and Filip Radlinski<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/events\/lr4ir-2009\/\" target=\"_blank\" rel=\"noopener noreferrer\"> SIGIR 2009 Workshop on Learning to Rank for Information Retrieval (LR4IR'09)<\/a>\r\nby Hang Li, Tie-Yan Liu and ChengXiang Zhai<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/events\/LR4IR-2008\/\" target=\"_blank\" rel=\"noopener noreferrer\"> SIGIR 2008 Workshop on Learning to Rank for Information Retrieval (LR4IR'08)<\/a>\r\nby Hang Li, Tie-Yan Liu and ChengXiang Zhai<\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/um\/beijing\/events\/LR4IR-2007\/\" target=\"_blank\" rel=\"noopener noreferrer\"> SIGIR 2007 Workshop on Learning to Rank for Information Retrieval (LR4IR'07)<\/a>\r\nby Thorsten Joachims, Hang Li, Tie-Yan Liu and ChengXiang Zhai<\/li>\r\n \t<li><a href=\"http:\/\/user.cs.tu-berlin.de\/%7Ebrefeld\/lisos\/index.html?9\" target=\"_blank\" rel=\"noopener noreferrer\">ICML 2006 Workshop on Learning in Structured Output Space<\/a>\r\nby Ulf Brefeld, Thorsten Joachims, Ben Taskar and Eric P. Xing<\/li>\r\n \t<li><a href=\"http:\/\/web.mit.edu\/shivani\/www\/Ranking-NIPS-05\/\" target=\"_blank\" rel=\"noopener noreferrer\">NIPS 2005 Workshop on Learning to Rank<\/a>\r\nby Shivani Agarwal, Corinna Cortes and Ralf Herbrich<\/li>\r\n<\/ul>\r\n[\/panel]\r\n\r\n[\/accordion]\r\n\r\n&nbsp;\r\n\r\n&nbsp;"},{"id":3,"name":"Other","content":"[accordion]\r\n[panel header=\"Community\"]\r\n<h3>Research Groups<\/h3>\r\nLearning to rank has become a hot research topics in recent years. The following research groups are very active in this field.\r\n<ul>\r\n \t<li><a href=\"http:\/\/www.cse.iitb.ac.in\/%7Esoumen\/\" target=\"_blank\" rel=\"noopener noreferrer\">Soumen Chakrabarti's group<\/a><\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/en-us\/groups\/irm\/\" target=\"_blank\" rel=\"noopener noreferrer\">Information Retrieval and Mining Group, Microsoft Research Asia<\/a><\/li>\r\n \t<li><a href=\"http:\/\/research.microsoft.com\/%7Ecburges\/\" target=\"_blank\" rel=\"noopener noreferrer\">Web Learning Group, Microsoft Research<\/a><\/li>\r\n \t<li><a href=\"http:\/\/www.cs.cornell.edu\/People\/tj\/\" target=\"_blank\" rel=\"noopener noreferrer\">Thorsten Joachims's group<\/a><\/li>\r\n \t<li><a href=\"http:\/\/www.cc.gatech.edu\/%7Ezha\/\" target=\"_blank\" rel=\"noopener noreferrer\">Hongyuan Zha's group<\/a><\/li>\r\n \t<li><a href=\"http:\/\/l2r.cs.uiuc.edu\/%7Ecogcomp\/projects.php\" target=\"_blank\" rel=\"noopener noreferrer\">Dan Roth's group<\/a><\/li>\r\n \t<li><a href=\"http:\/\/olivier.chapelle.cc\/index.html\" target=\"_blank\" rel=\"noopener noreferrer\">Olivier Chapelle's group<\/a><\/li>\r\n<\/ul>\r\nIf you want to add your own group to this list, please send email to <a href=\"mailto:letor@microsoft.com\">letor@microsoft.com<\/a> with the name of your group and a brief description.\r\n<h3><a href=\"http:\/\/learningtorank.spaces.live.com\/default.aspx\">Tech Blogs<\/a><\/h3>\r\n[\/panel]\r\n[panel header=\"Contact Us\"]\r\n<h3>LETOR Team<\/h3>\r\n<a href=\"http:\/\/research.microsoft.com\/en-us\/people\/taoqin\/\" target=\"_blank\" rel=\"noopener noreferrer\">Tao Qin<\/a>, <a href=\"http:\/\/research.microsoft.com\/users\/tyliu\/\" target=\"_blank\" rel=\"noopener noreferrer\">Tie-Yan Liu<\/a>, <a href=\"http:\/\/research.microsoft.com\/%7Ejunxu\/\" target=\"_blank\" rel=\"noopener noreferrer\">Jun Xu<\/a>, and <a href=\"http:\/\/research.microsoft.com\/users\/hangli\/\" target=\"_blank\" rel=\"noopener noreferrer\">Hang Li<\/a>.\r\n\r\nLearning to Rank Project, Microsoft Research Asia\r\n\r\nEmail: <a href=\"mailto:letor@microsoft.com\">letor@microsoft.com<\/a>\r\n<h3><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-asia\/\">MSRA<\/a><\/h3>\r\n[\/panel]\r\n[\/accordion]"}],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Tao Qin","user_id":33871,"people_section":"Section name 1","alias":"taoqin"},{"type":"user_nicename","display_name":"Tie-Yan Liu","user_id":34431,"people_section":"Section name 1","alias":"tyliu"}],"msr_research_lab":[199560],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/273699"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":5,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/273699\/revisions"}],"predecessor-version":[{"id":443451,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/273699\/revisions\/443451"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=273699"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=273699"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=273699"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=273699"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=273699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}