{"id":154712,"date":"2006-06-17T00:00:00","date_gmt":"2006-06-17T00:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/msr-research-item\/principled-hybrids-of-generative-and-discriminative-models\/"},"modified":"2018-10-16T21:22:57","modified_gmt":"2018-10-17T04:22:57","slug":"principled-hybrids-of-generative-and-discriminative-models","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/principled-hybrids-of-generative-and-discriminative-models\/","title":{"rendered":"Principled Hybrids of Generative and Discriminative Models"},"content":{"rendered":"<div class=\"asset-content\">\n<p>When labelled training data is plentiful, discriminative\u00a0techniques are widely used since they give excellent generalization\u00a0performance. However, for large-scale applications\u00a0such as object recognition, hand labelling of data\u00a0is expensive, and there is much interest in semi-supervised techniques based on generative models in which the majority\u00a0of the training data is unlabelled. Although the generalization\u00a0performance of generative models can often be\u00a0improved by \u2018training them discriminatively\u2019, they can then\u00a0no longer make use of unlabelled data. In an attempt to\u00a0gain the benefit of both generative and discriminative approaches,\u00a0heuristic procedure have been proposed [2, 3]\u00a0which interpolate between these two extremes by taking a\u00a0convex combination of the generative and discriminative\u00a0objective functions. In this paper we adopt a new perspective which says that\u00a0there is only one correct way to train a given model, and\u00a0that a \u2018discriminatively trained\u2019 generative model is fundamentally\u00a0a new model [7]. From this viewpoint, generative\u00a0and discriminative models correspond to specific choices\u00a0for the prior over\u00a0 parameters. As well as giving a principled\u00a0interpretation of \u2018discriminative training\u2019, this approach\u00a0opens door to very general ways of interpolating between\u00a0generative and discriminative extremes through alternative\u00a0choices of prior. We illustrate this framework using\u00a0both synthetic data and a practical example in the domain of\u00a0multi-class object recognition. Our results show that, when\u00a0the supply of labelled training data is limited, the optimum\u00a0performance corresponds to a balance between the purely\u00a0generative and the purely discriminative.<\/p>\n<\/div>\n<p><!-- .asset-content --><\/p>\n","protected":false},"excerpt":{"rendered":"<p>When labelled training data is plentiful, discriminative\u00a0techniques are widely used since they give excellent generalization\u00a0performance. However, for large-scale applications\u00a0such as object recognition, hand labelling of data\u00a0is expensive, and there is much interest in semi-supervised techniques based on generative models in which the majority\u00a0of the training data is unlabelled. Although the generalization\u00a0performance of generative models can [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13556],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-154712","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_publishername":"IEEE Computer Society","msr_edition":"IEEE Conference on Computer Vision and Pattern Recognition","msr_affiliation":"","msr_published_date":"2006-06-17","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"87-94","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"1","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"209035","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"file","title":"LasserreBishopMinka06.pdf","viewUrl":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/LasserreBishopMinka06.pdf","id":209035,"label_id":0}],"msr_related_uploader":"","msr_attachments":[{"id":209035,"url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/LasserreBishopMinka06.pdf"}],"msr-author-ordering":[{"type":"text","value":"Julia A. Lasserre","user_id":0,"rest_url":false},{"type":"user_nicename","value":"cmbishop","user_id":31452,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=cmbishop"},{"type":"user_nicename","value":"minka","user_id":32943,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=minka"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[],"msr_group":[],"msr_project":[],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","related_content":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/154712"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":3,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/154712\/revisions"}],"predecessor-version":[{"id":535409,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/154712\/revisions\/535409"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=154712"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=154712"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=154712"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=154712"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=154712"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=154712"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=154712"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=154712"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=154712"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=154712"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=154712"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=154712"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=154712"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=154712"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=154712"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=154712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}