{"id":946512,"date":"2023-06-07T09:48:46","date_gmt":"2023-06-07T16:48:46","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-research-item&p=946512"},"modified":"2023-06-08T11:28:30","modified_gmt":"2023-06-08T18:28:30","slug":"statistical-learning-under-heterogenous-distribution-shift","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/statistical-learning-under-heterogenous-distribution-shift\/","title":{"rendered":"Statistical Learning under Heterogenous Distribution Shift"},"content":{"rendered":"

This paper studies the prediction of a target\u00a0z<\/span><\/span><\/span><\/span><\/span><\/span>\u00a0from a pair of random variables\u00a0(<\/span>x<\/span><\/span><\/span>,<\/span>y<\/span><\/span><\/span><\/strong>)<\/span><\/span><\/span><\/span>, where the ground-truth predictor is additive\u00a0E<\/span><\/span><\/span>[<\/span>z<\/span><\/span><\/span>\u2223<\/span>x<\/span><\/span><\/span>,<\/span>y<\/span><\/span><\/span><\/strong>]<\/span>=<\/span>f<\/span><\/em>\u22c6<\/span><\/span>(<\/span>x<\/span><\/span><\/span><\/strong>)<\/span>+<\/span>g<\/span><\/em>\u22c6<\/span><\/span><\/span><\/span>(<\/span>y<\/span><\/span><\/span><\/strong>)<\/span><\/span><\/span><\/span>. We study the performance of empirical risk minimization (ERM) over functions\u00a0f<\/span><\/em>+<\/span>g<\/span><\/em><\/span><\/span><\/span>,\u00a0f<\/span><\/em>\u2208<\/span>F<\/span><\/span><\/span><\/em><\/span><\/span><\/span>\u00a0and\u00a0g<\/span><\/em>\u2208<\/span>G<\/span><\/span><\/span><\/em><\/span><\/span><\/span>, fit on a given training distribution, but evaluated on a test distribution which exhibits covariate shift. We show that, when the class\u00a0F<\/span><\/span><\/span><\/span><\/span><\/span><\/em>\u00a0is “simpler” than\u00a0G<\/span><\/span><\/span><\/span><\/span><\/span><\/em>\u00a0(measured, e.g., in terms of its metric entropy), our predictor is more resilient to \\emph{heterogenous covariate shifts} in which the shift in\u00a0x<\/span><\/span><\/span><\/span><\/span><\/span><\/strong>\u00a0is much greater than that in\u00a0y<\/span><\/span><\/span><\/span><\/span><\/span><\/strong>. These results rely on a novel H\u00f6lder style inequality for the Dudley integral which may be of independent interest. Moreover, we corroborate our theoretical findings with experiments demonstrating improved resilience to shifts in “simpler” features across numerous domains.<\/p>\n","protected":false},"excerpt":{"rendered":"

This paper studies the prediction of a target\u00a0z\u00a0from a pair of random variables\u00a0(x,y), where the ground-truth predictor is additive\u00a0E[z\u2223x,y]=f\u22c6(x)+g\u22c6(y). We study the performance of empirical risk minimization (ERM) over functions\u00a0f+g,\u00a0f\u2208F\u00a0and\u00a0g\u2208G, fit on a given training distribution, but evaluated on a test distribution which exhibits covariate shift. We show that, when the class\u00a0F\u00a0is “simpler” than\u00a0G\u00a0(measured, e.g., […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13556],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246685],"msr-conference":[260284],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-946512","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-field-of-study-machine-learning"],"msr_publishername":"","msr_edition":"","msr_affiliation":"","msr_published_date":"2023-2-27","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"url","viewUrl":"false","id":"false","title":"https:\/\/arxiv.org\/abs\/2302.13934","label_id":"243109","label":0}],"msr_related_uploader":"","msr_attachments":[],"msr-author-ordering":[{"type":"text","value":"Max Simchowitz","user_id":0,"rest_url":false},{"type":"text","value":"Anurag Ajay","user_id":0,"rest_url":false},{"type":"text","value":"Pulkit Agrawal","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Akshay Krishnamurthy","user_id":30913,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Akshay Krishnamurthy"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[945816],"msr_group":[144902],"msr_project":[],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","related_content":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/946512"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/946512\/revisions"}],"predecessor-version":[{"id":946521,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/946512\/revisions\/946521"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=946512"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=946512"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=946512"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=946512"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=946512"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=946512"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=946512"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=946512"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=946512"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=946512"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=946512"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=946512"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=946512"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=946512"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=946512"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=946512"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}