{"id":1107414,"date":"2024-12-06T16:48:04","date_gmt":"2024-12-07T00:48:04","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1107414"},"modified":"2024-12-13T10:16:52","modified_gmt":"2024-12-13T18:16:52","slug":"abstracts-neurips-2024-with-weizhu-chen","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/abstracts-neurips-2024-with-weizhu-chen\/","title":{"rendered":"Abstracts: NeurIPS 2024 with Weizhu Chen"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788.jpg\" alt=\"Illustrated image of Weizhu Chen.\" class=\"wp-image-1108905\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788.jpg 1400w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/11\/Weizhu-Chen_Abstracts_Hero_Feature_No_Text_1400x788-1280x720.jpg 1280w\" sizes=\"auto, (max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n<div class=\"wp-block-msr-podcast-container my-4\">\n\t<iframe loading=\"lazy\" src=\"https:\/\/player.blubrry.com\/?podcast_id=139393524&modern=1\" class=\"podcast-player\" frameborder=\"0\" height=\"164px\" width=\"100%\" scrolling=\"no\" title=\"Podcast Player\"><\/iframe>\n<\/div>\n\n\n\n<p>Members of the research community at Microsoft work continuously to advance their respective fields. <em>Abstracts <\/em>brings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.<\/p>\n\n\n\n<p>In this episode, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/wzchen\/\" target=\"_blank\" rel=\"noreferrer noopener\">Weizhu Chen<\/a>, vice president of Microsoft GenAI, joins host Amber Tingle to discuss\u202fthe paper <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/not-all-tokens-are-what-you-need-for-pretraining\/\" target=\"_blank\" rel=\"noreferrer noopener\">\u201cNot All Tokens Are What You Need for Pretraining,\u201d<\/a> an oral presentation and a recipient of the Best Paper Runner-Up Award at this year\u2019s <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/neurips-2024\/?msockid=35739e94ab6c69d41b738b93aa076831\" target=\"_blank\" rel=\"noreferrer noopener\">Conference on Neural Information Processing Systems (NeurIPS)<\/a>. Based on an examination of model training at the token level, Chen and his coauthors present an alternate approach to model pretraining: instead of training language models to predict <em>all<\/em> tokens, they make a distinction between useful and \u201cnoisy\u201d tokens. Doing so, the work shows, improves token efficiency and model performance.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/not-all-tokens-are-what-you-need-for-pretraining\/\">Read the paper<\/a><\/div>\n\n\n\n<div class=\"wp-block-button is-style-fill-github\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/github.com\/microsoft\/rho\" target=\"_blank\" rel=\"noreferrer noopener\">Get the code<\/a><\/div>\n<\/div>\n\n\n<div class=\"wp-block-msr-show-more\">\n\t<div class=\"bg-neutral-100 p-5\">\n\t\t<div class=\"show-more-show-less\" data-mount=\"show-more-show-less\">\n\t\t\t<div>\n\t\t\t\t<span>\n\t\t\t\t\t\n\n<h2 class=\"wp-block-heading\" id=\"transcript\">Transcript<\/h2>\n\n\n\n<p>[MUSIC]<\/p>\n\n\n\n<p><strong>AMBER TINGLE:<\/strong>\u202fWelcome to\u202f<em>Abstracts<\/em>, a Microsoft Research Podcast that puts the spotlight on world-class research\u202f<em>in brief<\/em>. I\u2019m Amber Tingle. In this series, members of the research community at Microsoft give us a quick snapshot\u2014or a\u202f<em>podcast abstract<\/em>\u2014of their new and noteworthy papers.<\/p>\n\n\n\n<p>[MUSIC FADES]&nbsp;<\/p>\n\n\n\n<p>Our guest today is Weizhu Chen. He is vice president of Microsoft GenAI and coauthor of a paper called \u201cNot All Tokens Are What You Need for Pretraining.\u201d This paper is an oral presentation during the 38th annual Conference on Neural Information Processing Systems, also known as NeurIPS, which is happening this week in Vancouver. Weizhu, thank you for joining us today on <em>Abstracts<\/em>!&nbsp;<\/p>\n\n\n\n\t\t\t\t<\/span>\n\t\t\t\t<span id=\"show-more-show-less-toggle-1\" class=\"show-more-show-less-toggleable-content\">\n\t\t\t\t\t\n\n\n\n<p><strong>WEIZHU CHEN:<\/strong> Thank you for having me, Amber.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>So let&#8217;s start with a brief overview of your paper. In a couple sentences, tell us about the problem your research addresses and, more importantly, why the research community and beyond should know about <em>this<\/em> work.&nbsp;<\/p>\n\n\n\n<p><strong>CHEN:<\/strong> So my team basically in Microsoft GenAI, we are working on model training. So one of the things actually we do in the pretraining, we realize the importance of the data. And we found that actually when we do this kind of data for each of the tokens, some token is more important than the other. That&#8217;s one. The other one actually is some token actually is very, very hard to be predicted during the pretraining. So, for example, just like if someone see the text of \u201cWeizhu,\u201d and what&#8217;s the next token? It can be \u201cChen\u201d; it can be any of the last name. So it&#8217;s very hard to be predicted. And if we try to enforce a language model to focus on this, kind of, the hard-to-predict token, just like actually it&#8217;s going to confuse the language model. There are so many different kinds of the example like this. Just like, for example, the serial number in your UPS. So the focus of this paper is try to identify which token actually is more important for the language model to learn. And actually the other token maybe is just the noise. And how can we try to <em>discriminate<\/em> the token\u2014which is good token, which is noise token. Basically, you try to understand this kind of dynamic of the tokens.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>How did you conduct this research?&nbsp;<\/p>\n\n\n\n<p><strong>CHEN: <\/strong>Actually we do a lot of work in the model training, including the pretraining <em>and<\/em> the post-training. So for the pretraining side, actually the most important thing to us is the data. We also try to understand, how can we leverage the existing data, and how can we create much more data, as well? And data basically is one of the most important thing to build a better foundation model. So we try to understand how much more we can get from the data. And the important thing for the data is about data filtering. So you think about actually in the previous literature, we do the data filtering, for example, just like we build a classifier to classify, OK, this page is more important than the other. And this page actually is a noise because there&#8217;s so much noise data in the web. So we just keep the best data to get into the pretraining corpus. And further away, we think about, OK, yeah, so this is \u2026 maybe it&#8217;s not fine grain enough, so can we try to understand even for the same page we want to keep? So some token is more important than the other. Maybe some token just some noise token. Actually you put this data into the pretraining, it&#8217;s going to hurt the model quality. So there is the motivation actually we try to think about.<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>And what were your major findings?&nbsp;<\/p>\n\n\n\n<p><strong>CHEN: <\/strong>Our major finding is about basically, definitely this works so well. And it&#8217;s so important that actually we are able to get the best token from the corpus and then make it available and try to ask the model during the pretraining to ignore the token we don&#8217;t want to get into the model itself. So that is one. The second thing definitely data is the other very important thing. If you&#8217;re able to figure out the better way to build a better data is most likely you\u2019re able to build a much better foundation model. The third thing actually is also connected to a lot of other existing work, just like data synthesis, just like distillation, just like data filtering, and so a lot of things are really connected together. And actually, this work, basically, you can associate with also a lot of other work we are working on, just like distillation. You can think about, for example, for this work, we also try to build a model, a reference model\u2014we call as the reference model\u2014to try to identify actually this data, this token, is more important than the other and try to understand the discrepancy between the reference model and the running model, their prediction on each tokens. So you can think about also it&#8217;s some kind of the try to distill from the reference model to the existing model, as well.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> Let&#8217;s talk a little bit about real-world impact. Who benefits most from this work? And how significant is this within your discipline and even downstream for people using applications?&nbsp;<\/p>\n\n\n\n<p><strong>CHEN:<\/strong> This actually is very, very fundamental work because just like I share a little bit before, actually we build the data and this data is\u2014build the data much better\u2014is able to build a much better foundation model. If we&#8217;re able to build a better model actually is able to benefit so many different kinds of application. This also is going to help us to build a much better small language model. And we can also serve this model even in the edge side, in the client side, in the coding scenario. So we are going to see actually huge impact from this kind of the foundation model if you are able to benefit from building much better training data.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>Are there any unanswered questions or unsolved problems in this area? What&#8217;s next on your research agenda?&nbsp;<\/p>\n\n\n\n<p><strong>CHEN: <\/strong>Yeah, I think that is a very good questions. And definitely there&#8217;s a lot of things about how to build a better data [that] is unsolved yet in the literature. And especially because when you do the pretraining, the most important part is the data, but the data is very limited. And how can we make better use from the existing limited data is a big challenge. Because we can increase the model by 10x, but it\u2019s super hard to increase the data by 10x, especially when we want to deal with the high quality of data. The other way, even given the data, how can you identify, especially for this work, the importance of each token to build a much better model? I think all these things are very connected together. To me, actually, data is the oxygen. So there are still so many things we are able to do in the data, including building for even the small language model or the large model.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>Data is oxygen\u2014I love that! So other than <em>that<\/em> being a key takeaway, is there any other one thing that you&#8217;d like our listeners to walk away from this conversation knowing?&nbsp;<\/p>\n\n\n\n<p><strong>CHEN: <\/strong>I would love to say actually focus more on this kind of data and focus more about how can I get more from the data actually; it is the very important thing. And the other thing actually, we are working on something that&#8217;s very exciting. You can feel free to come to join us if you are very interested in this area.&nbsp;<\/p>\n\n\n\n<p>[MUSIC]&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>Well, Weizhu Chen, thank you for joining us today. We really appreciate it.&nbsp;<\/p>\n\n\n\n<p><strong>CHEN: <\/strong>Thank you. Thank you for having me.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>And thanks to our listeners for tuning in. If you\u2019d like to read the full paper, you may find a link at aka.ms\/abstracts. You can also find the paper on arXiv and on the NeurIPS conference website. I\u2019m Amber Tingle from Microsoft Research, and we hope you\u2019ll join us next time on <em>Abstracts<\/em>!&nbsp;<\/p>\n\n\n\n<p>[MUSIC FADES]&nbsp;<\/p>\n\n\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t<button\n\t\t\t\tclass=\"action-trigger glyph-prepend mt-2 mb-0 show-more-show-less-toggle\"\n\t\t\t\taria-expanded=\"false\"\n\t\t\t\tdata-show-less-text=\"Show less\"\n\t\t\t\ttype=\"button\"\n\t\t\t\taria-controls=\"show-more-show-less-toggle-1\"\n\t\t\t\taria-label=\"Show more content\"\n\t\t\t\tdata-alternate-aria-label=\"Show less content\">\n\t\t\t\tShow more\t\t\t<\/button>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team\u2019s 2024 NeurIPS paper on how distinguishing between useful and \u201cnoisy\u201d tokens in pretraining can improve token efficiency and model performance.<\/p>\n","protected":false},"author":38004,"featured_media":1110351,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/139393524","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[240054],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,269142,243990],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[268128],"class_list":["post-1107414","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river","msr-post-option-podcast-featured","msr-podcast-series-abstracts"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/139393524","podcast_episode":"","msr_research_lab":[199560],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[1088157],"related-researchers":[{"type":"user_nicename","value":"Amber Tingle","user_id":42681,"display_name":"Amber Tingle","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ambertingle\/\" aria-label=\"Visit the profile page for Amber Tingle\">Amber Tingle<\/a>","is_active":false,"last_first":"Tingle, Amber","people_section":0,"alias":"ambertingle"},{"type":"user_nicename","value":"Weizhu Chen","user_id":34863,"display_name":"Weizhu Chen","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/wzchen\/\" aria-label=\"Visit the profile page for Weizhu Chen\">Weizhu Chen<\/a>","is_active":false,"last_first":"Chen, Weizhu","people_section":0,"alias":"wzchen"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-960x540.jpg\" class=\"img-object-cover\" alt=\"Illustrated image of Weizhu Chen.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/12\/Weizhu-Chen_Abstracts_Hero_Feature_River_No_Text_1400x788.jpg 1400w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ambertingle\/\" title=\"Go to researcher profile for Amber Tingle\" aria-label=\"Go to researcher profile for Amber Tingle\" data-bi-type=\"byline author\" data-bi-cN=\"Amber Tingle\">Amber Tingle<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/wzchen\/\" title=\"Go to researcher profile for Weizhu Chen\" aria-label=\"Go to researcher profile for Weizhu Chen\" data-bi-type=\"byline author\" data-bi-cN=\"Weizhu Chen\">Weizhu Chen<\/a>","formattedDate":"December 6, 2024","formattedExcerpt":"Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team\u2019s 2024 NeurIPS paper on how distinguishing between useful and \u201cnoisy\u201d tokens in pretraining can improve token efficiency and model performance.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1107414","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38004"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1107414"}],"version-history":[{"count":15,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1107414\/revisions"}],"predecessor-version":[{"id":1112217,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1107414\/revisions\/1112217"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1110351"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1107414"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1107414"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1107414"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1107414"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1107414"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1107414"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1107414"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1107414"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1107414"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1107414"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1107414"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}