![\"Discovering](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/11\/NIPS_Neural-Architecture-Optimization_NAO_Site_1400x788-1024x576.png\")
<\/p>\n<\/p>\n
If you\u2019re a deep learning practitioner, you may find yourself faced with the same critical question on a regular basis: Which neural network architecture should I choose for my current task? The decision depends on a variety of factors and the answers to a number of other questions. What operations should I choose for this layer\u2014convolution, depth separable convolution, or max pooling? What is the kernel size for convolution? 3×3 or 1×1? And which previous node should serve as the input to the current recurrent neural network (RNN) cell? Such decisions are crucial to the architecture\u2019s success. If you\u2019re a domain expert in both neural network modeling and the specific task at hand, it might be easy for you to make the correct decisions. But what if your experience with either of them is limited?<\/p>\n
In that case, you might turn to neural architecture search (NAS), an automated process in which an additional machine learning algorithm is leveraged to guide the creation of better neural architecture given the historically observed architectures and their performances. Thanks to NAS, we can pinpoint the neural network architectures that achieve the best results on widely used benchmark datasets, such as ImageNet, without any human intervention.<\/p>\n
But while existing methods in automatic neural architecture design\u2014typically based on reinforcement learning or evolutionary algorithm\u2014have generally conducted their searches in the exponentially large discrete space, my collaborators and I in the Machine Learning group at Microsoft Research Asia (opens in new tab)<\/span><\/a> have designed a simplified, more efficient method based on optimization in the continuous space. With our new approach, called neural architecture optimization (NAO) (opens in new tab)<\/span><\/a>, we leverage the power of a gradient-based method to conduct optimization in the more compact space. The work is part of the program at this year\u2019s Conference on Neural Information Processing Systems (NeurIPS) (opens in new tab)<\/span><\/a>.<\/p>\n Figure 1: The workflow of NAO<\/p><\/div>\n Driving NAO\u2019s ability to perform gradient-based optimization in the continuous space are three components (see Figure 1):<\/p>\n These three components are trained jointly. After we have finished training, starting from an architecture x, we map it using encoder E to its representation ex<\/sub>, then move ex<\/sub> to a new embedding vector, denoted as ex’<\/sub>, along the gradient direction of the performance prediction function f (the green line). Since we are essentially doing gradient ascent, we are guaranteed that f(ex’<\/sub>) \u2265 f(ex<\/sub>) as long as the step size is small enough. Finally, we map ex’<\/sub> into a discrete architecture x’ using the decoder D. In this way, we obtain a potentially better architecture x’. By iteratively updating architectures in this way, we obtain the final architecture, which is assumed to provide the best performance.<\/span><\/p>\n We conducted extensive experiments to verify the effectiveness of using NAO to automatically discover the best neural architecture. Table 1 (below) demonstrates the results on the CIFAR-10 image classification dataset using various convolutional neural network (CNN) architectures discovered by different NAS algorithms. From the table, we can observe that the neural network discovered by NAO achieves the lowest error rate among the studied NAS algorithms. Furthermore, we can achieve significant search speed when combining NAO with the weight sharing mechanism (opens in new tab)<\/span><\/a> (referred to as \u201cNAO-WS\u201d), a method used to significantly reduce the computational cost of obtaining the performances of various neural network architectures by letting them share the same copy of weight parameters. In our experiments, we found we can use one graphics processing unit (GPU) in seven hours to obtain a CNN architecture achieving a 3.53 error rate. With weight sharing, there is no need to train different neural networks from scratch.<\/p>\n Table 2 (below) summarizes the results on PTB language modeling. The lower perplexity scores indicate better performance. Again, we can observe that the RNN architecture found by our NAO achieves impressive performance with very limited computational resources.<\/p>\n By using continuous optimization, NAO achieves better performance compared to existing NAS methods directly searching among discrete architecture space. As for future application, we plan to use NAO to search for architecture for other important AI tasks, such as neural machine translation. And just as important, the availability of a simpler, more efficient automatic neural architecture design continues to make machine learning technologies accessible to practitioners of all experience levels.<\/p>\n Table 1: Results on CIFAR-10 classification<\/p>\n Table 2: Results on PTB language modeling<\/p>\n","protected":false},"excerpt":{"rendered":" If you\u2019re a deep learning practitioner, you may find yourself faced with the same critical question on a regular basis: Which neural network architecture should I choose for my current task? The decision depends on a variety of factors and the answers to a number of other questions. What operations should I choose for this […]<\/p>\n","protected":false},"author":37074,"featured_media":552897,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[241770],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-552894","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199560],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[269241],"related-projects":[272661],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"![\"Figure](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/11\/Figure-1_The-workflow-of-NAO.png\")
The key components of NAO<\/h3>\n
\n
High performance with limited computational resources<\/h3>\n
\n
\n Method<\/strong><\/td>\n Error Rate<\/strong><\/td>\n Resource (#GPU x <\/strong>#Hours)<\/strong><\/td>\n<\/tr>\n \n ENAS (opens in new tab)<\/span><\/a><\/td>\n 3.54<\/td>\n 11<\/td>\n<\/tr>\n \n NAO-WS<\/strong><\/td>\n 3.53<\/strong><\/td>\n 7<\/strong><\/td>\n<\/tr>\n \n AmoebaNet (opens in new tab)<\/span><\/a><\/td>\n 2.13<\/td>\n 3150 * 24<\/td>\n<\/tr>\n \n Hier-EA (opens in new tab)<\/span><\/a><\/td>\n 3.75<\/td>\n 300 * 24<\/td>\n<\/tr>\n \n NAO<\/strong><\/td>\n 2.11<\/strong><\/td>\n 200 * 24<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n
\n Method<\/strong><\/td>\n Perplexity<\/strong><\/td>\n Resource (#GPU x<\/strong> #Hours)<\/strong><\/td>\n<\/tr>\n \n NASNet (opens in new tab)<\/span><\/a><\/td>\n 62.4<\/td>\n 1e4 CPU days<\/td>\n<\/tr>\n \n ENAS (opens in new tab)<\/span><\/a><\/td>\n 58.6<\/td>\n 12<\/td>\n<\/tr>\n \n DARTS (opens in new tab)<\/span><\/a><\/td>\n 56.1<\/td>\n 24<\/td>\n<\/tr>\n \n NAO<\/strong><\/td>\n 56.0<\/strong><\/td>\n 300 * 24<\/strong><\/td>\n<\/tr>\n \n NAO-WS<\/strong><\/td>\n 56.6<\/strong><\/td>\n 10<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n ![\"Discovering](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/11\/NIPS_Neural-Architecture-Optimization_NAO_Site_1400x788.png\")
","byline":"Fei Tian","formattedDate":"November 30, 2018","formattedExcerpt":"If you\u2019re a deep learning practitioner, you may find yourself faced with the same critical question on a regular basis: Which neural network architecture should I choose for my current task? The decision depends on a variety of factors and the answers to a number…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/552894"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37074"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=552894"}],"version-history":[{"count":30,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/552894\/revisions"}],"predecessor-version":[{"id":554352,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/552894\/revisions\/554352"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/552897"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=552894"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=552894"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=552894"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=552894"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=552894"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=552894"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=552894"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=552894"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=552894"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=552894"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=552894"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}