{"id":840565,"date":"2022-05-24T09:15:52","date_gmt":"2022-05-24T16:15:52","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=840565"},"modified":"2022-08-16T13:47:26","modified_gmt":"2022-08-16T20:47:26","slug":"emit-less-carbon-from-ai","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/emit-less-carbon-from-ai\/","title":{"rendered":"Emit less carbon from AI"},"content":{"rendered":"\n
\"conceptual<\/figure>
\n

Multiple activities are involved in developing and using machine learning models, including selection of model architectures and algorithms, hyperparameter tuning, training on existing datasets, and making predictions on new data (aka inference). Optimizing results across these activities involves many complex problems that researchers are addressing, as described in the sections below.<\/p>\n<\/div><\/div>\n\n\n\n

Efficient model architectures and hyperparameter tuning<\/h3>\n\n\n\n

Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) are both automated optimization techniques that aim to identify promising candidates within combinatorial search spaces that are typically too large to search exhaustively. For NAS, the search is conducted through a space of potential neural network architectures; for HPO, the search is for high-performing combinations of hyperparameters. While \u201chigh-performing\u201d traditionally refers to the prediction accuracy of the resulting model, techniques like NAS and HPO can also be used to satisfy different objective functions, such as computational efficiency or cost. Thus, NAS and HPO can be useful for identifying more resource-efficient machine learning models \u2013 but it is also critical that these techniques themselves operate efficiently. Research-based meta-learning techniques (opens in new tab)<\/span><\/a>, in which an algorithm learns from experience to guide more efficient exploration of the search space, have been incorporated into Azure Machine Learning. These techniques are the basis for efficient model selection in Automated Machine Learning (opens in new tab)<\/span><\/a> and for efficient hyperparameter optimization in the HyperDrive service (opens in new tab)<\/span><\/a>. Other research approaches include Probabilistic Neural Architecture Search (PARSEC (opens in new tab)<\/span><\/a>), which uses a memory-efficient sampling procedure that requires only as much memory as is needed to train a single architecture in the search space, greatly reducing memory requirements compared to previous methods of identifying high-performing neural network architectures. Weightless PARSEC built on that result to achieve comparable results with 100x less computational cost, with implications for both embodied and emitted carbon reduction.<\/p>\n\n\n\n

\"animation<\/figure>\n\n\n\n
\n\t