{"id":664110,"date":"2020-07-22T11:37:44","date_gmt":"2020-07-22T18:37:44","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-event&p=664110"},"modified":"2022-08-02T12:30:08","modified_gmt":"2022-08-02T19:30:08","slug":"directions-in-ml","status":"publish","type":"msr-event","link":"https:\/\/www.microsoft.com\/en-us\/research\/event\/directions-in-ml\/","title":{"rendered":"Directions in ML: AutoML and Automating Algorithms"},"content":{"rendered":"\n\n\n\n\n

The Directions in ML: AutoML and Automating Algorithms speaker series brought together AutoML research communities from across the world to learn from each other and build on the latest knowledge and exploration. This virtual speaker series featured talks by outstanding academics and domain experts whose research covered a broad swath of the topic. The live talks were presented virtually from 10 to 11 AM pacific on the day of and the recorded sessions are now available for on demand viewing. <\/p>\n\n\n\n

Learn more about AutoML research and team at Microsoft Research ><\/a><\/p>\n\n\n\n

<\/div>\n\n\n\n

Speaker lineup<\/h2>\n\n\n\n
Date<\/th>Speaker<\/th><\/tr><\/thead>
July 28, 2020<\/td>Professor Ameet Talwalkar, Carnegie Mellon University<\/td><\/tr>
September 9, 2020<\/td>Professor Frank Hutter, University of Freiburg<\/td><\/tr>
October 6, 2020<\/td>Professor Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute<\/td><\/tr>
November 18, 2020<\/td>Dr. David Alvarez-Melis, Microsoft Research<\/td><\/tr>
December 15, 2020<\/td>Professor Yisong Yue, California Institute of Technology<\/td><\/tr>
January 27, 2021<\/td>Professor Sanmi Koyejo, University of Illinois at Urbana-Champaign<\/td><\/tr>
February 25, 2021<\/td>Professor Ryan Adams, Princeton University<\/td><\/tr>
March 31, 2021<\/td>Professor Roger Grosse, University of Toronto<\/td><\/tr>
April 28, 2021<\/td>Dr. Lester Mackey, Microsoft Research<\/td><\/tr>
May 25, 2021<\/td>Professor David Duvenaud, University of Toronto<\/td><\/tr>
June 29, 2021<\/td>Professor Madeleine Udell, Cornell University<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Microsoft\u2019s Event Code of Conduct<\/h3>\n\n\n\n

Microsoft\u2019s mission is to empower every person and every organization on the planet to achieve more. This includes virtual events Microsoft hosts and participates in, where we seek to create a respectful, friendly, and inclusive experience for all participants. As such, we do not tolerate harassing or disrespectful behavior, messages, images, or interactions by any event participant, in any form, at any aspect of the program including business and social activities, regardless of location. We do not tolerate any behavior that is degrading to any gender, race, sexual orientation or disability, or any behavior that would violate Microsoft\u2019s Anti-Harassment and Anti-Discrimination Policy, Equal Employment Opportunity Policy, or Standards of Business Conduct (opens in new tab)<\/span><\/a>. In short, the entire experience must meet our culture standards. We encourage everyone to assist in creating a welcoming and safe environment. Please report (opens in new tab)<\/span><\/a> any concerns, harassing behavior, or suspicious or disruptive activity. Microsoft reserves the right to ask attendees to leave at any time at its sole discretion.<\/p>\n\n\n\n

\n
Report a concern<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n\n\n

Structured Models for Automated Machine Learning<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Madeleine Udell, Cornell University
Date:<\/strong> June 29, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n
Watch on demand<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Automated machine learning (AutoML) seeks algorithmic methods for finding the best machine learning pipeline and hyperparameters to fit a new dataset. The complexity of this problem is astounding: viewed as an optimization problem, it entails search over an exponentially large space, with discrete and continuous variables. An efficient solution requires a strong structural prior on the optimization landscape of this problem.<\/p>\n\n\n\n

In this talk, we survey some of the most powerful techniques for AutoML on tabular datasets. We will focus in particular on techniques for meta-learning: how to quickly learn good models on a new dataset given good models for a large collection of datasets. We will see that remarkably simple structural priors, such as the low-dimensional structure used by the AutoML method Oboe, produce state-of-the-art results. The success of these simple models suggests that AutoML may be simpler than was previously understood.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Madeleine Udell is Assistant Professor of Operations Research and Information Engineering and Richard and Sybil Smith Sesquicentennial Fellow at Cornell University. She studies optimization and machine learning for large scale data analysis and control, with applications in marketing, demographic modeling, medical informatics, engineering system design, and automated machine learning. Her research in optimization centers on detecting and exploiting novel structures in optimization problems, with a particular focus on convex and low rank problems. These structures lead the way to automatic proofs of optimality, better complexity guarantees, and faster, more memory-efficient algorithms. She has developed a number of open source libraries for modeling and solving optimization problems, including Convex.jl, one of the top tools in the Julia language for technical computing.<\/p>\n\n\n\n

\n\n\n\n

Latent Stochastic Differential Equations: An Unexplored Model Class<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Professor David Duvenaud, University of Toronto
\nDate:<\/strong> May 25, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n
Watch on demand<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

We show how to do gradient-based stochastic variational inference in stochastic differential equations (SDEs), in a way that allows the use of adaptive SDE solvers. This allows us to scalably fit a new family of richly-parameterized distributions over irregularly-sampled time series. We apply latent SDEs to motion capture data, and to demonstrate infinitely-deep Bayesian neural networks. We also discuss the pros and cons of this barely-explored model class, comparing it to Gaussian processes and neural processes.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

David Duvenaud is an assistant professor in computer science at the University of Toronto. His research focuses on continuous-time models, latent-variable models, and deep learning. His postdoc was done at Harvard University, and his Ph.D. at the University of Cambridge. David is a founding member of the Vector Institute for Artificial Intelligence, and also co-founded Invenia, an energy forecasting company.<\/p>\n\n\n\n

\n\n\n\n

Knowledge Distillation as Semiparametric Inference<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Dr. Lester Mackey, Microsoft Research
\nDate:<\/strong> April 28, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory- constrained devices. Knowledge distillation alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements\u2014cross-fitting and loss correction\u2014to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Lester is a statistical machine learning researcher at Microsoft Research New England and an adjunct professor at Stanford University. He received his Ph.D. in Computer Science (2012), his M.A. in Statistics (2011) from UC Berkeley, and his B.S.E. in Computer Science (2007) from Princeton University. Before joining Microsoft, Lester spent three wonderful years as an assistant professor of Statistics and, by courtesy, Computer Science at Stanford and one as a Simons Math+X postdoctoral fellow, working with Emmanuel Candes. Lester\u2019s Ph.D. advisor was Mike Jordan, and his undergraduate research advisors were Maria Klawe and David Walker. He got his first taste of research at the Research Science Institute and learned to think deeply of simple things at the Ross Program. Lester\u2019s current research interests include statistical machine learning, scalable algorithms, high-dimensional statistics, approximate inference, and probability. Lately, he\u2019s been developing and analyzing scalable learning algorithms for healthcare, climate forecasting, approximate posterior inference, high-energy physics, recommender systems, and the social good.<\/p>\n\n\n\n

\n\n\n\n

Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Roger Grosse, University of Toronto
\nDate:<\/strong> March 31, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Optimization of many deep learning hyperparameters can be formulated as a bilevel optimization problem. While most black-box and gradient-based approaches require many independent training runs, we aim to adapt hyperparameters online as the network trains. The main challenge is to approximate the response Jacobian, which captures how the minimum of the inner objective changes as the hyperparameters are perturbed. To do this, we introduce the self-tuning network (STN), which fits a hypernetwork to approximate the best response function in the vicinity of the current hyperparameters. Differentiating through the hypernetwork lets us efficiently approximate the gradient of the validation loss with respect to the hyperparameters. We train the hypernetwork and hyperparameters jointly. Empirically, we can find hyperparameter settings competitive with Bayesian Optimization in a single run of training, and in some cases find hyperparameter schedules that outperform any fixed hyperparameter value.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Roger Grosse is an Assistant Professor of Computer Science at the University of Toronto, and a founding member of the Vector Institute for Artificial Intelligence. He received his Ph.D. in computer science from MIT, and then spent two years as a postdoc at the University of Toronto. He holds a Canada Research Chair in Probabilistic Inference and Deep Learning, an Ontario MRIS Early Researcher Award, and a CIFAR Canadian AI Chair.<\/p>\n\n\n\n

\n\n\n\n

Taking Advantage of Randomness in Expensive Optimization Problems<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Ryan Adams, Princeton University
\nDate:<\/strong> February 25, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Optimization is at the heart of machine learning, and gradient computation is central to many optimization techniques. Stochastic optimization, in particular, has taken center stage as the principal method of fitting many models, from deep neural networks to variational Bayesian posterior approximations. Generally, one uses data subsampling to efficiently construct unbiased gradient estimators for stochastic optimization, but this is only one possibility. In this talk, I will discuss two alternative approaches to constructing unbiased gradient estimates in machine learning problems. The first approach uses randomized truncation of objective functions defined as loops or limits. Such objectives arise in settings ranging from hyperparameter selection, to fitting parameters of differential equations, to variational inference using lower bounds on the log-marginal likelihood. The second approach revisits the Jacobian accumulation problem at the heart of automatic differentiation, observing that it is possible to collapse the linearized computational graph of, e.g., deep neural networks, in a randomized way such that less memory is used but little performance is lost. These projects are joint work with students Alex Beatson, Deniz Oktay, Joshua Aduol, and Nick McGreivy.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Ryan Adams is interested in machine learning, artificial intelligence, and computational statistics, with applications across science and engineering. Ryan has broad interests but often works on probabilistic methods and approximate Bayesian inference. Ryan is the director of the Undergraduate Certificate in Statistics and Machine Learning (opens in new tab)<\/span><\/a>. He co-founded Whetlab (sold to Twitter in 2015) and formerly co-hosted the Talking Machines (opens in new tab)<\/span><\/a> podcast. Ryan was faculty at Harvard from 2011 to 2016 and was at Twitter and then Google Brain before joining the faculty at Princeton in 2018. He calls his group the Laboratory for Intelligent Probabilistic Systems (LIPS) (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

\n\n\n\n

Automating ML Performance Metric Selection<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Sanmi Koyejo, University of Illinois at Urbana-Champaign
\nDate:<\/strong> January 27, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

From music recommendations to high-stakes medical treatment selection, complex decision-making tasks are increasingly automated as classification problems.  Thus, there is a growing need for classifiers that accurately reflect complex decision-making goals. One often formalizes these learning goals via a performance metric, which, in turn, can be used to evaluate and compare classifiers. Yet, choosing the appropriate metric remains a challenging problem. This talk will outline metric elicitation as a formal strategy to address the metric selection problem. Metric elicitation automates the discovery of implicit preferences from an expert or an expert panel using relatively efficient and straightforward interactive queries. Beyond standard classification settings, I will also outline early work on metric selection for group-fair classification.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Sanmi (Oluwasanmi) Koyejo is an Assistant Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. Koyejo\u2019s research interests are in developing the principles and practice of adaptive and robust machine learning. Additionally, Koyejo focuses on applications to neuroscience and biomedical imaging. Koyejo completed his Ph.D. in Electrical Engineering at the University of Texas at Austin advised by Joydeep Ghosh and completed postdoctoral research at Stanford University with a focus on developing machine learning techniques for neuroimaging data. His postdoctoral research was primarily with Russell A. Poldrack and Pradeep Ravikumar. Koyejo has been the recipient of several awards including a best paper award from the conference on uncertainty in artificial intelligence (UAI), a Kavli Fellowship, an IJCAI early career spotlight, and a trainee award from the Organization for Human Brain Mapping (OHBM). Koyejo serves on the board of the Black in AI organization.<\/p>\n\n\n\n

\n\n\n\n

AI for Adaptive Experiment Design<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Yisong Yue, California Institute of Technology
\nDate:<\/strong> December 15, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Experiment design is hallmark of virtually all research disciplines. In many settings, one important challenge is how to automatically design experiments over large action\/design spaces. Furthermore, it is also important for such a procedure to be adaptive, i.e., to adapt to the outcomes of previous experiments. In this talk, I will describe recent progress in using data-driven algorithmic techniques for adaptive experiment design, also known as active learning and Bayesian optimization in the machine learning community. Building upon the Gaussian process (GP) framework, I will describe case studies in personalized clinical therapy and nanophotonic structure design. Motivated by these applications, I will show how to incorporate real-world considerations such as safety, preference elicitation, and multi-fidelity experiment design into the GP framework, with new algorithms, theoretical guarantees, and empirical validation. Time permitting, I will also briefly overview a few other case studies as well.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Yisong Yue is a professor of\u202fComputing and Mathematical Sciences (opens in new tab)<\/span><\/a>\u202fat the California Institute of Technology. He was previously a research scientist at\u202fDisney Research (opens in new tab)<\/span><\/a>. Before that, he was a postdoctoral researcher in the\u202fMachine Learning Department (opens in new tab)<\/span><\/a>\u202fand the\u202fiLab (opens in new tab)<\/span><\/a>\u202fat Carnegie Mellon University. He received a Ph.D. from\u202fCornell University (opens in new tab)<\/span><\/a>\u202fand a B.S. from the\u202fUniversity of Illinois at Urbana-Champaign (opens in new tab)<\/span><\/a>. Yisong\u2019s research interests are centered around machine learning, and in particular getting theory to work in practice. To that end, his research agenda spans both fundamental research and in-depth collaborations with machine learning practitioners. In the past, his research has been applied to information retrieval, recommender systems, text classification, learning from rich user interfaces, analyzing implicit human feedback, data-driven animation, behavior analysis, sports analytics, experiment design for science, protein engineering, program synthesis, learning-accelerated optimization, robotics, and adaptive planning & allocation problems.<\/p>\n\n\n\n

\n\n\n\n

Automating Dataset Comparison and Manipulation with Optimal Transport<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Dr. David Alvarez-Melis, Microsoft Research
Date:<\/strong> November 18, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Machine learning research has traditionally been model-centric, focusing on architectures, parameter optimization, \u202fand model transfer. Much less attention has been given to the datasets on which these models are trained, which are often assumed to be fixed, or subject to extrinsic and inevitable change. However, successful application of ML in practice often requires substantial effort in terms of dataset preprocessing and manipulation, such as augmenting, merging, mixing, or reducing datasets. In this talk I will present some of our recent work that seeks to formalize and automatize these and other flavors of dataset manipulation under a unified approach. First, I will introduce the Optimal Transport Dataset Distance, which provides a fundamental theoretical building block: a formal notion of similarity between labeled datasets. In the second part of the talk, I will discuss how this notion of distance can be used to formulate a general framework of dataset optimization by means of gradient flows in probability space. I will end by presenting various exciting potential applications of this dataset optimization framework.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

David Alvarez-Melis is a postdoctoral researcher at Microsoft Research, New England, and a research affiliate at MIT. He obtained a PhD in computer science from the MIT Computer Science and Artificial Intelligence Lab in 2019, where his thesis work spanned various topics in machine learning and natural language processing. He also holds BSc and MS degrees in mathematics from ITAM and Courant Institute (NYU), respectively. He has previously spent time at IBM Research, and is a recipient of CONACYT, Hewlett Packard and AI2 awards. David\u2019s current research sits at the intersection of machine learning, optimization and applied mathematics, with a focus on optimal transport and its applications to dataset processing, transfer learning, and generative modeling.<\/p>\n\n\n\n

\n\n\n\n

AutoML and Interpretability: Powering the machine learning revolution in healthcare<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute
Date:<\/strong> October 6, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

AutoML and interpretability are both fundamental to the successful uptake of machine learning by non-expert end users. The former will lower barriers to entry and unlock potent new capabilities that are out of reach when working with ad-hoc models, while the latter will ensure that outputs are transparent, trustworthy, and meaningful. In healthcare, AutoML and interpretability are already beginning to empower the clinical community by enabling the crafting of actionable analytics that can inform and improve decision-making by clinicians, administrators, researchers, policymakers, and beyond. This talk presents state-of-the-art AutoML and interpretability methods for healthcare developed in our lab and how they have been applied in various clinical settings (including cancer, cardiovascular disease, cystic fibrosis, and recently Covid-19), and then explains how these approaches form part of a broader vision for the future of machine learning in healthcare.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Mihaela van der Schaar (opens in new tab)<\/span><\/a> is the John Humphrey Plummer (opens in new tab)<\/span><\/a> Professor of Machine Learning, Artificial Intelligence and Medicine at the University of Cambridge, a Fellow (opens in new tab)<\/span><\/a> at The Alan Turing Institute in London, and a Chancellor\u2019s Professor at UCLA (opens in new tab)<\/span><\/a>. Mihaela was elected IEEE Fellow in 2009. She has received numerous awards, including the Oon Prize on Preventative Medicine (opens in new tab)<\/span><\/a> from the University of Cambridge (2018), a National Science Foundation CAREER Award (2004), 3 IBM Faculty Awards, the IBM Exploratory Stream Analytics Innovation Award, the Philips Make a Difference Award and several best paper awards, including the IEEE Darlington Award. Mihaela\u2019s work has also led to 35 USA patents (many widely cited and adopted in standards) and 45+ contributions to international standards for which she received 3 International ISO (International Organization for Standardization) Awards. In 2019, she was identified by National Endowment for Science, Technology and the Arts as the most-cited female AI researcher in the UK (opens in new tab)<\/span><\/a>. She was also elected as a 2019 \u201cStar in Computer Networking and Communications (opens in new tab)<\/span><\/a>\u201d by N\u00b2Women. Her research expertise span signal and image processing, communication networks, network science, multimedia, game theory, distributed systems, machine learning and AI. Mihaela\u2019s current research focus is on machine learning, AI and operations research for healthcare and medicine.<\/p>\n\n\n\n

\n\n\n\n

Neural architecture search: Coming of age<\/h3>\n\n\n\n
\"Head<\/figure>\n\n\n\n

Speaker:<\/strong> Frank Hutter, University of Freiburg
\nDate:<\/strong> September 9, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Neural Architecture Search (NAS) is a very promising but still young field. I will start this talk by discussing various works aiming to build a scientific community around NAS, including benchmarks, best practices, and open source frameworks. Then, I will discuss several exciting directions for the field: (1) a broad range of possible speedup techniques for NAS; (2) joint NAS + hyperparameter optimization in Auto-PyTorch to allow off-the-shelf AutoML; and (3) the extended problem definition of neural ensemble search (NES) that searches for a set of complementary architectures rather than a single one as in NAS.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Frank Hutter (opens in new tab)<\/span><\/a> is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), as well as Chief Expert AutoML at the Bosch Center for Artificial Intelligence (opens in new tab)<\/span><\/a>. Frank holds a PhD from the University of British Columbia (2009) and a MSc from TU Darmstadt (2004). He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, and with his coauthors, several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. He is the recipient of a 2013 Emmy Noether Fellowship, a 2016 ERC Starting Grant, a 2018 Google Faculty Research Award, and a 2020 ERC PoC Award (opens in new tab)<\/span><\/a>. He is also a Fellow of ELLIS (opens in new tab)<\/span><\/a> and Program Chair at ECML 2020. In the field of AutoML, Frank co-founded the ICML workshop series on AutoML (opens in new tab)<\/span><\/a> in 2014 and has co-organized it every year since, co-authored the prominent AutoML tools Auto-WEKA (opens in new tab)<\/span><\/a> and Auto-sklearn (opens in new tab)<\/span><\/a>, won the first two AutoML challenges with his team, co-authored the first book on AutoML, worked extensively on efficient hyperparameter optimization and neural architecture search, and gave a NeurIPS 2018 tutorial with over 3000 attendees.<\/p>\n\n\n\n

\n\n\n\n

Algorithmic foundations of neural architecture search<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Ameet Talwalkar, Carnegie Mellon University
\nDate:<\/strong> July 28, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Neural architecture search (NAS)\u2014the problem of selecting which neural model to use for your learning problem\u2014is a promising direction for automating and democratizing machine learning. Early NAS methods achieved impressive results on canonical image classification and language modeling problems, yet these methods were algorithmically complex and massively expensive computationally. More recent heuristics relying on weight-sharing and gradient-based optimization are drastically more computationally efficient while also achieving state-of-the-art performance. However, these heuristics are also complex, are poorly understood, and have recently come under scrutiny because of inconsistent results on new benchmarks and poor performance as a surrogate for fully trained models. In this talk, we introduce the NAS problem and then present our work studying recent NAS heuristics from first principles. We first perform an extensive ablation study to identify the necessary components of leading NAS methods. We next introduce our geometry-aware framework called GAEA, which exploits the underlying structure of the weight-sharing NAS optimization problem to quickly find high-performance architectures. This leads to simple yet novel algorithms that enjoy faster convergence guarantees than existing gradient-based methods and achieve state-of-the-art accuracy on a wide range of leading NAS benchmarks. Together, our theory and experiments demonstrate a principled way to co-design optimizers and continuous parameterizations of discrete NAS search spaces.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Ameet Talwalkar (opens in new tab)<\/span><\/a> is an assistant professor in the machine learning department at Carnegie Mellon University and is also co-founder and chief scientist at Determined AI. His interests are in the field of statistical machine learning. His current work is motivated by the goal of democratizing machine learning and focuses on topics related to scalability, automation, fairness, and interpretability of learning algorithms and systems.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"

Join us as we bring together AutoML research communities from across the world to learn from each other and building on the latest knowledge and exploration. This speaker series features talks by outstanding academics and domain experts whose research covers a broad swath of the topic.<\/p>\n","protected":false},"featured_media":674697,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr_startdate":"2020-07-28","msr_enddate":"2021-06-29","msr_location":"Virtual","msr_expirationdate":"","msr_event_recording_link":"","msr_event_link":"","msr_event_link_redirect":false,"msr_event_time":"10:00 AM\u201311:00 AM PT","msr_hide_region":false,"msr_private_event":false,"footnotes":""},"research-area":[13556],"msr-region":[256048],"msr-event-type":[197944],"msr-video-type":[],"msr-locale":[268875],"msr-program-audience":[],"msr-post-option":[],"msr-impact-theme":[],"class_list":["post-664110","msr-event","type-msr-event","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-region-global","msr-event-type-hosted-by-microsoft","msr-locale-en_us"],"msr_about":"\n\n\n\n\n

The Directions in ML: AutoML and Automating Algorithms speaker series brought together AutoML research communities from across the world to learn from each other and build on the latest knowledge and exploration. This virtual speaker series featured talks by outstanding academics and domain experts whose research covered a broad swath of the topic. The live talks were presented virtually from 10 to 11 AM pacific on the day of and the recorded sessions are now available for on demand viewing. <\/p>\n\n\n\n

Learn more about AutoML research and team at Microsoft Research ><\/a><\/p>\n\n\n\n

<\/div>\n\n\n\n

Speaker lineup<\/h2>\n\n\n\n
Date<\/th>Speaker<\/th><\/tr><\/thead>
July 28, 2020<\/td>Professor Ameet Talwalkar, Carnegie Mellon University<\/td><\/tr>
September 9, 2020<\/td>Professor Frank Hutter, University of Freiburg<\/td><\/tr>
October 6, 2020<\/td>Professor Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute<\/td><\/tr>
November 18, 2020<\/td>Dr. David Alvarez-Melis, Microsoft Research<\/td><\/tr>
December 15, 2020<\/td>Professor Yisong Yue, California Institute of Technology<\/td><\/tr>
January 27, 2021<\/td>Professor Sanmi Koyejo, University of Illinois at Urbana-Champaign<\/td><\/tr>
February 25, 2021<\/td>Professor Ryan Adams, Princeton University<\/td><\/tr>
March 31, 2021<\/td>Professor Roger Grosse, University of Toronto<\/td><\/tr>
April 28, 2021<\/td>Dr. Lester Mackey, Microsoft Research<\/td><\/tr>
May 25, 2021<\/td>Professor David Duvenaud, University of Toronto<\/td><\/tr>
June 29, 2021<\/td>Professor Madeleine Udell, Cornell University<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Microsoft\u2019s Event Code of Conduct<\/h3>\n\n\n\n

Microsoft\u2019s mission is to empower every person and every organization on the planet to achieve more. This includes virtual events Microsoft hosts and participates in, where we seek to create a respectful, friendly, and inclusive experience for all participants. As such, we do not tolerate harassing or disrespectful behavior, messages, images, or interactions by any event participant, in any form, at any aspect of the program including business and social activities, regardless of location. We do not tolerate any behavior that is degrading to any gender, race, sexual orientation or disability, or any behavior that would violate Microsoft\u2019s Anti-Harassment and Anti-Discrimination Policy, Equal Employment Opportunity Policy, or Standards of Business Conduct<\/a>. In short, the entire experience must meet our culture standards. We encourage everyone to assist in creating a welcoming and safe environment. Please report<\/a> any concerns, harassing behavior, or suspicious or disruptive activity. Microsoft reserves the right to ask attendees to leave at any time at its sole discretion.<\/p>\n\n\n\n

\n
Report a concern<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n\n\n

Structured Models for Automated Machine Learning<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Madeleine Udell, Cornell University
Date:<\/strong> June 29, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n
Watch on demand<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Automated machine learning (AutoML) seeks algorithmic methods for finding the best machine learning pipeline and hyperparameters to fit a new dataset. The complexity of this problem is astounding: viewed as an optimization problem, it entails search over an exponentially large space, with discrete and continuous variables. An efficient solution requires a strong structural prior on the optimization landscape of this problem.<\/p>\n\n\n\n

In this talk, we survey some of the most powerful techniques for AutoML on tabular datasets. We will focus in particular on techniques for meta-learning: how to quickly learn good models on a new dataset given good models for a large collection of datasets. We will see that remarkably simple structural priors, such as the low-dimensional structure used by the AutoML method Oboe, produce state-of-the-art results. The success of these simple models suggests that AutoML may be simpler than was previously understood.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Madeleine Udell is Assistant Professor of Operations Research and Information Engineering and Richard and Sybil Smith Sesquicentennial Fellow at Cornell University. She studies optimization and machine learning for large scale data analysis and control, with applications in marketing, demographic modeling, medical informatics, engineering system design, and automated machine learning. Her research in optimization centers on detecting and exploiting novel structures in optimization problems, with a particular focus on convex and low rank problems. These structures lead the way to automatic proofs of optimality, better complexity guarantees, and faster, more memory-efficient algorithms. She has developed a number of open source libraries for modeling and solving optimization problems, including Convex.jl, one of the top tools in the Julia language for technical computing.<\/p>\n\n\n\n

\n\n\n\n

Latent Stochastic Differential Equations: An Unexplored Model Class<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Professor David Duvenaud, University of Toronto
\nDate:<\/strong> May 25, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n
Watch on demand<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

We show how to do gradient-based stochastic variational inference in stochastic differential equations (SDEs), in a way that allows the use of adaptive SDE solvers. This allows us to scalably fit a new family of richly-parameterized distributions over irregularly-sampled time series. We apply latent SDEs to motion capture data, and to demonstrate infinitely-deep Bayesian neural networks. We also discuss the pros and cons of this barely-explored model class, comparing it to Gaussian processes and neural processes.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

David Duvenaud is an assistant professor in computer science at the University of Toronto. His research focuses on continuous-time models, latent-variable models, and deep learning. His postdoc was done at Harvard University, and his Ph.D. at the University of Cambridge. David is a founding member of the Vector Institute for Artificial Intelligence, and also co-founded Invenia, an energy forecasting company.<\/p>\n\n\n\n

\n\n\n\n

Knowledge Distillation as Semiparametric Inference<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Dr. Lester Mackey, Microsoft Research
\nDate:<\/strong> April 28, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory- constrained devices. Knowledge distillation alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements\u2014cross-fitting and loss correction\u2014to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Lester is a statistical machine learning researcher at Microsoft Research New England and an adjunct professor at Stanford University. He received his Ph.D. in Computer Science (2012), his M.A. in Statistics (2011) from UC Berkeley, and his B.S.E. in Computer Science (2007) from Princeton University. Before joining Microsoft, Lester spent three wonderful years as an assistant professor of Statistics and, by courtesy, Computer Science at Stanford and one as a Simons Math+X postdoctoral fellow, working with Emmanuel Candes. Lester\u2019s Ph.D. advisor was Mike Jordan, and his undergraduate research advisors were Maria Klawe and David Walker. He got his first taste of research at the Research Science Institute and learned to think deeply of simple things at the Ross Program. Lester\u2019s current research interests include statistical machine learning, scalable algorithms, high-dimensional statistics, approximate inference, and probability. Lately, he\u2019s been developing and analyzing scalable learning algorithms for healthcare, climate forecasting, approximate posterior inference, high-energy physics, recommender systems, and the social good.<\/p>\n\n\n\n

\n\n\n\n

Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Roger Grosse, University of Toronto
\nDate:<\/strong> March 31, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Optimization of many deep learning hyperparameters can be formulated as a bilevel optimization problem. While most black-box and gradient-based approaches require many independent training runs, we aim to adapt hyperparameters online as the network trains. The main challenge is to approximate the response Jacobian, which captures how the minimum of the inner objective changes as the hyperparameters are perturbed. To do this, we introduce the self-tuning network (STN), which fits a hypernetwork to approximate the best response function in the vicinity of the current hyperparameters. Differentiating through the hypernetwork lets us efficiently approximate the gradient of the validation loss with respect to the hyperparameters. We train the hypernetwork and hyperparameters jointly. Empirically, we can find hyperparameter settings competitive with Bayesian Optimization in a single run of training, and in some cases find hyperparameter schedules that outperform any fixed hyperparameter value.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Roger Grosse is an Assistant Professor of Computer Science at the University of Toronto, and a founding member of the Vector Institute for Artificial Intelligence. He received his Ph.D. in computer science from MIT, and then spent two years as a postdoc at the University of Toronto. He holds a Canada Research Chair in Probabilistic Inference and Deep Learning, an Ontario MRIS Early Researcher Award, and a CIFAR Canadian AI Chair.<\/p>\n\n\n\n

\n\n\n\n

Taking Advantage of Randomness in Expensive Optimization Problems<\/h3>\n\n\n\n
\"Portrait<\/figure>\n\n\n\n

Speaker:<\/strong> Ryan Adams, Princeton University
\nDate:<\/strong> February 25, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Optimization is at the heart of machine learning, and gradient computation is central to many optimization techniques. Stochastic optimization, in particular, has taken center stage as the principal method of fitting many models, from deep neural networks to variational Bayesian posterior approximations. Generally, one uses data subsampling to efficiently construct unbiased gradient estimators for stochastic optimization, but this is only one possibility. In this talk, I will discuss two alternative approaches to constructing unbiased gradient estimates in machine learning problems. The first approach uses randomized truncation of objective functions defined as loops or limits. Such objectives arise in settings ranging from hyperparameter selection, to fitting parameters of differential equations, to variational inference using lower bounds on the log-marginal likelihood. The second approach revisits the Jacobian accumulation problem at the heart of automatic differentiation, observing that it is possible to collapse the linearized computational graph of, e.g., deep neural networks, in a randomized way such that less memory is used but little performance is lost. These projects are joint work with students Alex Beatson, Deniz Oktay, Joshua Aduol, and Nick McGreivy.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Ryan Adams is interested in machine learning, artificial intelligence, and computational statistics, with applications across science and engineering. Ryan has broad interests but often works on probabilistic methods and approximate Bayesian inference. Ryan is the director of the Undergraduate Certificate in Statistics and Machine Learning<\/a>. He co-founded Whetlab (sold to Twitter in 2015) and formerly co-hosted the Talking Machines<\/a> podcast. Ryan was faculty at Harvard from 2011 to 2016 and was at Twitter and then Google Brain before joining the faculty at Princeton in 2018. He calls his group the Laboratory for Intelligent Probabilistic Systems (LIPS)<\/a>.<\/p>\n\n\n\n

\n\n\n\n

Automating ML Performance Metric Selection<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Sanmi Koyejo, University of Illinois at Urbana-Champaign
\nDate:<\/strong> January 27, 2021 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

From music recommendations to high-stakes medical treatment selection, complex decision-making tasks are increasingly automated as classification problems.  Thus, there is a growing need for classifiers that accurately reflect complex decision-making goals. One often formalizes these learning goals via a performance metric, which, in turn, can be used to evaluate and compare classifiers. Yet, choosing the appropriate metric remains a challenging problem. This talk will outline metric elicitation as a formal strategy to address the metric selection problem. Metric elicitation automates the discovery of implicit preferences from an expert or an expert panel using relatively efficient and straightforward interactive queries. Beyond standard classification settings, I will also outline early work on metric selection for group-fair classification.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Sanmi (Oluwasanmi) Koyejo is an Assistant Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. Koyejo\u2019s research interests are in developing the principles and practice of adaptive and robust machine learning. Additionally, Koyejo focuses on applications to neuroscience and biomedical imaging. Koyejo completed his Ph.D. in Electrical Engineering at the University of Texas at Austin advised by Joydeep Ghosh and completed postdoctoral research at Stanford University with a focus on developing machine learning techniques for neuroimaging data. His postdoctoral research was primarily with Russell A. Poldrack and Pradeep Ravikumar. Koyejo has been the recipient of several awards including a best paper award from the conference on uncertainty in artificial intelligence (UAI), a Kavli Fellowship, an IJCAI early career spotlight, and a trainee award from the Organization for Human Brain Mapping (OHBM). Koyejo serves on the board of the Black in AI organization.<\/p>\n\n\n\n

\n\n\n\n

AI for Adaptive Experiment Design<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Professor Yisong Yue, California Institute of Technology
\nDate:<\/strong> December 15, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Experiment design is hallmark of virtually all research disciplines. In many settings, one important challenge is how to automatically design experiments over large action\/design spaces. Furthermore, it is also important for such a procedure to be adaptive, i.e., to adapt to the outcomes of previous experiments. In this talk, I will describe recent progress in using data-driven algorithmic techniques for adaptive experiment design, also known as active learning and Bayesian optimization in the machine learning community. Building upon the Gaussian process (GP) framework, I will describe case studies in personalized clinical therapy and nanophotonic structure design. Motivated by these applications, I will show how to incorporate real-world considerations such as safety, preference elicitation, and multi-fidelity experiment design into the GP framework, with new algorithms, theoretical guarantees, and empirical validation. Time permitting, I will also briefly overview a few other case studies as well.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Yisong Yue is a professor of\u202fComputing and Mathematical Sciences<\/a>\u202fat the California Institute of Technology. He was previously a research scientist at\u202fDisney Research<\/a>. Before that, he was a postdoctoral researcher in the\u202fMachine Learning Department<\/a>\u202fand the\u202fiLab<\/a>\u202fat Carnegie Mellon University. He received a Ph.D. from\u202fCornell University<\/a>\u202fand a B.S. from the\u202fUniversity of Illinois at Urbana-Champaign<\/a>. Yisong\u2019s research interests are centered around machine learning, and in particular getting theory to work in practice. To that end, his research agenda spans both fundamental research and in-depth collaborations with machine learning practitioners. In the past, his research has been applied to information retrieval, recommender systems, text classification, learning from rich user interfaces, analyzing implicit human feedback, data-driven animation, behavior analysis, sports analytics, experiment design for science, protein engineering, program synthesis, learning-accelerated optimization, robotics, and adaptive planning & allocation problems.<\/p>\n\n\n\n

\n\n\n\n

Automating Dataset Comparison and Manipulation with Optimal Transport<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Dr. David Alvarez-Melis, Microsoft Research
Date:<\/strong> November 18, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Machine learning research has traditionally been model-centric, focusing on architectures, parameter optimization, \u202fand model transfer. Much less attention has been given to the datasets on which these models are trained, which are often assumed to be fixed, or subject to extrinsic and inevitable change. However, successful application of ML in practice often requires substantial effort in terms of dataset preprocessing and manipulation, such as augmenting, merging, mixing, or reducing datasets. In this talk I will present some of our recent work that seeks to formalize and automatize these and other flavors of dataset manipulation under a unified approach. First, I will introduce the Optimal Transport Dataset Distance, which provides a fundamental theoretical building block: a formal notion of similarity between labeled datasets. In the second part of the talk, I will discuss how this notion of distance can be used to formulate a general framework of dataset optimization by means of gradient flows in probability space. I will end by presenting various exciting potential applications of this dataset optimization framework.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

David Alvarez-Melis is a postdoctoral researcher at Microsoft Research, New England, and a research affiliate at MIT. He obtained a PhD in computer science from the MIT Computer Science and Artificial Intelligence Lab in 2019, where his thesis work spanned various topics in machine learning and natural language processing. He also holds BSc and MS degrees in mathematics from ITAM and Courant Institute (NYU), respectively. He has previously spent time at IBM Research, and is a recipient of CONACYT, Hewlett Packard and AI2 awards. David\u2019s current research sits at the intersection of machine learning, optimization and applied mathematics, with a focus on optimal transport and its applications to dataset processing, transfer learning, and generative modeling.<\/p>\n\n\n\n

\n\n\n\n

AutoML and Interpretability: Powering the machine learning revolution in healthcare<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute
Date:<\/strong> October 6, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

AutoML and interpretability are both fundamental to the successful uptake of machine learning by non-expert end users. The former will lower barriers to entry and unlock potent new capabilities that are out of reach when working with ad-hoc models, while the latter will ensure that outputs are transparent, trustworthy, and meaningful. In healthcare, AutoML and interpretability are already beginning to empower the clinical community by enabling the crafting of actionable analytics that can inform and improve decision-making by clinicians, administrators, researchers, policymakers, and beyond. This talk presents state-of-the-art AutoML and interpretability methods for healthcare developed in our lab and how they have been applied in various clinical settings (including cancer, cardiovascular disease, cystic fibrosis, and recently Covid-19), and then explains how these approaches form part of a broader vision for the future of machine learning in healthcare.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Mihaela van der Schaar<\/a> is the John Humphrey Plummer<\/a> Professor of Machine Learning, Artificial Intelligence and Medicine at the University of Cambridge, a Fellow<\/a> at The Alan Turing Institute in London, and a Chancellor\u2019s Professor at UCLA<\/a>. Mihaela was elected IEEE Fellow in 2009. She has received numerous awards, including the Oon Prize on Preventative Medicine<\/a> from the University of Cambridge (2018), a National Science Foundation CAREER Award (2004), 3 IBM Faculty Awards, the IBM Exploratory Stream Analytics Innovation Award, the Philips Make a Difference Award and several best paper awards, including the IEEE Darlington Award. Mihaela\u2019s work has also led to 35 USA patents (many widely cited and adopted in standards) and 45+ contributions to international standards for which she received 3 International ISO (International Organization for Standardization) Awards. In 2019, she was identified by National Endowment for Science, Technology and the Arts as the most-cited female AI researcher in the UK<\/a>. She was also elected as a 2019 \u201cStar in Computer Networking and Communications<\/a>\u201d by N\u00b2Women. Her research expertise span signal and image processing, communication networks, network science, multimedia, game theory, distributed systems, machine learning and AI. Mihaela\u2019s current research focus is on machine learning, AI and operations research for healthcare and medicine.<\/p>\n\n\n\n

\n\n\n\n

Neural architecture search: Coming of age<\/h3>\n\n\n\n
\"Head<\/figure>\n\n\n\n

Speaker:<\/strong> Frank Hutter, University of Freiburg
\nDate:<\/strong> September 9, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Neural Architecture Search (NAS) is a very promising but still young field. I will start this talk by discussing various works aiming to build a scientific community around NAS, including benchmarks, best practices, and open source frameworks. Then, I will discuss several exciting directions for the field: (1) a broad range of possible speedup techniques for NAS; (2) joint NAS + hyperparameter optimization in Auto-PyTorch to allow off-the-shelf AutoML; and (3) the extended problem definition of neural ensemble search (NES) that searches for a set of complementary architectures rather than a single one as in NAS.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Frank Hutter<\/a> is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), as well as Chief Expert AutoML at the Bosch Center for Artificial Intelligence<\/a>. Frank holds a PhD from the University of British Columbia (2009) and a MSc from TU Darmstadt (2004). He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, and with his coauthors, several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. He is the recipient of a 2013 Emmy Noether Fellowship, a 2016 ERC Starting Grant, a 2018 Google Faculty Research Award, and a 2020 ERC PoC Award<\/a>. He is also a Fellow of ELLIS<\/a> and Program Chair at ECML 2020. In the field of AutoML, Frank co-founded the ICML workshop series on AutoML<\/a> in 2014 and has co-organized it every year since, co-authored the prominent AutoML tools Auto-WEKA<\/a> and Auto-sklearn<\/a>, won the first two AutoML challenges with his team, co-authored the first book on AutoML, worked extensively on efficient hyperparameter optimization and neural architecture search, and gave a NeurIPS 2018 tutorial with over 3000 attendees.<\/p>\n\n\n\n

\n\n\n\n

Algorithmic foundations of neural architecture search<\/h3>\n\n\n\n
\"head<\/figure>\n\n\n\n

Speaker:<\/strong> Ameet Talwalkar, Carnegie Mellon University
\nDate:<\/strong> July 28, 2020 | 10:00 AM\u201311:00 AM PT<\/p>\n\n\n\n

\n\t
\n\t\tWatch on demand\t<\/a>

<\/p><\/div>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Neural architecture search (NAS)\u2014the problem of selecting which neural model to use for your learning problem\u2014is a promising direction for automating and democratizing machine learning. Early NAS methods achieved impressive results on canonical image classification and language modeling problems, yet these methods were algorithmically complex and massively expensive computationally. More recent heuristics relying on weight-sharing and gradient-based optimization are drastically more computationally efficient while also achieving state-of-the-art performance. However, these heuristics are also complex, are poorly understood, and have recently come under scrutiny because of inconsistent results on new benchmarks and poor performance as a surrogate for fully trained models. In this talk, we introduce the NAS problem and then present our work studying recent NAS heuristics from first principles. We first perform an extensive ablation study to identify the necessary components of leading NAS methods. We next introduce our geometry-aware framework called GAEA, which exploits the underlying structure of the weight-sharing NAS optimization problem to quickly find high-performance architectures. This leads to simple yet novel algorithms that enjoy faster convergence guarantees than existing gradient-based methods and achieve state-of-the-art accuracy on a wide range of leading NAS benchmarks. Together, our theory and experiments demonstrate a principled way to co-design optimizers and continuous parameterizations of discrete NAS search spaces.<\/p>\n\n\n\n

Biography<\/h4>\n\n\n\n

Ameet Talwalkar<\/a> is an assistant professor in the machine learning department at Carnegie Mellon University and is also co-founder and chief scientist at Determined AI. His interests are in the field of statistical machine learning. His current work is motivated by the goal of democratizing machine learning and focuses on topics related to scalability, automation, fairness, and interpretability of learning algorithms and systems.<\/p>\n\n\n","tab-content":[{"id":0,"name":"About","content":"The Directions in ML: AutoML and Automating Algorithms speaker series brought together AutoML research communities from across the world to learn from each other and build on the latest knowledge and exploration. This virtual speaker series featured talks by outstanding academics and domain experts whose research covered a broad swath of the topic. The live talks were presented virtually from 10 to 11 AM pacific on the day of and the recorded sessions are now available for on demand viewing. Learn more about AutoML research and team at Microsoft Research ><\/a>\r\n

<\/div>\r\n

Speaker lineup<\/h2>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Date<\/strong><\/td>\r\nSpeaker<\/strong><\/td>\r\n<\/tr>\r\n
July 28, 2020<\/td>\r\nProfessor Ameet Talwalkar, Carnegie Mellon University<\/td>\r\n<\/tr>\r\n
September 9, 2020<\/td>\r\nProfessor Frank Hutter, University of Freiburg<\/td>\r\n<\/tr>\r\n
October 6, 2020<\/td>\r\nProfessor Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute<\/td>\r\n<\/tr>\r\n
November 18, 2020<\/td>\r\nDr. David Alvarez-Melis, Microsoft Research<\/td>\r\n<\/tr>\r\n
December 15, 2020<\/td>\r\nProfessor Yisong Yue, California Institute of Technology<\/td>\r\n<\/tr>\r\n
January 27, 2021<\/td>\r\nProfessor Sanmi Koyejo, University of Illinois at Urbana-Champaign<\/td>\r\n<\/tr>\r\n
February 25, 2021<\/td>\r\nProfessor Ryan Adams, Princeton University<\/td>\r\n<\/tr>\r\n
March 31, 2021<\/td>\r\nProfessor Roger Grosse, University of Toronto<\/td>\r\n<\/tr>\r\n
April 28, 2021<\/td>\r\nDr. Lester Mackey, Microsoft Research<\/td>\r\n<\/tr>\r\n
May 25, 2021<\/td>\r\nProfessor David Duvenaud, University of Toronto<\/td>\r\n<\/tr>\r\n
June 29, 2021<\/td>\r\nProfessor Madeleine Udell, Cornell University<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
<\/div>\r\n

Microsoft\u2019s Event Code of Conduct<\/h3>\r\nMicrosoft\u2019s mission is to empower every person and every organization on the planet to achieve more. This includes virtual events Microsoft hosts and participates in, where we seek to create a respectful, friendly, and inclusive experience for all participants. As such, we do not tolerate harassing or disrespectful behavior, messages, images, or interactions by any event participant, in any form, at any aspect of the program including business and social activities, regardless of location. We do not tolerate any behavior that is degrading to any gender, race, sexual orientation or disability, or any behavior that would violate Microsoft\u2019s Anti-Harassment and Anti-Discrimination Policy, Equal Employment Opportunity Policy, or Standards of Business Conduct<\/a>. In short, the entire experience must meet our culture standards. We encourage everyone to assist in creating a welcoming and safe environment. Please report<\/a> any concerns, harassing behavior, or suspicious or disruptive activity. Microsoft reserves the right to ask attendees to leave at any time at its sole discretion.\r\n
<\/div>\r\n
[msr-button text=\"Report a concern\" url=\"https:\/\/app.convercent.com\/en-us\/Anonymous\/IssueIntake\/LandingPage\/65d3b907-0933-e611-8105-000d3ab03673\" new-window=\"true\" ]<\/div>"},{"id":1,"name":"Past speakers","content":"

Structured Models for Automated Machine Learning<\/h3>\r\n\"PortraitSpeaker:<\/strong> Professor Madeleine Udell, Cornell University\r\nDate:<\/strong> June 29, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-structured-models-for-automated-machine-learning\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nAutomated machine learning (AutoML) seeks algorithmic methods for finding the best machine learning pipeline and hyperparameters to fit a new dataset. The complexity of this problem is astounding: viewed as an optimization problem, it entails search over an exponentially large space, with discrete and continuous variables. An efficient solution requires a strong structural prior on the optimization landscape of this problem.\r\n\r\nIn this talk, we survey some of the most powerful techniques for AutoML on tabular datasets. We will focus in particular on techniques for meta-learning: how to quickly learn good models on a new dataset given good models for a large collection of datasets. We will see that remarkably simple structural priors, such as the low-dimensional structure used by the AutoML method Oboe, produce state-of-the-art results. The success of these simple models suggests that AutoML may be simpler than was previously understood.\r\n

Biography<\/h4>\r\nMadeleine Udell is Assistant Professor of Operations Research and Information Engineering and Richard and Sybil Smith Sesquicentennial Fellow at Cornell University. She studies optimization and machine learning for large scale data analysis and control, with applications in marketing, demographic modeling, medical informatics, engineering system design, and automated machine learning. Her research in optimization centers on detecting and exploiting novel structures in optimization problems, with a particular focus on convex and low rank problems. These structures lead the way to automatic proofs of optimality, better complexity guarantees, and faster, more memory-efficient algorithms. She has developed a number of open source libraries for modeling and solving optimization problems, including Convex.jl, one of the top tools in the Julia language for technical computing.\r\n\r\n
\r\n\r\n

Latent Stochastic Differential Equations: An Unexplored Model Class<\/h3>\r\n\"PortraitSpeaker:<\/strong> Professor David Duvenaud, University of Toronto\r\nDate:<\/strong> May 25, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-latent-stochastic-differential-equations-an-unexplored-model-class-talk\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nWe show how to do gradient-based stochastic variational inference in stochastic differential equations (SDEs), in a way that allows the use of adaptive SDE solvers. This allows us to scalably fit a new family of richly-parameterized distributions over irregularly-sampled time series. We apply latent SDEs to motion capture data, and to demonstrate infinitely-deep Bayesian neural networks. We also discuss the pros and cons of this barely-explored model class, comparing it to Gaussian processes and neural processes.\r\n

Biography<\/h4>\r\nDavid Duvenaud is an assistant professor in computer science at the University of Toronto. His research focuses on continuous-time models, latent-variable models, and deep learning. His postdoc was done at Harvard University, and his Ph.D. at the University of Cambridge. David is a founding member of the Vector Institute for Artificial Intelligence, and also co-founded Invenia, an energy forecasting company.\r\n\r\n
\r\n\r\n

Knowledge Distillation as Semiparametric Inference<\/h3>\r\n\"PortraitSpeaker:<\/strong> Dr. Lester Mackey, Microsoft Research\r\nDate:<\/strong> April 28, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/knowledge-distillation-as-semiparametric-inference\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nMore accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory- constrained devices. Knowledge distillation alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements\u2014cross-fitting and loss correction\u2014to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.\r\n

Biography<\/h4>\r\nLester is a statistical machine learning researcher at Microsoft Research New England and an adjunct professor at Stanford University. He received his Ph.D. in Computer Science (2012), his M.A. in Statistics (2011) from UC Berkeley, and his B.S.E. in Computer Science (2007) from Princeton University. Before joining Microsoft, Lester spent three wonderful years as an assistant professor of Statistics and, by courtesy, Computer Science at Stanford and one as a Simons Math+X postdoctoral fellow, working with Emmanuel Candes. Lester\u2019s Ph.D. advisor was Mike Jordan, and his undergraduate research advisors were Maria Klawe and David Walker. He got his first taste of research at the Research Science Institute and learned to think deeply of simple things at the Ross Program. Lester\u2019s current research interests include statistical machine learning, scalable algorithms, high-dimensional statistics, approximate inference, and probability. Lately, he\u2019s been developing and analyzing scalable learning algorithms for healthcare, climate forecasting, approximate posterior inference, high-energy physics, recommender systems, and the social good.\r\n\r\n
\r\n\r\n

Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization<\/h3>\r\n\"PortraitSpeaker:<\/strong> Roger Grosse, University of Toronto\r\nDate:<\/strong> March 31, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/self-tuning-networks-amortizing-the-hypergradient-computation-for-hyperparameter-optimization\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nOptimization of many deep learning hyperparameters can be formulated as a bilevel optimization problem. While most black-box and gradient-based approaches require many independent training runs, we aim to adapt hyperparameters online as the network trains. The main challenge is to approximate the response Jacobian, which captures how the minimum of the inner objective changes as the hyperparameters are perturbed. To do this, we introduce the self-tuning network (STN), which fits a hypernetwork to approximate the best response function in the vicinity of the current hyperparameters. Differentiating through the hypernetwork lets us efficiently approximate the gradient of the validation loss with respect to the hyperparameters. We train the hypernetwork and hyperparameters jointly. Empirically, we can find hyperparameter settings competitive with Bayesian Optimization in a single run of training, and in some cases find hyperparameter schedules that outperform any fixed hyperparameter value.\r\n

Biography<\/h4>\r\nRoger Grosse is an Assistant Professor of Computer Science at the University of Toronto, and a founding member of the Vector Institute for Artificial Intelligence. He received his Ph.D. in computer science from MIT, and then spent two years as a postdoc at the University of Toronto. He holds a Canada Research Chair in Probabilistic Inference and Deep Learning, an Ontario MRIS Early Researcher Award, and a CIFAR Canadian AI Chair.\r\n\r\n
\r\n\r\n

Taking Advantage of Randomness in Expensive Optimization Problems<\/h3>\r\n\"PortraitSpeaker:<\/strong> Ryan Adams, Princeton University\r\nDate:<\/strong> February 25, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-taking-advantage-of-randomness-in-expensive-optimization-problems\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nOptimization is at the heart of machine learning, and gradient computation is central to many optimization techniques. Stochastic optimization, in particular, has taken center stage as the principal method of fitting many models, from deep neural networks to variational Bayesian posterior approximations. Generally, one uses data subsampling to efficiently construct unbiased gradient estimators for stochastic optimization, but this is only one possibility. In this talk, I will discuss two alternative approaches to constructing unbiased gradient estimates in machine learning problems. The first approach uses randomized truncation of objective functions defined as loops or limits. Such objectives arise in settings ranging from hyperparameter selection, to fitting parameters of differential equations, to variational inference using lower bounds on the log-marginal likelihood. The second approach revisits the Jacobian accumulation problem at the heart of automatic differentiation, observing that it is possible to collapse the linearized computational graph of, e.g., deep neural networks, in a randomized way such that less memory is used but little performance is lost. These projects are joint work with students Alex Beatson, Deniz Oktay, Joshua Aduol, and Nick McGreivy.\r\n

Biography<\/h4>\r\nRyan Adams is\u00a0interested in machine learning, artificial intelligence, and computational statistics, with applications across science and engineering.\u00a0Ryan\u00a0has\u00a0broad interests but often works\u00a0on probabilistic methods and approximate Bayesian inference.\u00a0Ryan is\u00a0the director of the\u00a0Undergraduate Certificate in Statistics and Machine Learning<\/a>.\u00a0He\u00a0co-founded\u00a0Whetlab\u00a0(sold to Twitter in 2015) and formerly co-hosted the Talking Machines<\/a> podcast.\u00a0Ryan\u00a0was\u00a0faculty\u00a0at Harvard from 2011 to 2016 and was at Twitter and then Google Brain before joining the faculty at Princeton in 2018. He calls\u00a0his\u00a0group the Laboratory for Intelligent Probabilistic Systems (LIPS)<\/a>.\r\n\r\n
\r\n\r\n

Automating ML Performance Metric Selection<\/h3>\r\n\"headSpeaker:<\/strong> Professor Sanmi Koyejo, University of Illinois at Urbana-Champaign\r\nDate:<\/strong> January 27, 2021 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-automating-ml-performance-metric-selection\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nFrom music recommendations to high-stakes medical treatment selection, complex decision-making tasks are increasingly automated as classification problems.\u00a0 Thus, there is a growing need for classifiers that accurately reflect complex decision-making goals. One often formalizes these learning goals via a performance metric, which, in turn, can be used to evaluate and compare classifiers. Yet, choosing the appropriate metric remains a challenging problem. This talk will outline metric elicitation as a formal strategy to address the metric selection problem. Metric elicitation automates the discovery of implicit preferences from an expert or an expert panel using relatively efficient and straightforward interactive queries. Beyond standard classification settings, I will also outline early work on metric selection for group-fair classification.\r\n

Biography<\/h4>\r\nSanmi (Oluwasanmi) Koyejo is an Assistant Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign.\u00a0Koyejo's\u00a0research interests are in developing the principles and practice of adaptive and robust machine learning. Additionally, Koyejo focuses on applications to neuroscience and biomedical imaging. Koyejo completed his Ph.D. in Electrical Engineering at the University of Texas at Austin advised by\u00a0Joydeep\u00a0Ghosh and completed postdoctoral research at Stanford University with a focus on developing machine learning techniques for neuroimaging data. His postdoctoral research was primarily with Russell A.\u00a0Poldrack\u00a0and Pradeep Ravikumar. Koyejo has been the recipient of several awards including a best paper award from the conference on uncertainty in artificial intelligence (UAI), a\u00a0Kavli\u00a0Fellowship, an IJCAI early career spotlight, and a trainee award from the Organization for Human Brain Mapping (OHBM). Koyejo serves on the board of the Black in AI organization.\r\n\r\n
\r\n\r\n

AI for Adaptive Experiment Design<\/h3>\r\n\"headSpeaker:<\/strong> Professor Yisong Yue, California Institute of Technology\r\nDate:<\/strong> December 15, 2020 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-ai-for-adaptive-experiment-design\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nExperiment design is hallmark of virtually all research disciplines. In many settings, one important challenge is how to automatically design experiments over large action\/design spaces. Furthermore, it is also important for such a procedure to be adaptive, i.e., to adapt to the outcomes of previous experiments. In this talk, I will describe recent progress in using data-driven algorithmic techniques for adaptive experiment design, also known as active learning and Bayesian optimization in the machine learning community. Building upon the Gaussian process (GP) framework, I will describe case studies in personalized clinical therapy and nanophotonic structure design. Motivated by these applications, I will show how to incorporate real-world considerations such as safety, preference elicitation, and multi-fidelity experiment design into the GP framework, with new algorithms, theoretical guarantees, and empirical validation. Time permitting, I will also briefly overview a few other case studies as well.\r\n

Biography<\/h4>\r\nYisong Yue is a professor of\u202fComputing and Mathematical Sciences<\/a>\u202fat the California Institute of Technology. He was previously a research scientist at\u202fDisney Research<\/a>. Before that, he was a postdoctoral researcher in the\u202fMachine Learning Department<\/a>\u202fand the\u202fiLab<\/a>\u202fat Carnegie Mellon University. He received a Ph.D. from\u202fCornell University<\/a>\u202fand a B.S. from the\u202fUniversity of Illinois at Urbana-Champaign<\/a>. Yisong's research interests are centered around machine learning, and in particular getting theory to work in practice. To that end, his research agenda spans both fundamental research and in-depth collaborations with machine learning practitioners. In the past, his research has been applied to information retrieval, recommender systems, text classification, learning from rich user interfaces, analyzing implicit human feedback, data-driven animation, behavior analysis, sports analytics, experiment design for science, protein engineering, program synthesis, learning-accelerated optimization, robotics, and adaptive planning & allocation problems.\r\n\r\n
\r\n\r\n

Automating Dataset Comparison and Manipulation with Optimal Transport<\/h3>\r\n\"headSpeaker:<\/strong> Dr. David Alvarez-Melis, Microsoft Research\r\nDate:<\/strong> November 18, 2020 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-automating-dataset-comparison-and-manipulation-with-optimal-transport\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nMachine learning research has traditionally been model-centric, focusing on architectures, parameter optimization, \u202fand model transfer. Much less attention has been given to the datasets on which these models are trained, which are often assumed to be fixed, or subject to extrinsic and inevitable change. However, successful application of ML in practice often requires substantial effort in terms of dataset preprocessing and manipulation, such as augmenting, merging, mixing, or reducing datasets. In this talk I will present some of our recent work that seeks to formalize and automatize these and other flavors of dataset manipulation under a unified approach. First, I will introduce the Optimal Transport Dataset Distance, which provides a fundamental theoretical building block: a formal notion of similarity between labeled datasets. In the second part of the talk, I will discuss how this notion of distance can be used to formulate a general framework of dataset optimization by means of gradient flows in probability space. I will end by presenting various exciting potential applications of this dataset optimization framework.\r\n

Biography<\/h4>\r\nDavid Alvarez-Melis is a postdoctoral researcher at Microsoft Research, New England, and a research affiliate at MIT. He obtained a PhD in computer science from the MIT Computer Science and Artificial Intelligence Lab in 2019, where his thesis work spanned various topics in machine learning and natural language processing. He also holds BSc and MS degrees in mathematics from ITAM and Courant Institute (NYU), respectively. He has previously spent time at IBM Research, and is a recipient of CONACYT, Hewlett Packard and AI2 awards. David's current research sits at the intersection of machine learning, optimization and applied mathematics, with a focus on optimal transport and its applications to dataset processing, transfer learning, and generative modeling.\r\n\r\n
\r\n\r\n

AutoML and Interpretability: Powering the machine learning revolution in healthcare<\/h3>\r\n\"headSpeaker:<\/strong> Mihaela van der Schaar, University of Cambridge, UCLA, and The Alan Turing Institute\r\nDate:<\/strong> October 6, 2020 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-automl-interpretability-powering-the-machine-learning-revolution-in-healthcare\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nAutoML and interpretability are both fundamental to the successful uptake of machine learning by non-expert end users. The former will lower barriers to entry and unlock potent new capabilities that are out of reach when working with ad-hoc models, while the latter will ensure that outputs are transparent, trustworthy, and meaningful. In healthcare, AutoML and interpretability are already beginning to empower the clinical community by enabling the crafting of actionable analytics that can inform and improve decision-making by clinicians, administrators, researchers, policymakers, and beyond. This talk presents state-of-the-art AutoML and interpretability methods for healthcare developed in our lab and how they have been applied in various clinical settings (including cancer, cardiovascular disease, cystic fibrosis, and recently Covid-19), and then explains how these approaches form part of a broader vision for the future of machine learning in healthcare.\r\n

Biography<\/h4>\r\nMihaela van der\u00a0Schaar<\/a> is the John Humphrey Plummer<\/a> Professor of Machine Learning, Artificial Intelligence and Medicine at the University of Cambridge, a Fellow<\/a> at The Alan Turing Institute in London, and a Chancellor\u2019s Professor at UCLA<\/a>. Mihaela was elected IEEE Fellow in 2009. She has received numerous awards, including the Oon\u00a0Prize on Preventative Medicine<\/a> from the University of Cambridge (2018), a National Science Foundation CAREER Award (2004), 3 IBM Faculty Awards, the IBM Exploratory Stream Analytics Innovation Award, the Philips Make a Difference Award and several best paper awards, including the IEEE Darlington Award. Mihaela\u2019s work has also led to 35 USA patents (many widely cited and adopted in standards) and 45+ contributions to international standards for which she received 3 International ISO (International Organization for Standardization) Awards. In 2019, she was identified by\u00a0National Endowment for Science, Technology and the Arts as the most-cited female AI researcher in the UK<\/a>. She was also elected as a 2019 \u201cStar in Computer Networking and Communications<\/a>\u201d by N\u00b2Women. Her research expertise\u00a0span\u00a0signal and image processing, communication networks, network science, multimedia, game theory, distributed systems, machine learning and AI. Mihaela\u2019s current research focus is on machine learning, AI and operations research for healthcare and medicine.\r\n\r\n
\r\n\r\n

Neural architecture search: Coming of age<\/h3>\r\n\"HeadSpeaker:<\/strong> Frank Hutter, University of Freiburg\r\nDate:<\/strong> September 9, 2020 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/directions-in-ml-neural-architecture-search-coming-of-age\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nNeural Architecture Search (NAS) is a very promising but still young field. I will start this talk by discussing various works aiming to build a scientific community around NAS, including benchmarks, best practices, and open source frameworks. Then, I will discuss several exciting directions for the field: (1) a broad range of possible speedup techniques for NAS; (2) joint NAS + hyperparameter optimization in Auto-PyTorch to allow off-the-shelf AutoML; and (3) the extended problem definition of neural ensemble search (NES) that searches for a set of complementary architectures rather than a single one as in NAS.\r\n

Biography<\/h4>\r\nFrank Hutter<\/a> is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), as well as Chief Expert AutoML at the Bosch Center for Artificial Intelligence<\/a>. Frank holds a PhD from the University of British Columbia (2009) and a MSc from TU Darmstadt (2004). He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, and with his coauthors, several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. He is the recipient of a 2013 Emmy Noether Fellowship, a 2016 ERC Starting Grant, a 2018 Google Faculty Research Award, and a 2020 ERC PoC Award<\/a>. He is also a Fellow of ELLIS<\/a> and Program Chair at ECML 2020. In the field of AutoML, Frank co-founded the ICML workshop series on AutoML<\/a> in 2014 and has co-organized it every year since, co-authored the prominent AutoML tools Auto-WEKA<\/a> and Auto-sklearn<\/a>, won the first two AutoML challenges with his team, co-authored the first book on AutoML, worked extensively on efficient hyperparameter optimization and neural architecture search, and gave a NeurIPS 2018 tutorial with over 3000 attendees.\r\n\r\n
\r\n\r\n

Algorithmic foundations of neural architecture search<\/h3>\r\n\"headSpeaker:<\/strong> Ameet Talwalkar, Carnegie Mellon University\r\nDate:<\/strong> July 28, 2020 | 10:00 AM\u201311:00 AM PT\r\n
[msr-button text=\"Watch on demand\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/algorithmic-foundations-of-neural-architecture-search\/\" ]<\/div>\r\n
<\/div>\r\n

Abstract<\/h4>\r\nNeural architecture search (NAS)\u2014the problem of selecting which neural model to use for your learning problem\u2014is a promising direction for automating and democratizing machine learning. Early NAS methods achieved impressive results on canonical image classification and language modeling problems, yet these methods were algorithmically complex and massively expensive computationally. More recent heuristics relying on weight-sharing and gradient-based optimization are drastically more computationally efficient while also achieving state-of-the-art performance. However, these heuristics are also complex, are poorly understood, and have recently come under scrutiny because of inconsistent results on new benchmarks and poor performance as a surrogate for fully trained models. In this talk, we introduce the NAS problem and then present our work studying recent NAS heuristics from first principles.\u00a0We first perform an extensive ablation study to identify the necessary components of leading NAS methods. We next introduce our geometry-aware framework called GAEA, which exploits the underlying structure of the weight-sharing NAS optimization problem to quickly find high-performance architectures. This leads to simple yet novel algorithms that enjoy faster convergence guarantees than existing gradient-based methods and achieve state-of-the-art accuracy on a wide range of leading NAS benchmarks. Together, our theory and experiments demonstrate a principled way to co-design optimizers and continuous parameterizations of discrete NAS search spaces.\r\n

Biography<\/h4>\r\nAmeet Talwalkar<\/a> is an assistant professor in the machine learning department at Carnegie Mellon University and is also co-founder and chief scientist at Determined AI. His interests are in the field of statistical machine learning. His current work is motivated by the goal of democratizing machine learning and focuses on topics related to scalability, automation, fairness, and interpretability of learning algorithms and systems."}],"msr_startdate":"2020-07-28","msr_enddate":"2021-06-29","msr_event_time":"10:00 AM\u201311:00 AM PT","msr_location":"Virtual","msr_event_link":"","msr_event_recording_link":"","msr_startdate_formatted":"July 28, 2020","msr_register_text":"Watch now","msr_cta_link":"","msr_cta_text":"","msr_cta_bi_name":"","featured_image_thumbnail":"\"Tiled,","event_excerpt":"Join us as we bring together AutoML research communities from across the world to learn from each other and building on the latest knowledge and exploration. This speaker series features talks by outstanding academics and domain experts whose research covers a broad swath of the topic.","msr_research_lab":[199563],"related-researchers":[],"msr_impact_theme":[],"related-academic-programs":[],"related-groups":[],"related-projects":[545241],"related-opportunities":[],"related-publications":[],"related-videos":[],"related-posts":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/664110"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-event"}],"version-history":[{"count":32,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/664110\/revisions"}],"predecessor-version":[{"id":866556,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/664110\/revisions\/866556"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/674697"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=664110"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=664110"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=664110"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=664110"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=664110"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=664110"},{"taxonomy":"msr-program-audience","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-program-audience?post=664110"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=664110"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=664110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}