From SqueezeNet to SqueezeBERT: Developing Efficient Deep Neural Networks
Deep neural networks have been trained to interpret images and text at increasingly high levels of accuracy. In many cases, these accuracy improvements are the result of developing increasingly large and computationally-intensive neural network models. These models tend to incur high latency during inference, especially when deployed on smartphones and edge-devices. In this talk, we present two lines of work that focus on mitigating the high cost of neural network inference on edge-devices. First, we review the last four years of progress in the computer vision (CV) community towards developing efficient neural networks for edge-devices, ranging from early work such as SqueezeNet, to recent work leveraging neural architecture search. Second, we present SqueezeBERT, a mobile-optimized neural network design for natural language processing (NLP) that draws on ideas from efficient CV network design. SqueezeBERT achieves a 4.3x speedup over BERT-base on a Pixel 3 smartphone. Finally, we believe that SqueezeBERT is just the beginning of several years of fruitful research in the NLP community to develop efficient neural architectures.
- Series:
- Microsoft Research Talks
- Date:
- Speakers:
- Forrest Iandola, Sujeeth Bharadwaj
- Affiliation:
- University of California Berkeley, Microsoft Research
-
-
Sujeeth Bharadwaj
Principal Research Manager
-
-
Series: Microsoft Research Talks
-
Decoding the Human Brain – A Neurosurgeon’s Experience
Speakers:- Pascal Zinn,
- Ivan Tashev
-
-
-
-
Galea: The Bridge Between Mixed Reality and Neurotechnology
Speakers:- Eva Esteban,
- Conor Russomanno
-
Current and Future Application of BCIs
Speakers:- Christoph Guger
-
Challenges in Evolving a Successful Database Product (SQL Server) to a Cloud Service (SQL Azure)
Speakers:- Hanuma Kodavalla,
- Phil Bernstein
-
Improving text prediction accuracy using neurophysiology
Speakers:- Sophia Mehdizadeh
-
-
DIABLo: a Deep Individual-Agnostic Binaural Localizer
Speakers:- Shoken Kaneko
-
-
Recent Efforts Towards Efficient And Scalable Neural Waveform Coding
Speakers:- Kai Zhen
-
-
Audio-based Toxic Language Detection
Speakers:- Midia Yousefi
-
-
From SqueezeNet to SqueezeBERT: Developing Efficient Deep Neural Networks
Speakers:- Sujeeth Bharadwaj
-
Hope Speech and Help Speech: Surfacing Positivity Amidst Hate
Speakers:- Monojit Choudhury
-
-
-
-
-
'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
Speakers:- Peter Clark
-
Checkpointing the Un-checkpointable: the Split-Process Approach for MPI and Formal Verification
Speakers:- Gene Cooperman
-
Learning Structured Models for Safe Robot Control
Speakers:- Ashish Kapoor
-