{"id":711436,"date":"2020-12-11T10:09:36","date_gmt":"2020-12-11T18:09:36","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=711436"},"modified":"2020-12-14T12:59:47","modified_gmt":"2020-12-14T20:59:47","slug":"seeing-on-tiny-battery-powered-microcontrollers-with-rnnpool","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/seeing-on-tiny-battery-powered-microcontrollers-with-rnnpool\/","title":{"rendered":"\u2018Seeing\u2019 on tiny battery-powered microcontrollers with RNNPool"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_No_logo-1.gif\" alt=\"\"\/><\/figure>\n\n\n\n<p>Computer vision has rapidly evolved over the past decade, allowing for such applications as <a href=\"https:\/\/www.microsoft.com\/en-us\/ai\/seeing-ai\">Seeing AI<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, a camera app that describes aloud a person\u2019s surroundings, helping those who are blind or have low vision; systems that can detect whether a product, such as a computer chip or article of clothing, has been assembled correctly, improving quality control; and services that can <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/support.microsoft.com\/en-us\/office\/insert-data-from-picture-3c1bb58d-2c59-4bc0-b04a-a671a6868fd7?ui=en-us&rs=en-us&ad=us\">convert information from hard-copy documents into a digital format<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, making it easier to manage personal and business data. All this has been made possible by the evolution of convolutional neural networks (CNNs) and faster hardware that can run increasingly deeper architectures. And thanks to specialized neural accelerators for CNNs, computer vision applications that were limited to the realm of plugged-in power-hungry workstations just a few years ago can even be performed by mobile devices.<\/p>\n\n\n\n<p>The next frontier in this natural progression is the edge.<em> The very edge<\/em>. Tiny devices that consume less power and are capable of collecting and analyzing data in a connected Internet of Things (IoT) world. Can we enable sophisticated vision intelligence on these devices? Vision on such devices could open up new scenarios. Imagine the walking stick of a person who is blind or has low vision outfitted with a small camera that can detect a dog, vehicle, or other object and provide feedback that would allow the person to avoid a potential accident or a retail system that can detect an empty shelf and alert a manager that a high-demand product needs to be restocked or reordered. This would not only enable users of such technology to understand their surroundings in new ways, but it would also give them more control over their data by retaining sensitive information on the edge instead of sending it over the network to the cloud for processing, which could compromise it.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">CODE<\/span>\n\t\t\t<a href=\"https:\/\/github.com\/microsoft\/EdgeML\/blob\/master\/pytorch\/edgeml_pytorch\/graph\/rnnpool.py\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"PyTorch RNNPool operator\" data-bi-aN=\"margin-callout\" data-bi-cN=\"PyTorch RNNPool operator\">\n\t\t\t\tPyTorch RNNPool operator&nbsp;<span class=\"glyph-append glyph-append-share glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>But today\u2019s vision architectures require large amounts of memory and compute, more than is available on typical IoT devices. To address this challenge, we introduce RNNPool, a pooling operator that can rapidly reduce the size of intermediate image representations to allow for the analysis of images on small memory- and compute-constrained devices. When used as a replacement for CNN blocks, RNNPool enables CNN models with much fewer layers, smaller intermediate representations, and hence lesser computational requirement while still ensuring state-of-the-art performance for small models. Because RNNPool is syntactically equivalent and semantically similar to existing pooling operators, it can easily be incorporated into standard CNN architectures. We\u2019re presenting \u201c<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/rnnpool-efficient-non-linear-pooling-for-ram-constrained-inference\/\">RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference<\/a>\u201d at the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/neurips-2020\/\">34th Conference on Neural Information Processing Systems (NeurIPS 2020)<\/a>.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/rnnpool-efficient-non-linear-pooling-for-ram-constrained-inference\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference\" data-bi-aN=\"margin-callout\" data-bi-cN=\"RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference\">\n\t\t\t\tRNNPool: Efficient Non-linear Pooling for RAM Constrained Inference&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/neurips-2020\/\" target=\"_self\" aria-label=\"Microsoft at NeurIPS 2020\" data-bi-type=\"annotated-link\" data-bi-cN=\"Microsoft at NeurIPS 2020\" class=\"annotations__list-thumbnail\" >\n\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"170\" height=\"96\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-300x169.jpg\" class=\"mb-2\" alt=\"\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-1024x577.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-768x433.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-2048x1154.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_Neurips_Banner_withcopy-1-1920x1080.jpg 1920w\" sizes=\"(max-width: 170px) 100vw, 170px\" \/>\t\t\t\t<\/a>\n\t\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">EVENT<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/neurips-2020\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Microsoft at NeurIPS 2020\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Microsoft at NeurIPS 2020\">\n\t\t\t\tMicrosoft at NeurIPS 2020&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">Check out Microsoft at NeurIPS 2020, including all of our NeurIPS publications, the Microsoft session schedule, and open career opportunities.<\/p>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure-1.jpg\" alt=\"\" class=\"wp-image-711442\" width=\"325\" height=\"264\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure-1.jpg 437w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure-1-300x244.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure-1-16x12.jpg 16w\" sizes=\"(max-width: 325px) 100vw, 325px\" \/><\/figure><\/div>\n\n\n\n<h2 id=\"the-memory-demands-of-convolutional-neural-networks\">The memory demands of convolutional neural networks<\/h2>\n\n\n\n<p>Most popular computer vision architectures are based on CNNs, which typically consist of interleaved convolution and pooling layers. Roughly speaking, a convolution layer extracts features from an input image, while a pooling layer combines the features to provide a more succinct representation of the image. An image moves through a network\u2019s layers until we obtain a final representation of the image, which is passed through a standard linear classifier\/regressor to get the prediction for the image. The output of each layer is called the <em>activation map<\/em> and is organized as a three-dimensional tensor\u2014<em>n<\/em> rows \u00d7<em> n <\/em>columns \u00d7 <em>n <\/em>channels\u2014where the number of channels is in general the property of the layer itself, independent of the input size. To capture a large range of discriminative features, CNNs tend to have convolution layers with a larger number of output channels, leading to large activation maps. For example, the red block in the below figure of DenseNet-121, a standard CNN architecture, represents a 56 \u00d7 56 \u00d7 256 activation map, which requires 800 KB of RAM even if each number in the map is rounded to an 8-bit integer. In our paper, we rigorously show that rearranging the computation graph of a CNN network can\u2019t reduce the working memory (RAM) requirement. This represents a significant roadblock in deploying accurate CNNs on typical tiny devices, as these devices are powered by microcontrollers with small amounts of RAM to limit the power consumption and die area. Most Arm Cortex-M4 microcontrollers, for example, house less than 256 KB of RAM.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"249\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed-1024x249.jpg\" alt=\"A diagram of DenseNet-121 in which a series of 11 cuboids represents an input image and then subsequent activation maps at various sizes, measured as n rows \u00d7 n columns \u00d7 n channels. The cuboids\u2014connected to one another by rectangles representing a different block of layers\u2014are arranged horizontally, and the number of rows and columns of each gradually decreases while the number of channels gradually increases. The activation map size starts at 112 \u00d7 112 \u00d7 64 and ends at 1 \u00d7 1 \u00d7 2024. \" class=\"wp-image-712291\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed-1024x249.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed-300x73.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed-768x187.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed-16x4.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNNPool_HighRes_fig1_fixed.jpg 1527w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>DenseNet-121 is a popular CNN architecture consisting of a series of convolution (C) and dense (D) blocks that extract features from an image and pooling blocks (P and T) that downsample the activation map. Intermediate activation maps have a large number of channels to accurately capture features of the image. The red block represents a 56 \u00d7 56 \u00d7 256 activation map, which requires 800 KB of RAM. FC + softmax represents the standard fully connected layer with softmax activation.<\/figcaption><\/figure><\/div>\n\n\n\n<p>A natural approach to reducing the activation map size is to decrease the number of output channels, but that can lead to drastically lower accuracy. Another approach is to downsample the number of rows\/columns of the map aggressively. For example, if we had a 28 \u00d7 28 \u00d7 256 activation map instead of the 56 \u00d7 56 \u00d7 256 map in the above example, then an image could be stored in just 200 KB. Pooling operators and strided convolutions are the standard approaches to downsampling activation maps, but they rely on relatively simple and lossy aggregation and so can lead to significant loss in accuracy if applied to a large <em>receptive field,<\/em> or patch of the image, for more aggressive downsampling. Hence, as evident in the figure of DenseNet-121, and in most standard CNN architectures, such operators are restricted to 2 \u00d7 2 receptive fields only, to ensure good accuracy. But the 2 \u00d7 2 pooling provides only a four-time reduction in activation map size, which is not sufficient for our purpose. We need to find a pooling operator that can summarize large patches of the activation map and bring down the size of the activation map in one shot. RNNPool is exactly that.<\/p>\n\n\n\n<h2 id=\"rnnpool-applying-recurrent-neural-networks-to-computer-vision\">RNNPool: Applying recurrent neural networks to computer vision<\/h2>\n\n\n\n<p>RNNPool consists of two learnt recurrent neural networks (RNNs) that sweep across each patch of an activation map horizontally and vertically to summarize each patch into a single vector. We use RNNs because they tend to have significant modeling power and hence can summarize\/aggregate patches well. But, unlike typical CNNs, they can still be executed with a small amount of working memory. Not only does RNNPool allow us to get rid of entire blocks of layers without losing accuracy, but we can use it in place of existing pooling layers anywhere in the network and get accuracy improvements. <\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure3.png\" alt=\"An RNNPool operator of the size (r, c, k, h1, h2), where r is the number of rows, c is the number of columns, k is the number of input channels, h1 is the hidden state of RNN1, and h2 is the hidden state of RNN2. An activation map of size r \u00d7 c \u00d7 k is shown as input with a patch of size 4 \u00d7 4 \u00d7 k outlined in orange. Two gray arrows each point from the patch to different orange cubes, one showing RNN1\u2019s horizontal sweep of each row, denoted by blue arrows, and RNN2\u2019s bidirectional sweep, denoted by green arrows, of the resulting 1 \u00d7 1 \u00d7 h1 vector output and the other showing RNN1\u2019s vertical sweep of each column, denoted by blue arrows, and RNN2\u2019s bidirectional sweep, denoted by green arrows, of the resulting 1 \u00d7 1 \u00d7 h1 vector output. RNNPool concatenates the output of all four passes of RNN2 into a 1 \u00d7 1 \u00d7 4h2 vector. \" class=\"wp-image-711448\" width=\"652\" height=\"395\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure3.png 509w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure3-300x181.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RNN-Pool-Figure3-16x10.png 16w\" sizes=\"(max-width: 652px) 100vw, 652px\" \/><figcaption>The RNNPool operator applied to patches of size r \u00d7 c with stride s. It summarizes each patch with two RNNs into a vector size of 4h2. The above process repeats across the entire activation map, producing a 1 \u00d7 1 \u00d7 4h2 vector for each patch. These vectors combine to form the next activation map.<\/figcaption><\/figure><\/div>\n\n\n\n<p>RNNPool takes a patch of an activation map and summarizes it into a 1 \u00d7 1 voxel and then strides by <em>s<\/em> steps to summarize the next patch. RNNPool can support patch sizes of 8 \u00d7 8 and even 16 \u00d7 16 and can be strided by <em>s<\/em> = 4 or <em>s<\/em> = 8 steps without significant loss in accuracy. Thus, unlike standard pooling operators, it can downsample activation maps significantly without loss in accuracy. To summarize a patch into a voxel, RNNPool uses four RNN runs: The first RNN goes over each row and column and summarizes all of them into 1 \u00d7 1 vectors of dimension h<sub>1<\/sub>, while the second RNN goes over these summarizations bidirectionally and produces a final 1 \u00d7 1 \u00d7 4h<sub>2 <\/sub>vector, where h<sub>2<\/sub> is the size of the hidden state size of the second RNN.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/RnnPool_HighRes_Figure2_fixed.jpg\" alt=\"A diagram of DenseNet-121 incorporated with RNNPool. A series of gray cuboids arranged horizontally represents an activation map at various sizes, measured as n rows \u00d7 n columns \u00d7 n channels, with the cuboids connected to one another by rectangles representing a different block of layers. Four blocks at the beginning of the DenseNet-121 architecture have been replaced by an \u201cRNNPoolLayer,\u201d represented by a green rectangle.\"\/><figcaption>In this DenseNet-121 CNN architecture, RNNPool replaces entire blocks of layers with a single pooling layer that downsamples the activation map by a factor of 8 \u00d7 8.<\/figcaption><\/figure><\/div>\n\n\n\n<p>Since RNNPool is syntactically equivalent to a pooling operator, it can be used to replace any pooling operator in a CNN. But placing RNNPool in the beginning of the CNN architecture allows us to rapidly downsample the activation map, bringing down the peak RAM requirement and also skipping several layers, which provides further computational gains.<\/p>\n\n\n\n<p>RNNPool can be used for a variety of tasks and with a variety of base architectures. The vision tasks we evaluated RNNPool on include image classification and the Visual Wake Words challenge, in which a model has to determine the presence or absence of an individual. In most of the cases, we observe that RNNPool-based models require eight to 10 times less RAM and two to three times less compute than the base models while still ensuring almost the same accuracy.<\/p>\n\n\n\n<h2 id=\"achieving-state-of-the-art-in-face-detection\">Achieving state-of-the-art in face detection<\/h2>\n\n\n\n<p>In a third evaluation, we incorporate our operator into an S3FD architecture with a MobileNetV2 backbone\u2014a standard face detection architecture\u2014and achieve a state-of-the-art face detection model for tiny devices. Face detection models are designed to determine if and where people are in the frame of an image or video without identifying any specific individual. In particular, the RNNPool-based model, which we call <em>RNNPool-Face-M4<\/em>, is significantly more accurate than the existing state-of-the-art model in this domain with far fewer parameters. More importantly, by using a quantization and code generation tool developed at Microsoft Research called <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/compiling-kb-sized-machine-learning-models-to-constrained-hardware\/\">SeeDot<\/a>, we were able to ensure that the working memory of our model fit within the RAM of an Arm Cortex-M4 device. RNNPool-Face-M4 processes a single image in 10.45 seconds on an Arm Cortex-M4 microcontroller\u2013based STM32F439-M4 device clocked at 168 megahertz. It has a peak RAM requirement of only 188 KB and model size of 160 KB.<\/p>\n\n\n\n<table style=\"border-spacing: inherit;border-collapse: collapse;width: 100%;padding: 10px;text-align: center;font-family:Segoe UI; border: 1px solid #000000\">\n\t<tbody>\n\t\t<tr style=\"border: 1px solid #000000\">\n\t\t\t<td rowspan=\"2\" style=\"color:white; background-color:#0078d7; padding: 10px; border: 1px solid #000000\"><strong>Method<\/strong><\/td>\n\t\t\t<td rowspan=\"2\" style=\"color:white; background-color:#0078d7; padding: 10px; border: 1px solid #000000\"><strong>RAM<\/strong><\/td>\n\t\t\t<td rowspan=\"2\" style=\"color:white; background-color:#0078d7; padding: 10px; border: 1px solid #000000\"><strong>#Params<\/strong><\/td>\n\t\t\t<td rowspan=\"2\" style=\"color:white; background-color:#0078d7; padding: 10px; border: 1px solid #000000\"><strong>MAdds<\/strong><\/td>\n\t\t\t<td colspan=\"3\" style=\"color:white; background-color:#0078d7; padding: 10px; border: 1px solid #000000\"><strong>MAP<\/strong><\/td>\n\t\t<tr style=\"border: 1px solid #000000\">\n\t\t\t<td style=\"color:white; background-color:#3691d8; padding: 10px; border: 1px solid #000000\">E<\/td>\n\t\t\t<td style=\"color:white; background-color:#3691d8; padding: 10px; border: 1px solid #000000\">M<\/td>\n\t\t\t<td style=\"color:white; background-color:#3691d8; padding: 10px; border: 1px solid #000000\">H<\/td>\n\t\t<tr style=\"border: 1px solid #000000\">\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">EagleEye<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">1.17MB<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">0.23M<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">0.1G<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">0.74<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">0.70<\/td>\n\t\t\t<td style=\"padding: 10px; border: 1px solid #000000\">0.44<\/td><\/tr>\n\t\t<tr style=\"border: 1px solid #000000\">\n\t\t\t<td style=\"padding: 10px;border: 1px solid #000000\">Rpool-Face-Quant<\/td>\n\t\t\t<td style=\"padding: 10px; background-color:#7ebe42; border: 1px solid #000000\"><strong>225KB<\/strong><\/td>\n\t\t\t<td style=\"padding: 10px;border: 1px solid #000000\">0.07M<\/td>\n\t\t\t<td style=\"padding: 10px;border: 1px solid #000000\">0.1G<\/td>\n\t\t\t<td style=\"padding: 10px; background-color:#7ebe42; border: 1px solid #000000\"><strong>0.80<\/strong><\/td>\n\t\t\t<td style=\"padding: 10px; background-color:#7ebe42; border: 1px solid #000000\"><strong>0.78<\/strong><\/td>\n\t\t\t<td style=\"padding: 10px; background-color:#7ebe42; border: 1px solid #000000\"><strong>0.53<\/strong><\/td><\/tr><\/tbody><\/table><figcaption>The above table compares mean average precision (MAP), a standard metric for face detection, of two face detectors, the RNNPool-based RNNPool-Face-Quant model and EagleEye, the current state of the art in the small device domain. They were compared on three dataset difficulty levels: easy (E), medium (M), and hard (H). As indicated by the higher MAP, RNNPool-Face-Quant outperforms EagleEye. MAdds counts the total number of multiplications and additions performed during the inference.<\/figcaption>\n<div style=\"height: 20px;\"><\/div>\n\n\n\n<p>Such a model could be used for operational planning. For example, a system powered by RNNPool-Face-M4 could be deployed in an office\u2019s conference rooms to facilitate better management of these spaces. Instead of an office manager physically walking the building to observe which rooms are being used at any given time, the system could provide a head count, helping to identify space that had been reserved but isn\u2019t actually being used. Entertainment venues, such as sports stadiums, could employ a similar system to more efficiently staff its customer-services booths, directing staff members to and from booths depending on how long a line the system detects.<\/p>\n\n\n\n<p>As mentioned earlier in the post, the local processing enabled by RNNPool can offer users of such systems, as well as any people who might be in photos taken by the system, significantly more protection of their data, as images captured by the system won\u2019t be transferred over any networks. The images can be processed completely on the device deploying the system, after which they can be deleted from the device. Because such systems as those described above are counters, the only information being transferred off the device would be the number of people in a frame, not the images. <\/p>\n\n\n\n<p>For the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/EdgeML\/blob\/master\/pytorch\/edgeml_pytorch\/graph\/rnnpool.py\">RNNPool code<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/EdgeML\/tree\/master\/examples\/pytorch\/vision\">code for the face detection and Visual Wake Words applications<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, check out our GitHub repo.<\/p>\n\n\n\n<h2 id=\"machine-learning-for-all\">Machine learning for all<\/h2>\n\n\n\n<p>Computer vision is enhancing how people go about their lives and how companies conduct their business, and bringing the technology onto the devices themselves\u2014moving the intelligence from the cloud to the device\u2014opens new application avenues. It also has the potential to bring the technology to even more people and scenarios.<\/p>\n\n\n\n<p>The RNNPool operator is part of a larger effort to enable not only computer vision on resource-constrained IoT devices, but machine learning in general. RNNPool is part of EdgeML, a growing open-source library of efficient ML algorithms. Our goal with EdgeML is to democratize machine learning. The power-intensive hardware used to send data to the cloud for processing and the inferencing itself can be expensive. Microcontrollers processing data locally offer a more affordable option\u2014intelligence at a lower cost\u2014for businesses and individuals. Additionally, on-device processing can help in scenarios in which latency is critical. It can contribute to lowering energy demands, as well. These small devices use less power, and the energy required to transfer images to the cloud\u2014which tends to be significantly more energy expensive than the inference itself\u2014is also saved. The EdgeML team has been developing algorithms and architectures in the domains of speech and initial measurement unit (IMU) sensors, so RNNPool represents our initial work in the domain of computer vision. To further explore our work, visit the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/resource-efficient-ml-for-the-edge-and-endpoint-iot-devices\/\">EdgeML project page<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/EdgeML\">EdgeML GitHub page<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Computer vision has rapidly evolved over the past decade, allowing for such applications as Seeing AI (opens in new tab), a camera app that describes aloud a person\u2019s surroundings, helping those who are blind or have low vision; systems that can detect whether a product, such as a computer chip or article of clothing, has [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":712642,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-711436","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[638550],"related-events":[708199],"related-researchers":[{"type":"guest","value":"shikhar-jaiswal","user_id":"712306","display_name":"Shikhar Jaiswal","author_link":"Shikhar Jaiswal","is_active":true,"last_first":"Jaiswal, Shikhar","people_section":0,"alias":"shikhar-jaiswal"},{"type":"guest","value":"aayan-kumar","user_id":"712309","display_name":"Aayan  Kumar","author_link":"Aayan  Kumar","is_active":true,"last_first":"Kumar, Aayan ","people_section":0,"alias":"aayan-kumar"},{"type":"user_nicename","value":"Harsha Simhadri","user_id":36146,"display_name":"Harsha Simhadri","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/harshasi\/\" aria-label=\"Visit the profile page for Harsha Simhadri\">Harsha Simhadri<\/a>","is_active":false,"last_first":"Simhadri, Harsha","people_section":0,"alias":"harshasi"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-960x540.jpg\" class=\"img-object-cover\" alt=\"\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-1536x864.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-2048x1152.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/12\/1400x788_RNN_Pool_Still_nologo-1-1920x1080.jpg 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"December 11, 2020","formattedExcerpt":"Computer vision has rapidly evolved over the past decade, allowing for such applications as Seeing AI (opens in new tab), a camera app that describes aloud a person\u2019s surroundings, helping those who are blind or have low vision; systems that can detect whether a product,&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/711436"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=711436"}],"version-history":[{"count":11,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/711436\/revisions"}],"predecessor-version":[{"id":712651,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/711436\/revisions\/712651"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/712642"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=711436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=711436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=711436"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=711436"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=711436"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=711436"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=711436"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=711436"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=711436"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=711436"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=711436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}