Microsoft Rocket for Live Video Analytics

This page is about Microsoft’s research work on building efficient live video analytics. To learn about Microsoft’s commercial product addressing this need, see Live Video Analytics (opens in new tab) from Azure Media Services. We also provide instructions and code samples (opens in new tab) to set up Microsoft Rocket containers for analyzing videos using Live Video Analytics.

Cameras are now everywhere. Large-scale video processing is a grand challenge representing an important frontier for analytics, what with videos from factory floors, traffic intersections, police vehicles, and retail shops. It’s the golden era for computer vision, AI, and machine learning – it’s a great time now to extract value from videos to impact science, society, and business!

Project Rocket‘s goal is to democratize video analytics: build a system for real-time, low-cost, accurate analysis of live videos. This system will work across a geo-distributed hierarchy of intelligent edges and large clouds, with the ultimate goal of making it easy and affordable for anyone with a camera stream to benefit from video analytics. For information regarding our project, please see our publications.

Rocket: a powerful configurable platform for live video analytics

Microsoft Rocket Video Analytics Platform is now available on GitHub!
Download from GitHub (opens in new tab) |  ✍ Learn about the key features (opens in new tab)

Rocket is an extensible software stack for democratizing video analytics: making it easy and affordable for anyone with a camera stream to benefit from computer vision and machine learning algorithms. It comes with a default object counting pipeline that includes a cascade of DNNs. With Rocket, you can plug in any TensorFlow, Darknet or ONNX DNN model, including custom-built models. You can also augment the above pipeline with simpler motion filters based on OpenCV background subtraction, as shown in the figure below. Rocket’s pipelined architecture can be easily configured to execute over a distributed infrastructure, potentially spanning specialized edge hardware (e.g., Azure Stack Edge (opens in new tab)) and the cloud (e.g., Azure Machine Learning (opens in new tab) and Cognitive Services (opens in new tab)).

For developers, Rocket allows for plugging in new analytics modules to video analytics pipelines. Customized modules can be developed for various applications (as shown in the figure below). These modules can be written to consume and process the data from upstream modules and pass their outputs to the downstream modules.

Video analytics for Vision Zero

One of the verticals this project is focused on is video streams from cameras at traffic intersections (opens in new tab). Traffic-related accidents are among the top 10 reasons for fatalities worldwide. This project partners with jurisdictions to identify traffic details—vehicles, pedestrians, bikes—that impact traffic planning and safety.

We conducted a pilot study in Bellevue, Washington for active traffic monitoring of traffic intersections live 24×7. We hosted a traffic dashboard powered by Rocket’s video analytics live at Bellevue’s Traffic Management Center. The dashboard alerts the traffic authorities on abnormal traffic volumes. Read our case study report. (opens in new tab)

Awards

«Safer Cities, Safer People» US Department of Transportation Award
Institute of Transportation Engineering 2017 Achievements Award – «Video Analytics for Vision Zero»
ACM MobiSys 2017 Best Demo
Microsoft 2017 Hackathon Grand Prize Winner
ACM MobiSys 2019 Best Demo (Runner-up)
ACM Symposium on Edge Computing 2020 Best Paper Award
CSAW 2020 Applied Research Competition Award (Runner-up)