Platform for Situated Intelligence

Platform for Situated Intelligence (opens in new tab) is an open-source framework that significantly accelerates development and research on multimodal, integrative-AI applications. The framework was designed to alleviate the engineering challenges that arise when developing systems and applications that process multimodal streaming sensor data (such as audio, video, depth, etc.) and need to leverage and coordinate a variety of component AI technologies. Examples include multimodal interactive systems such as social robots and embodied conversational agents, systems for ambient intelligence and smart spaces, applications based on small devices that work with streaming sensor data, etc.

The framework is cross platform, built on .NET standard, and leverages the affordances of a modern, managed programming language, while addressing the needs of multimodal, integrative-AI applications. The framework provides:

• infrastructure for working with multimodal, temporally streaming data;
• tools for multimodal data visualization, annotation, and processing;
• components that encapsulate various sensors, processing technologies.

To learn more about Platform for Situated Intelligence, please see a high-level overview provided in this blog post (opens in new tab). An in-depth description of various aspects of the framework is available in this technical report (opens in new tab). To get started using the framework and to find in-depth documentation, please see the GitHub repository (opens in new tab). We continue to evolve the framework with novel capabilities, and we invite contributions across the board: from simply using it and filing issues or providing bug fixes, to writing and releasing new components, to developing and contributing new features.