{"id":1052988,"date":"2024-07-10T18:31:31","date_gmt":"2024-07-11T01:31:31","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&p=1052988"},"modified":"2024-07-10T18:31:35","modified_gmt":"2024-07-11T01:31:35","slug":"video-in-context-learning","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/video-in-context-learning\/","title":{"rendered":"Video In-Context Learning"},"content":{"rendered":"
\n\t
\n\t\t
\n\t\t\t\t\t<\/div>\n\t\t\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n

Video In-Context Learning<\/h1>\n\n\n\n

Driving large vision models with video demonstrations<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n

\n
Paper<\/a><\/div>\n<\/div>\n\n\n\n

In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this project, we propose and study video in-context learning, where the model starts from an existing video clip and generates diverse potential future sequences, each semantically guided by the prompted video demonstrations. To achieve this, we provide a clear definition of the task, and train an autoregressive Transformer on video datasets. We thoroughly analyze the effect of different datasets and represent frames as discrete tokens, and then model them by next token predictions.<\/p>\n\n\n\n

As a result, the obtained vision Transformer model is able to generate a subsequent video sequence of a query video clip, which is semantically aligned with the demonstration video. Demonstration videos are highly versatile and capable of conveying a wide range of information, such as examples for various tasks including moving or grabbing objects, or movements of the camera in an ego-centric video. This allows video in-context learning to address multiple downstream tasks, such as embodied planning and simulating, by letting a query robot imitate the actions demonstrated by other robots, as shown below. Video in-context learning serves as a new and crucial interface for models to interact with the real world<\/em><\/strong>, as videos are good at describing low-level details (where language may fall short) and temporal dynamics (where images are insufficient).<\/p>\n\n\n\n

\"Model
The pipeline of Vid-ICL. Left: Training of Vid-ICL. The data used for training are continuous video clips and the Transformer is trained by next token prediction. Right: In-context Inference of Vid-ICL. The model is conditioned on demonstration videos and generates future frames.<\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Generated Samples on Something-Something v2<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

DeLVM<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, ft)<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n
\n
\n
\n
\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

Qualitative results on the generated samples by Vid-ICL on Something-Something v2 dataset, a dataset of real world human activities. . The first column shows the demonstration video. The second column shows the generated samples by DeLVM. The third to sixth columns show the generated samples by Vid-ICL with different pretraining and finetuning strategies. It shows that Vid-ICL generates more diverse and plausible samples than DeLVM, and the quality of the generated samples is improved by scaling model sizes and in-domain fine-tuning.<\/p>\n<\/div>\n<\/div>\n\n\n\n

Generated Samples on Robotics Transformer Dataset<\/h4>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, pt)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (300M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (700M, ft)<\/p>\n<\/div>\n\n\n\n

\n

Vid-ICL (1.1B, ft)<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

Qualitative results on the generated samples by Vid-ICL on Robotics Transformer(RT-1) dataset, a dataset of robotic manipulation tasks. Vid-ICL offers more concise control over various robotic tasks with the demonstration video.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Contrastive Demonstration<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Generation<\/p>\n<\/div>\n\n\n\n

\n

Contrastive Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Contrastive Generation<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n\n\n\n

In this section, we give shared prompts to Vid-ICL in the second and fourth columns of each row, and a contrastive prompt in the first and third columns. Results shows an important feature of Vid-ICL that when conditioned on demonstration with contrastive semantics, Vid-ICL generates contrary samples given the same demonstration, which demonstrates Vid-ICL’s ability to precisely understand the demonstration dynamics and generate diverse future sequences.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

Text Conditioned Generation<\/h4>\n\n\n\n

<\/p>\n\n\n\n

\n
\n

Text<\/p>\n<\/div>\n\n\n\n

\n

Demonstration<\/p>\n<\/div>\n\n\n\n

\n

Generation w\/o Text<\/p>\n<\/div>\n\n\n\n

\n

Generation w\/ Text<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Push [something] from left to right<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Turning the camera left while filming [something]<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n

Moving [something] closer to [something]<\/p>\n<\/div>\n\n\n\n

\n
\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n\n\n\n
\"gif<\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n

We show that text can be used as an additional in-context condition signals to augment the generation of Vid-ICL. After aligning Vid-ICL to the text, the generated samples are more consistent with the text instructions and demonstration videos.<\/p>\n\n\n\n

<\/p>\n\n\n\n

On Reinforcement Learning Tasks<\/h4>\n\n\n\n
\"gif<\/figure>\n\n\n\n

We demonstrate that Vid-ICL can also function as a simulator in reinforcement learning tasks by evaluating it on the RoboDesk, a test benchmark containing several reinforcement learning tasks. Vid-ICL generates future frames with videos that accomplishing the same task as the demonstration, where the generated frames inversely reflect the corresponding actions that correctly interact with the environment. Evaluating on the Push_red<\/strong><\/em> task, we find that Vid-ICL provides more precise control over the environment interaction.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Ethics Statement<\/h4>\n\n\n\n

Vid-ICL is exclusively a research initiative with no current plans for product integration or public access. We are committed to adhering to Microsoft AI principles during the ongoing development of our models. The datasets utilized in this study are publicly available and have been thoroughly reviewed to ensure they do not include personally identifiable information or offensive content. Nonetheless, as these datasets are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the training data to minimize the potential for the model to generate inappropriate content.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Citation<\/h4>\n\n\n\n
@article{zhang2024video,\n  title={Video In-Context Learning},\n  author={Zhang, Wentao and Guo, Junliang and He, Tianyu and Zhao, Li and Xu, Linli and Bian, Jiang},\n  journal={arXiv preprint arXiv:2407.07356},\n  year={2024}\n}<\/code><\/pre>\n\n\n","protected":false},"excerpt":{"rendered":"
Driving large vision models with video demonstrations In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this project, we propose and study video in-context learning, where the model starts from an existing video […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556,13562],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1052988","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[1056966],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Junliang Guo","user_id":41188,"people_section":"Related people","alias":"junliangguo"},{"type":"user_nicename","display_name":"Tianyu He","user_id":42141,"people_section":"Related people","alias":"tianyuhe"},{"type":"user_nicename","display_name":"Li Zhao","user_id":36152,"people_section":"Related people","alias":"lizo"},{"type":"user_nicename","display_name":"Jiang Bian","user_id":38481,"people_section":"Related people","alias":"jiabia"}],"msr_research_lab":[199560],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":47,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988\/revisions"}],"predecessor-version":[{"id":1082442,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1052988\/revisions\/1082442"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1052988"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1052988"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1052988"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1052988"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1052988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}