{"id":992553,"date":"2024-08-23T03:31:35","date_gmt":"2024-08-23T10:31:35","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=992553"},"modified":"2024-09-27T21:51:38","modified_gmt":"2024-09-28T04:51:38","slug":"compositional-3d-aware-video-generation","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/compositional-3d-aware-video-generation\/","title":{"rendered":"Compositional 3D-aware Video Generation"},"content":{"rendered":"\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-top is-style-default is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:100%\"><section class=\"mb-3 moray-highlight\">\n\t<div class=\"card-img-overlay mx-lg-0\">\n\t\t<div class=\"card-background  has-background-catalina-blue card-background--full-bleed\">\n\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR.png\" class=\"attachment-full size-full\" alt=\"background pattern\" style=\"\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR-240x135.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/MSR-640x360.png 640w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>\t\t<\/div>\n\t\t<!-- Foreground -->\n\t\t<div class=\"card-foreground d-flex mt-md-n5 my-lg-5 px-g px-lg-0\">\n\t\t\t<!-- Container -->\n\t\t\t<div class=\"container d-flex mt-md-n5 my-lg-5 \">\n\t\t\t\t<!-- Card wrapper -->\n\t\t\t\t<div class=\"w-100 w-lg-col-5\">\n\t\t\t\t\t<!-- Card -->\n\t\t\t\t\t<div class=\"card material-md-card py-5 px-md-5\">\n\t\t\t\t\t\t<div class=\"card-body \">\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n<h1 class=\"wp-block-heading has-text-align-left\" id=\"compositional-3d-aware-video-generation\">Compositional 3D-aware Video Generation<\/h1>\n\n\n\n<div class=\"wp-block-button is-style-pill\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/arxiv.org\/pdf\/2409.00558\">Paper (NeurIPS 2024)<\/a><\/div>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n<\/div>\n<\/div>\n\n\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-2 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_result\/1.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\nIn a Magician&#8217;s magical cabin alone in a serene forest, an alien walking on the floor, starting from the cabin&#8217;s door to the mow near the bottom right corner of this image.\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_result\/2.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\nFour characters stood on the stage. In front of the stage, a man and a woman are performing China Kung Fu and dancing respectively. On the right side of the stage, a skeleton man is dancing, and behind them, a clown is performing.\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_result\/6.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\nIn a long anime-style road with anime-blocks and little anime-grass, anime-houses and anime-tree on the side of the anime-style road, an alchemist is walking while swinging his arms.\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_result\/5.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\nIn an idyllic park with vibrant flowers, lush greenery, and a serene lake, a person with a curly beard and a black hoodie is walking with touching  on tiptoe.\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"abstract\">Abstract<\/h4>\n\n\n\n<p>Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video (e.g., scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"method\">Method<\/h4>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1120\" height=\"720\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework.png\" alt=\"graphical user interface, website\" class=\"wp-image-1079262\" style=\"width:828px;height:auto\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework.png 1120w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework-300x193.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework-1024x658.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework-768x494.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/framework-240x154.png 240w\" sizes=\"(max-width: 1120px) 100vw, 1120px\" \/><\/figure>\n\n\n\n<p>Our method consists of three stages: 1) The input textual prompt is decomposed into individual concepts by the LLM. Then we generate each concept in the form of 3D with the corresponding pre-trained expert model. 2) We leverage knowledge in multi-modal LLM to estimate the 2D trajectory of objects step-by-step. 3) After lifting the estimated 2D trajectory into 3D as initialization, we refine the scales, locations, and rotations of objects within the 3D scene using 2D diffusion priors.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"extension-object-editing\">Extension: Object Editing<\/h4>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-3 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_object\/appearedit1_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_object\/appearedit2_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_object\/appearedit3_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"extension-motion-editing\">Extension: Motion Editing<\/h4>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-4 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_motion\/motionedit1_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_motion\/motionedit2_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_motion\/motionedit3_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"extension-scene-editing\">Extension: Scene Editing<\/h4>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-5 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_bg\/sceneedit1_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_bg\/sceneedit2_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<iframe loading=\"lazy\" title=\"Video Example\" width=\"300\" height=\"300\" src=\"https:\/\/c3vv.github.io\/C3V\/ours_editing_bg\/sceneedit3_video.mp4\" frameborder=\"0\" scrolling=\"auto\"><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"comparisons\">Comparisons<\/h4>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"565\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1-1024x565.png\" alt=\"graphical user interface, website\" class=\"wp-image-1079415\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1-1024x565.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1-300x165.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1-768x424.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1-240x132.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/1.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">In a Magician&#8217;s magical cabin alone in a serene forest, an alien is walking on the floor, starting from the cabin&#8217;s door to the mow near the bottom right corner of this image.<\/figcaption><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"552\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5-1024x552.png\" alt=\"graphical user interface, website\" class=\"wp-image-1079418\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5-1024x552.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5-300x162.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5-768x414.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5-240x129.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/2-66c872f94f0a5.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Four characters stood on the stage. In front of the stage, a man and a woman are performing Kung Fu and dancing respectively. On the right side of the stage, a skeleton man is dancing, and behind them, a clown is performing.<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"ethics-statement\">Ethics Statement<\/h4>\n\n\n\n<p>C3V&nbsp;is exclusively a research initiative with no current plans for product integration or public access. We are committed to adhering to Microsoft AI principles during the ongoing development of our models. The model is trained on AI-generated content, which has been thoroughly reviewed to ensure that they do not include personally identifiable information or offensive content. Nonetheless, as these generated data are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the data to minimize the potential for the model to generate inappropriate content.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"acknowledgments\">Acknowledgments<\/h4>\n\n\n\n<p>Our work is based on the following excellent works: <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/luciddreamer-cvlab.github.io\/\">LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/alvinliu0.github.io\/projects\/HumanGaussian\">HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/motion-x-dataset.github.io\/\">Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h4 class=\"wp-block-heading has-text-align-center\" id=\"bibtex\">BibTeX<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>@article{zhu2024compositional,\n  title={Compositional 3D-aware Video Generation with LLM Director},\n  author={Zhu, Hanxin and He, Tianyu and Tang, Anni and Guo, Junliang and Chen, Zhibo and Bian, Jiang},\n  journal={Advances in Neural Information Processing Systems},\n  year={2024}\n}<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<p>This work is accomplished in Microsoft, April 2024.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n","protected":false},"excerpt":{"rendered":"<p>In a Magician&#8217;s magical cabin alone in a serene forest, an alien walking on the floor, starting from the cabin&#8217;s door to the mow near the bottom right corner of this image. Four characters stood on the stage. In front of the stage, a man and a woman are performing China Kung Fu and dancing [&hellip;]<\/p>\n","protected":false},"featured_media":1079220,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13562],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-992553","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Tianyu He","user_id":42141,"people_section":"Related people","alias":"tianyuhe"},{"type":"user_nicename","display_name":"Junliang Guo","user_id":41188,"people_section":"Related people","alias":"junliangguo"},{"type":"user_nicename","display_name":"Jiang Bian","user_id":38481,"people_section":"Related people","alias":"jiabia"}],"msr_research_lab":[],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/992553"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":52,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/992553\/revisions"}],"predecessor-version":[{"id":1088868,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/992553\/revisions\/1088868"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1079220"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=992553"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=992553"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=992553"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=992553"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=992553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}