{"id":689814,"date":"2020-09-22T21:43:29","date_gmt":"2020-09-23T04:43:29","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=689814"},"modified":"2022-08-24T10:56:02","modified_gmt":"2022-08-24T17:56:02","slug":"project-florence-vl","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-florence-vl\/","title":{"rendered":"Project Florence-VL"},"content":{"rendered":"<section class=\"mb-3 moray-highlight\">\n\t<div class=\"card-img-overlay mx-lg-0\">\n\t\t<div class=\"card-background  has-background- card-background--full-bleed\">\n\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1920\" height=\"720\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720.png\" class=\"attachment-full size-full\" alt=\"Azure Florence project header: eye + language = AI represents visual-language learning\" style=\"object-position: 69% 53%\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720.png 1920w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720-300x113.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720-1024x384.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720-768x288.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720-1536x576.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/ProjAzureFlorence_DPA_header-v3_09-2020_1920x720-1600x600.png 1600w\" sizes=\"(max-width: 1920px) 100vw, 1920px\" \/>\t\t<\/div>\n\t\t<!-- Foreground -->\n\t\t<div class=\"card-foreground d-flex mt-md-n5 my-lg-5 px-g px-lg-0\">\n\t\t\t<!-- Container -->\n\t\t\t<div class=\"container d-flex mt-md-n5 my-lg-5 align-self-center\">\n\t\t\t\t<!-- Card wrapper -->\n\t\t\t\t<div class=\"w-100 w-lg-col-5\">\n\t\t\t\t\t<!-- Card -->\n\t\t\t\t\t<div class=\"card material-md-card py-5 px-md-5\">\n\t\t\t\t\t\t<div class=\"card-body \">\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n<h1 id=\"project-florence-vl\" class=\"h2\">Project Florence-VL<\/h1>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n<p>Humans perceive the world through many channels, such as images viewed by the eyes or voices heard by the ears. Though any individual channel might be incomplete or noisy, humans can naturally align and fuse information collected from multiple channels, in order to grasp the key concepts needed for a better understanding of the world.<\/p>\n\n\n\n<h3 id=\"so-what-is-florence-vl-about\">So, what is Florence-VL about?<\/h3>\n\n\n\n<p>One of the core aspirations in artificial intelligence is to develop algorithms that endow computers with an ability to effectively learn from multi-modality (or multi-channel) data. This data is similar to sights and sounds attained from vision and language that help humans make sense of the world around us. For example, computers could mimic this ability by searching the most similar images for a text query (or vice versa) and by describing the content of an image using natural language.<\/p>\n\n\n\n<p>Azure Florence-Vision and Language, short for Florence-VL, is launched to achieve this goal, where we aim to build new foundation models for Multimodal Intelligence. Florence-VL, as part of <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/florence-a-new-foundation-model-for-computer-vision\/\">Project Florence<\/a>, is funded by the Microsoft AI Cognitive Service team since 2020. Motivated by the strong demand from real applications and recent research progresses on computer vision, natural language processing, and vision-language understanding, we strive to advance the state of the art on vision-language modeling and develop the best computer vision technologies as part of our mission to empower everyone on the planet to achieve more.<\/p>\n\n\n\n<h3 id=\"our-journey-starts-from-vl-pre-training\">Our journey starts from VL pre-training<\/h3>\n\n\n\n<p>Recently, Vision-Language Pre-training (VLP) has shown great progress toward learning general-purpose multimodal representations. The most representative approach is to train large transformer-based models on massive image-text pair data in a self-supervised manner, such as predicting the masked elements based on their context. The cross-modal representations of the pretrained models can then be finetuned to adapt to various downstream vision-language tasks.<\/p>\n\n\n\n<p>There are tons of images on the web and social media that have annotated texts. These texts can be used as a &#8220;free&#8221; source of labels. There are also a large number of videos that have audio channels describing what happens in these videos, and this audio can be transcribed into text as labels as well. In addition to the benefit of not requiring manual data labeling, VLP has another important benefit of cross-modal knowledge distillation, where knowledge learned in one modality helps learning in a different modality.<\/p>\n\n\n\n<p>Along the journey, our team has developed a series of seminal works, including <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/uniter-learning-universal-image-text-representations\/\">UNITER<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/oscar-object-semantics-aligned-pre-training-for-vision-language-tasks\/\">OSCAR<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/large-scale-adversarial-training-for-vision-and-language-representation-learning\/\">VILLA<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/vinvl-making-visual-representations-matter-in-vision-language-models\/\">VinVL<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. These models, when equipped with large-scale pre-training, have helped us build up state-of-the-art techniques for challenging vision-language tasks. For example, in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/vivo-surpassing-human-performance-in-novel-object-captioning-with-visual-vocabulary-pre-training\/\">VIVO<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, we have achieved the first human parity on the novel image captioning (<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eval.ai\/web\/challenges\/challenge-page\/355\/leaderboard\/1011\" target=\"_blank\" rel=\"noreferrer noopener\">nocaps<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>) task. By enhancing pre-training with detected scene text in images, our <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/tap-text-aware-pre-training-for-text-vqa-and-text-caption\/\">TAP<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> model has also achieved No. 1 on the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eval.ai\/web\/challenges\/challenge-page\/906\/leaderboard\/2382\" target=\"_blank\" rel=\"noreferrer noopener\">TextCaps Challenge 2021<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<h3 id=\"how-to-further-modernize-our-florence-vl-efforts\">How to further modernize our Florence-VL efforts?<\/h3>\n\n\n\n<p>These successes are encouraging, which have driven us to further modernize our Florence-VL efforts, detailed below.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>End-to-end pre-training<\/strong>: Our previous efforts adopt transformers for multimodal fusion, while pre-trained object detectors on the vision side are used to extract regional features offline. Recently, end-to-end pre-training methods have become appealing and increasingly popular. To modernize our Florence-VL models, we have developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/ufo-a-unified-transformer-for-vision-language-representation-learning\/\">UFO<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/an-empirical-study-of-training-end-to-end-vision-and-language-transformers\/\">METER<\/a>, that systematically investigate how to train a performant multimodal transformer in an end-to-end manner.<\/li><li><strong>Scaling up<\/strong>: Scale is believed to be an important factor for the recent advances in VLP. For this purpose, we have developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/scaling-up-vision-language-pre-training-for-image-captioning\/\">LEMON<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/git-a-generative-image-to-text-transformer-for-vision-and-language\/\">GIT<\/a>, where GIT is our most recent multimodal generative foundation model trained on 800M image-text pairs. GIT achieves new state of the arts across 12 image\/video captioning and QA tasks, including the first human-parity on TextCaps. GIT achieves an accuracy of 88.79% on ImageNet-1k using a generative scheme. GIT can recognize logos, landmarks, characters, etc.<\/li><li><strong>Few-shot learning<\/strong>: Modern language models, such as GPT-3, has shown strong few-shot capabilities. That is, by providing only a few in-context examples, GPT-3 can achieve very strong performance on downstream language understanding tasks. However, such an ability is not seen in VLP yet. As a first step towards multimodal few-shot learning, we have developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/an-empirical-study-of-gpt-3-for-few-shot-knowledge-based-vqa\/\">PICa<\/a>, a simple but effective method that prompts GPT-3 via the use of image captions. We envision our work can inspire future studies to train a real multimodal GPT-3.<\/li><li><strong>Efficiency and robustness<\/strong>: In terms of efficiency, we have developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/minivlm-a-smaller-and-faster-vision-language-model\/\">MiniVLM<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/compressing-visual-linguistic-model-via-knowledge-distillation\/\">DistillVLM<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/playing-lottery-tickets-with-vision-and-language\/\">VL tickets<\/a> that aim to investigate the use of knowledge distillation and lottery ticket hypothesis for large VL model compression. Further, despite rapid progress of VLP, it remains unclear whether these state-of-the-art models are robust when encountering examples in the wild, which is critical for real-life applications. To this end, we have introduced a new <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/adversarial-vqa-a-new-benchmark-for-evaluating-the-robustness-of-vqa-models\/\">Adversarial VQA<\/a> benchmark, that we hope to shed new light on robustness study in the community and serve as a valuable benchmark for future work.<\/li><\/ul>\n\n\n\n<p>All our modernization endeavors are deeply connected with each other, with a same final goal in mind. For example, the model architectures developed in <strong>end-to-end pre-training<\/strong> can serve as building blocks for our multimodal foundation model. And the techniques we developed for <strong>scaling up VLP<\/strong> can be used for scaling our final solution. Further, we believe when the model is really scaled up, the capability of <strong>few-shot learning<\/strong> will emerge naturally. With a few light-weight adapters, and a few in-context examples, we envision that our final unified VL foundation model is capable of adapting to different tasks readily. Lastly, in order to deploy these state-of-the-art models into products, we also need to improve the <strong>efficiency and robustness<\/strong> of these large-scale models for fast and reliable use.<\/p>\n\n\n\n<h3 id=\"extension-to-video-language-pre-training\">Extension to video-language pre-training<\/h3>\n\n\n\n<p>So far, we\u2019ve been focusing on static images. There are a lot of videos and live cameras in the world, and many application scenarios require video understanding. We have extended our image-text pre-training methods to the video domain, and leverage large amounts of video-text data to learn image and motion features for diverse video-language tasks. Along the journey, we have explored several directions to pursue state-of-the-art video-language research.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><b>Learning from multi-channel videos<\/b>: \u00a0Videos are multi-channeled in nature, usually containing two or more modalities from vision (video frames), text (subtitle text), speech, and non-speech audio. To enrich the video representations for video-language tasks, we have developed\u00a0<a title=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/hero-hierarchical-encoder-for-videolanguage-omni-representation-pre-training-2\/\" aria-label=\"Link HERO\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/hero-hierarchical-encoder-for-videolanguage-omni-representation-pre-training-2\/\" target=\"_blank\" rel=\"noopener noreferrer\">HERO<\/a>,\u00a0that learns multi-channel video representations from both video frames and their accompanying subtitles with large-scale video-text corpus. Although our model design focuses on multi-channel videos, HERO can be generalized to different video types (multi- and single-channel).<\/li><li><b>End-to-end pre-training<\/b>: Traditional video-language methods often adopt a two-stage approach with offline video feature extraction, followed by the training of video-language fusion models. To remedy the disconnections in video domain and training objectives between video feature extractor and video-language fusion model, we introduced\u00a0<a title=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/less-is-more-clipbert-for-video-and-language-learning-via-sparse-sampling\/\" aria-label=\"Link ClipBERT\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/less-is-more-clipbert-for-video-and-language-learning-via-sparse-sampling\/\" target=\"_blank\" rel=\"noopener noreferrer\">ClipBERT<\/a>,\u00a0in which a CNN+transformer model operates on top of sparsely sampled frames and is optimized in an end-to-end fashion to solve popular video-language tasks.\u00a0<a title=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/violet-end-to-end-video-language-transformers-with-masked-visual-token-modeling\/\" aria-label=\"Link VIOLET\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/violet-end-to-end-video-language-transformers-with-masked-visual-token-modeling\/\" target=\"_blank\" rel=\"noopener noreferrer\">VIOLET<\/a>\u00a0and\u00a0<a title=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/swinbert-end-to-end-transformers-with-sparse-attention-for-video-captioning\/\" aria-label=\"Link SwinBERT\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/swinbert-end-to-end-transformers-with-sparse-attention-for-video-captioning\/\" target=\"_blank\" rel=\"noopener noreferrer\">SwinBERT<\/a>, further enhance ClipBERT by introducing Masked Visual-token Modeling and Sparse Attention. More recently, we have developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/lavender-unifying-video-language-understanding-as-masked-language-modeling\/\">LAVENDER<\/a>, where Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks. Such unification leads to a simplified model architecture, and surprisingly, it provides strong performance on a wide range of benchmarks, covering video question answering, text-to-video retrieval and video captioning.<\/li><li><b>Benchmarking video-language models<\/b>: Unlike image-text pretrained models, video-language models are often evaluated on their own choices of tasks, datasets and video domains, making it difficult to have a comprehensive comparison across different models. To facilitate the advances in video-language research, we released\u00a0<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/value-a-multi-task-benchmark-for-video-and-language-understanding-evaluation\/\">VALUE<\/a>, a comprehensive multi-task video-and-language understanding evaluation benchmark. VALUE includes diverse and challenging video-language tasks built on multi-channel videos, covering a broad range of video genres, video lengths, data volumes. We hope VALUE can inspire active research and discussion in the community.<\/li><\/ul>\n\n\n\n<h3 id=\"unification-of-localization-and-vl-understanding\">Unification of localization and VL understanding<\/h3>\n\n\n\n<p>We ask how language can play a more fundamental role for core vision tasks (such as image classification and object detection (OD)) and aim to unify both vision tasks and vision-language (VL) tasks into one general framework and enable computer vision in the wild via using natural language as a general-purpose interface. To this end, we have developed several models along the way.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Grounded Language-Image Pre-training<\/strong>: In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/grounded-language-image-pre-training\/\">GLIP<\/a>, we have unified object detection and phrase grounding for grounded pre-training, and this unification allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model. In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/glipv2-unifying-localization-and-vision-language-understanding\/\">GLIPv2<\/a>, we have tried to further unify localization and VL understanding, which not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Besides the use of standard OD heads (e.g., <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2106.08322\" target=\"_blank\" rel=\"noreferrer noopener\">DyHead<\/a>) for outputting bounding boxes as used in GLIP and GLIPv2, in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/crossing-the-format-boundary-of-text-and-boxes-towards-unified-vision-language-modeling\/\">UniTAB<\/a>, we have tried to unify text generation and bounding box prediction into a single transformer encoder-decoder architecture via representing each bounding box as a few discrete tokens, inspired by <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2109.10852\" target=\"_blank\" rel=\"noreferrer noopener\">Pix2seq<\/a>. This enables UniTAB to approach different VL tasks with a single set of parameters.<\/li><li><strong>Coarse-to-fine vision-language pre-training<\/strong>: In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/glipv2-unifying-localization-and-vision-language-understanding\/\">FIBER<\/a>, we provide another elegant solution that aim to tackle both localization and VL understanding tasks. FIBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones, bringing gains in terms of memory and performance. In addition, unlike previous work that is either only pre-trained on image-text data or on fine-grained data with box-level annotations, we present a two-stage pre-training strategy that uses both these kinds of data efficiently: (i) coarse-grained pre-training based on image-text data; followed by (ii) fine-grained pre-training based on image-text-box data.\u00a0<\/li><li><strong>External knowledge<\/strong>: In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/glipv2-unifying-localization-and-vision-language-understanding\/\">K-LITE<\/a>, we provide a simple strategy to leverage external knowledge to build transferable visual systems: In training, it enriches entities in natural language with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that can understand both visual concepts and their knowledge; In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models.  We study the performance of K-LITE on both image classification and object detection tasks.\u00a0<\/li><li><strong>Others<\/strong>: In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/dap-detection-aware-pre-training-with-weak-supervision\/\">DAP<\/a>,\u00a0we have developed detection-aware pre-training, which leverages only weakly-labeled classification-style datasets (e.g., ImageNet) for pre-training, but is specifically tailored to benefit OD tasks. In our <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/end-to-end-semi-supervised-object-detection-with-soft-teacher\/\">Soft Teacher<\/a> project, we aim to enhance semi-supervised end-to-end OD training via gradually improving pseudo label qualities during the curriculum, and show that the more and more accurate pseudo labels in turn benefit OD training. Besides new methods, we have also developed a new task, called <a style=\"font-size: 18px\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/ovis-open-vocabulary-visual-instance-search-via-visual-semantic-aligned-representation-learning-2\/\">Open-Vocabulary Visual Instance Search (OVIS)<\/a>. Given a textual search query, OVIS aims to return a ranked list of visual instances, i.e., image patches, that satisfies the search intent from an image database. Further, we leverage massive image-caption pairs as weak image-level supervision to tackle this new task.<\/li><\/ul>\n\n\n\n<h3 id=\"visual-synthesis\">Visual synthesis<\/h3>\n\n\n\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/nuwa-infinity-autoregressive-over-autoregressive-generation-for-infinite-visual-synthesis\/\">NUWA-Infinity<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> is our new multimodal generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos. An autoregressive over autoregressive generation mechanism is proposed to deal with this variable-size generation task, where a global patch-level autoregressive model considers the dependencies between patches, and a local token-level autoregressive model considers dependencies between visual tokens within each patch. Compared to DALL-E, Imagen and Parti, NUWA-Infinity can generate high-resolution images with arbitrary sizes and support long-duration video generation additionally. Check our project website&nbsp;<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/nuwa-infinity.microsoft.com\/\">here<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n\n\n<p>Vision-Language Pre-training (VLP) has recently attracted rapidly growing attention from both the computer vision and NLP communities, especially due to the emergence of multimodal foundation models like CLIP, DALL-E, CoCa, and Flamingo. To summarize the most recent advances at the frontier of VLP research, at CVPR 2022, we organized a tutorial on this increasingly important topic, where we have discussed the recent advances on (1) VLP for image-text tasks, such as visual question answering, image captioning and retrieval, and visual grounding; (2) VLP for core vision tasks, such as image classification and object detection; (3) VLP for video-text tasks, such as video question answering, captioning and retrieval; and (4) VLP for text-to-image\/video synthesis. <\/p>\n\n\n\n<p>All our tutorial slides and video recordings can be found at our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/vlp-tutorial.github.io\/\">tutorial website<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. <\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n\n\n<p>Please have a check of our blog post <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/techcommunity.microsoft.com\/t5\/azure-ai-blog\/apps-can-now-narrate-what-they-see-in-the-world-as-well-as\/ba-p\/1667146\" target=\"_blank\" rel=\"noreferrer noopener\">here<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> for a detailed description of related services and apps.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Microsoft Azure Florence-VL aims to develop state-of-the-art vision-language learning technologies to endow computers with an ability to effectively learn from multi-modality data.<\/p>\n","protected":false},"featured_media":694923,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13562],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-689814","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[781243,803152,717262,864669,792371,803158,738088,801448,803167,738103,802084,804010,751855,802099,804583,751873,802105,847417,751966,802111,655836,851014,779947,802117,695106,852624,780745,802126,695775,864615,780763,802132,699919,864648,780865,802141,699946,864657],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[{"id":0,"name":"Related services & apps","content":"Please have a check of our blog post <a href=\"https:\/\/techcommunity.microsoft.com\/t5\/azure-ai-blog\/apps-can-now-narrate-what-they-see-in-the-world-as-well-as\/ba-p\/1667146\">here<\/a> for a detailed description of related services and apps.\r\n\r\n&nbsp;"}],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Faisal Ahmed","user_id":31810,"people_section":"Azure Cognitive Services Research","alias":"fiahmed"},{"type":"guest","display_name":"Ehsan Azarnasab","user_id":693762,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"user_nicename","display_name":"Jianfeng Gao","user_id":32246,"people_section":"Azure Cognitive Services Research","alias":"jfgao"},{"type":"guest","display_name":"Xuedong  Huang","user_id":803173,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Linjie Li","user_id":786499,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Lin Liang","user_id":694149,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"user_nicename","display_name":"Kevin Lin","user_id":39694,"people_section":"Azure Cognitive Services Research","alias":"keli"},{"type":"guest","display_name":"Ce  Liu","user_id":803179,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Yumao Lu","user_id":694269,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Jianfeng Wang","user_id":693753,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"user_nicename","display_name":"Lijuan Wang","user_id":32680,"people_section":"Azure Cognitive Services Research","alias":"lijuanw"},{"type":"guest","display_name":"Zhengyuan Yang","user_id":786502,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Michael  Zeng","user_id":864204,"people_section":"Azure Cognitive Services Research","alias":""},{"type":"guest","display_name":"Yue Cao","user_id":803311,"people_section":"Collaborators","alias":""},{"type":"user_nicename","display_name":"Xiyang Dai","user_id":40384,"people_section":"Collaborators","alias":"xidai"},{"type":"guest","display_name":"Li Dong","user_id":803317,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Baining  Guo","user_id":803242,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Han Hu","user_id":803305,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Houdong  Hu","user_id":803350,"people_section":"Collaborators","alias":""},{"type":"user_nicename","display_name":"Lei Ji","user_id":32636,"people_section":"Collaborators","alias":"leiji"},{"type":"guest","display_name":"Chung-Ching  Lin","user_id":803296,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Furu Wei","user_id":803332,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Chenfei Wu","user_id":825022,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Bin  Xiao","user_id":803272,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Jianwei Yang","user_id":803263,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Lu Yuan","user_id":803278,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Pengchuan Zhang","user_id":803248,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Zheng  Zhang","user_id":803338,"people_section":"Collaborators","alias":""},{"type":"guest","display_name":"Luowei Zhou","user_id":803284,"people_section":"Collaborators","alias":""}],"msr_research_lab":[],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/689814"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":53,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/689814\/revisions"}],"predecessor-version":[{"id":877395,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/689814\/revisions\/877395"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/694923"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=689814"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=689814"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=689814"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=689814"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=689814"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}