Learning Commonsense Understanding through Language and Vision

As humans, we parse language and visual scenes — often together — into a rich understanding of what is going on in the world. Given even a still image or a sentence describing an event, like “my friends eating at a restaurant”, we can infer who’s doing what, where they’re at, and what might happen next. Though existing models seem strong at tasks involving language or vision, these models often struggle at combining these two modalities into such a unified commonsense understanding.

In this talk, I will cover two recent works that seek to help bridge this gap. First, I’ll introduce a model named PIGLeT that learns physical commonsense understanding by interacting with the world in simulation, and uses this knowledge to ground language. PIGLeT learns linguistic form and meaning, together, and outperforms text-to-text only models that are orders of magnitude larger. I’ll then introduce a model named MERLOT, which learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech. MERLOT learns the layered inferences that go beyond recognition at the level of individual scenes, and towards cognition-level reasoning that understands what is happening globally over time; it gets SOTA performance on 12 vision-and-language datasets.

I’ll conclude with a sketch of future directions for how we can better learn multimodal commonsense understanding, as well as on the social impacts of this work.

Speaker Details

Rowan Zellers is a 5th year PhD student at the University of Washington in Computer Science & Engineering, working with Yejin Choi and Ali Farhadi. He is interested in enabling machines to learn language, grounded in a commonsense understanding of the world. This line of research includes benchmarking, analysis, and model building, and spans natural language processing, computer vision, and artificial intelligence. His work has been funded by an NSF graduate fellowship, and has appeared in several media outlets, including Wired, the Washington Post, and the New York Times.

Series:
Microsoft Vision+Language Summer Talk Series
Date:
Speakers:
Rowan Zellers
Affiliation:
University of Washington in Computer Science & Engineering

Series: Microsoft Vision+Language Summer Talk Series