{"id":8600,"date":"2020-08-25T06:05:17","date_gmt":"2020-08-25T13:05:17","guid":{"rendered":"http:\/\/www.microsoft.com\/garage-en-us\/?p=8600"},"modified":"2020-08-25T06:04:46","modified_gmt":"2020-08-25T13:04:46","slug":"mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage","status":"publish","type":"post","link":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/","title":{"rendered":"MIT students build Mosaic to explore art across cultures at Microsoft Garage"},"content":{"rendered":"

Art is one of the few languages which transcends barriers of country, culture, and time. Most people view art subjectively through a lens shaped by their experiences and environment. Finding commonalities among pieces from different eras and mediums calls for an open mind and a sharp eye. The Mosaic<\/a> project is taking this intellectual and human approach and augmenting it with the help of artificial intelligence to find connections between works of art from diverse artists. Working with data from The Rijksmuseum<\/a> \u2013 the Museum of the Netherlands and The Metropolitan Museum of Art<\/a> (The Met), a group of MIT students implemented a new algorithm for visual search that finds similarities across cultures in the museums\u2019 collections as part of their externship project at Microsoft Garage.<\/p>\n

 <\/p>\n

\"MosaicBased on a starting image of a Banyan, the algorithm finds similar works of art from different artists in the Rijksmuseum and The Met collections.<\/h5>\n

The MIT externship with Microsoft Garage takes a traditional semester-long internship experience and condenses it to a month of rapid development where students take an impactful project from concept to reality. Microsoft Garage at NERD was the home base for the externs during the month of January, where a culture of innovation is fostered, and growth mindset is put into action. Some of the externs have since returned to Microsoft as Garage Interns, as part of a virtual summer internship experience where students collaborate online through Microsoft Teams and other productivity tools.<\/p>\n

The idea started with long-time Garage collaborator and Mosaic project sponsor Mark Hamilton, Research Engineer pursuing a PhD at MIT alongside his full-time job on the Cognitive Services Research Group<\/a> at Microsoft\u2019s New England Research and Development Center (NERD)<\/a>. Mark was inspired by a visit to an exhibit at the Rijksmuseum. This exhibit compared works from the studios of Rembrandt and Velazquez, two prolific artists that never met, yet told similar stories in their works. He began thinking of possible ways the same idea could be enhanced by AI, and developed an algorithm which searches for similar works of art based on a starting image and a feature of that image such as the medium, a color, or the culture (e.g. \u201cFrench\u201d). Last January, Mark began working with The Met\u2019s Open Access<\/a> collection with the previous cohort of MIT externs, and out of that effort grew Gen Studio<\/a>, an interactive tool giving people the ability to generate new art from trained generative adversarial networks.<\/p>\n

This year\u2019s MIT students had a different goal with Mosaic<\/a>.<\/p>\n

 <\/p>\n

\"MosaicMosaic website exploring connections in art.<\/h5>\n

\u201cThis project leverages existing collections of art from both The Met and the Rijksmuseum to find connections between various cultures and time periods,\u201d Mark explained. \u201cWith Mosaic we can shine a light into areas of the collections that may not be as well-known and discover really interesting things.\u201d Curators and visitors can use Mosaic to explore art in new ways, such as charting trends that underlie works from different cultures and finding evidence of cultural exchanges and influences throughout history. One example Mark described was how the algorithm found similarities between a Ming Dynasty porcelain vase, a Dutch violin, and a banyan from the Netherlands. \u201cThe algorithm picked up on commonalities of form between the banyan and a string instrument, but also the color of the blue and white motif which originated in Ming Dynasty and spread through Europe. You can see evidence of cultural exchanges happening through the outputs of this algorithm.\u201d<\/p>\n

For a deeper dive into how this new way of conditional image retrieval can help improve GANs, Mark hosted a Microsoft Research Webinar<\/a> on the topic.<\/p>\n

Before the machine learning model could be implemented, however, there was the complex task of cleaning the data associated with each piece of art.<\/p>\n

Meet the team behind Mosaic<\/h3>\n

Students Mindren Lu and Stephanie Fu study computer science at MIT and worked closely to comb through data from both museums. No small feat, the two data scientists organized descriptive metadata like keywords, authors, and tags associated with each artwork into consistent datasets that could be fed into the algorithm. \u201cThere was a lot of manual processing since much of the data is written in another language, sometimes there were misspellings or missing information,\u201d said Stephanie. \u201cIt took a lot of time to boil all that down to a nice, formatted dataset that the back-end team could work with.\u201d<\/p>\n

Marina Rogers was part of the team building the back-end of Mosaic that interfaces with various Azure products and services. \u201cWe\u2019d bring the cleaned datasets and collections in and create the machine learning models based on Mark\u2019s research. Then, we\u2019d deploy the models and different APIs for sharing that connect to the front-end website,\u201d explained Marina, who is an electrical engineering and computer science student with a minor in design.<\/p>\n

\u201cBeing able to share our work with others is definitely a highlight,\u201d said Johnny Bui, also studying electrical engineering and computer science. He reflected on how getting technical pieces and services to \u201ctalk\u201d to each other in novel ways behind the scenes was a challenge. Felix Tran and Maggie Wang completed the team working on the back-end systems that provided uninterrupted end-to-end data flow, described by Jean-Yves Ntamwemezi, Software Engineer on the Microsoft Garage at NERD, as \u201cthe glue that brings the pieces together.\u201d<\/p>\n

Jean-Yves joined the Garage team just before the externship began, coming from a long tenure as a software engineer on the Office Docs team. \u201cIt was a very interesting, fast-paced learning environment, jumping in to lead the externs in many ways.\u201d In addition to general mentorship and support, Jean-Yves readied the Azure DevOps environment for the project, set up pull requests so the externs could have their code reviewed, and introduced them to the right tools and engineers with the expertise to help unblock progress. He also leads the Blacks at Microsoft employee resource group local chapter, building culture and leadership within the company.<\/p>\n

 <\/p>\n

\"MITMIT externs: Stephanie Fu, Mindren Lu, Ben Chen, Felix Tran, Darius Bopp, Maggie Wang, Marina Rogers, Johnny Bui<\/h5>\n

\u201cThe Garage is a perfect mix of working on exciting technologies and leading a team of engineers toward a solution,\u201d said Jean-Yves. \u201cInterns and new hires inform what the culture becomes, affecting what the work looks like, what teams look like, what the focus is in our working environment. Working in a technical space where empathy is a priority in your thought process \u2013 that\u2019s a place where I want to grow.\u201d For Jean-Yves, an important part of his day-to-day was making sure the students were building camaraderie throughout the process.<\/p>\n

Darius Bopp and Ben Chen were responsible for the front-end website that is a culmination of all the work the team performed packed into a visually engaging web experience that finds connections among art in new ways. \u201cThey took this nebulous design of a website and made it into something that looks great and is easy to navigate,\u201d said Mark. \u201cThe front-end team was so fast that we were able to spend a lot of time thinking about what was absolutely necessary on the site, making the website less complex and more intuitive.\u201d<\/p>\n

Visualizing machine learning<\/h3>\n

Though the data science team performed a lot of critical work with Mark\u2019s original algorithm, they didn\u2019t stop there. As a stretch goal, Mark introduced the SHAP (SHapley Additive exPlanations) algorithm to Mindren and Stephanie, a game theoretic approach to explain the output of any machine learning model. The two were immediately intrigued, delving further into the research and how to apply it to their ML model for Mosaic. \u201cSHAP is a tool that tries to explain why the ML model made the decisions that it did, why out of all of these paintings it chose this particular one as a match. ML models don\u2019t think like humans, even though it seems like they do,\u201d said Stephanie. What resulted is a fascinating visual representation of what the algorithm is looking at when it finds objects \u201csimilar\u201d.\u00a0 This can show what the model \u201csees\u201d\u2014 like a window into the brain of machine learning \u2014 but it takes extensive memory and space to run. For that reason, and because their externship was ending, the team could only implement a few rationales to show on the website as an example.<\/p>\n

 <\/p>\n

\"MachineMachine learning algorithm behind Mosaic identifying parts of pictured sculptures to query and find matches in other artworks.<\/h5>\n

Though the weekly development cycles went quickly, the externs had solid guidance from Mark throughout the entire process. Mark brings his deep appreciation for art to the work and enjoys applying his knowledge of AI to create new contexts in how art is explored and studied. \u201cSo many artists are trying to embody the metaphorical in the visual domain, and this algorithm just can\u2019t help but see that and pick up on it when it finds these matching works from all over the world,\u201d Mark explained. \u201cArt is aesthetic and beautiful, but a lot of times artists are trying to say really deep things about the nature of perception, of feelings and emotions.\u201d In most instances one could almost say the AI managed to extract the intent or idea of the paintings, not just the pixels, to find commonalities.<\/p>\n

Art also resonated in Marina\u2019s life as she grew up with a mother who studied fine art. Marina found the most important part of the experience was \u201cthe culture of improvement and personal growth. The overall environment was one of learning and encouragement.\u201d<\/p>\n

Johnny and his peers agreed the structure of the externship experience set them up for success. \u201cWorking on a project of this scale as well as interfacing with third party tools, and collaborating with other people, was a great learning experience. We had a running start on the project within the first couple of days,\u201d he explained, referring to the mini-hackathon that kicked off their experience at the\u00a0 Microsoft Garage where each person could try out what data science, back-end, and front-end development work on the project might look like.<\/p>\n

\u201cIt was my first real work experience in a big company on a fast-paced project team. It helped inform what direction I want to take in my career,\u201d said Mindren, who found the most exciting moments to be researching about the rationale tool and when he received those first matching artworks from their ML model. \u201cYou have a lot more fun than expected, and the culture was cool and super friendly.\u201d<\/p>\n

Explore more on the Mosaic<\/a> website<\/p>\n

Check out the Microsoft Research Webinar<\/a><\/p>\n

Read about 2019 MIT externship project Gen Studio<\/a><\/p>\n

Learn about the Microsoft Garage Internship Program<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Art is one of the few languages which transcends barriers of country, culture, and time. Most people view art subjectively through a lens shaped by their experiences and environment. Finding commonalities among pieces from different eras and mediums calls for an open mind and a sharp eye. The Mosaic project is taking this intellectual and human approach and augmenting it….<\/span><\/p>\n

CONTINUE READING \"MIT students build Mosaic to explore art across cultures at Microsoft Garage\"<\/span><\/a><\/p>","protected":false},"author":14,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[5,7],"tags":[11,70,116,15,115],"class_list":["post-8600","post","type-post","status-publish","format-standard","hentry","category-garage-general","category-garage-interns","tag-ai","tag-art","tag-externs","tag-machine-learning","tag-mit"],"acf":[],"yoast_head":"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage\" \/>\n<meta property=\"og:description\" content=\"Art is one of the few languages which transcends barriers of country, culture, and time. Most people view art subjectively through a lens shaped by their experiences and environment. Finding commonalities among pieces from different eras and mediums calls for an open mind and a sharp eye. The Mosaic project is taking this intellectual and human approach and augmenting it....\" \/>\n<meta property=\"og:url\" content=\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Garage\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-25T13:05:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-08-25T13:04:46+00:00\" \/>\n<meta name=\"author\" content=\"Meixia Huang\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@MSFTGarage\" \/>\n<meta name=\"twitter:site\" content=\"@MSFTGarage\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Meixia Huang\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/\",\"url\":\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/\",\"name\":\"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage\",\"isPartOf\":{\"@id\":\"http://approjects.co.za/?big=en-us/garage\/#website\"},\"datePublished\":\"2020-08-25T13:05:17+00:00\",\"dateModified\":\"2020-08-25T13:04:46+00:00\",\"author\":{\"@id\":\"http://approjects.co.za/?big=en-us/garage\/#\/schema\/person\/744bc20c9b6f354da52285115d0c58c1\"},\"breadcrumb\":{\"@id\":\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http://approjects.co.za/?big=en-us/garage\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"MIT students build Mosaic to explore art across cultures at Microsoft Garage\"}]},{\"@type\":\"WebSite\",\"@id\":\"http://approjects.co.za/?big=en-us/garage\/#website\",\"url\":\"http://approjects.co.za/?big=en-us/garage\/\",\"name\":\"Microsoft Garage\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http://approjects.co.za/?big=en-us/garage\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http://approjects.co.za/?big=en-us/garage\/#\/schema\/person\/744bc20c9b6f354da52285115d0c58c1\",\"name\":\"Meixia Huang\",\"url\":\"http://approjects.co.za/?big=en-us/garage\/blog\/author\/meixia\/\"}]}<\/script>","yoast_head_json":{"title":"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/","og_locale":"en_US","og_type":"article","og_title":"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage","og_description":"Art is one of the few languages which transcends barriers of country, culture, and time. Most people view art subjectively through a lens shaped by their experiences and environment. Finding commonalities among pieces from different eras and mediums calls for an open mind and a sharp eye. The Mosaic project is taking this intellectual and human approach and augmenting it....","og_url":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/","og_site_name":"Microsoft Garage","article_published_time":"2020-08-25T13:05:17+00:00","article_modified_time":"2020-08-25T13:04:46+00:00","author":"Meixia Huang","twitter_card":"summary_large_image","twitter_creator":"@MSFTGarage","twitter_site":"@MSFTGarage","twitter_misc":{"Written by":"Meixia Huang","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/","url":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/","name":"MIT students build Mosaic to explore art across cultures at Microsoft Garage | Microsoft Garage","isPartOf":{"@id":"http://approjects.co.za/?big=en-us/garage\/#website"},"datePublished":"2020-08-25T13:05:17+00:00","dateModified":"2020-08-25T13:04:46+00:00","author":{"@id":"http://approjects.co.za/?big=en-us/garage\/#\/schema\/person\/744bc20c9b6f354da52285115d0c58c1"},"breadcrumb":{"@id":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/"]}]},{"@type":"BreadcrumbList","@id":"http://approjects.co.za/?big=en-us/garage\/blog\/2020\/08\/mit-students-build-mosaic-to-explore-art-across-cultures-at-microsoft-garage\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http://approjects.co.za/?big=en-us/garage\/"},{"@type":"ListItem","position":2,"name":"MIT students build Mosaic to explore art across cultures at Microsoft Garage"}]},{"@type":"WebSite","@id":"http://approjects.co.za/?big=en-us/garage\/#website","url":"http://approjects.co.za/?big=en-us/garage\/","name":"Microsoft Garage","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http://approjects.co.za/?big=en-us/garage\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"http://approjects.co.za/?big=en-us/garage\/#\/schema\/person\/744bc20c9b6f354da52285115d0c58c1","name":"Meixia Huang","url":"http://approjects.co.za/?big=en-us/garage\/blog\/author\/meixia\/"}]}},"_links":{"self":[{"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/posts\/8600"}],"collection":[{"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/comments?post=8600"}],"version-history":[{"count":0,"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/posts\/8600\/revisions"}],"wp:attachment":[{"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/media?parent=8600"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/categories?post=8600"},{"taxonomy":"post_tag","embeddable":true,"href":"http://approjects.co.za/?big=en-us/garage\/wp-json\/wp\/v2\/tags?post=8600"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}