{"id":266109,"date":"2016-07-27T15:36:18","date_gmt":"2016-07-27T22:36:18","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=266109"},"modified":"2016-08-12T08:33:43","modified_gmt":"2016-08-12T15:33:43","slug":"open-source-commitment-proof-projects","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/open-source-commitment-proof-projects\/","title":{"rendered":"Our open source commitment: The proof is in the projects"},"content":{"rendered":"

By Miran Lee, Principal Research Program Manager &\u00a0Winnie Cui, Senior Research Program Manager, Microsoft Research Asia<\/em><\/p>\n

Openness allows innovation to evolve in unforeseen, novel and exciting ways, and sometimes even provides solutions that no one ever imagined were possible.<\/p>\n

Getting more done with crowdsourcing<\/h2>\n

One such innovation is GeoMission (opens in new tab)<\/span><\/a> (geo-location-based mission), a crowdsourcing platform developed by MSRA and a team of researchers from the Hong Kong University of Science and Technology (HKUST). GeoMission lets users share and accept tasks based on where they are located.<\/p>\n

Users submit location-based requests via GeoMission apps, which then push questions to other users near the target location (as long as they meet any additional criteria in the request.)<\/p>\n

\"The

The project owner Professor Lei Chen from HKUST is introducing GeoMission to audience<\/p><\/div>\n

Developed for IOS and Android clients, the GeoMission server platform allows users to initiate requests by audio, video, photo or plain old texting.<\/p>\n

All of GeoMission\u2019s source code (opens in new tab)<\/span><\/a> is hosted on GitHub, providing some critical benefits for a research-based project \u2014 like more people! Researchers can intricately study how users interact with the platform, and users can directly contribute to help make it better. Of course, making it open source extends the tools to the greatest possible number of spatial crowdsourcing researchers. Most importantly, we believe opening the source code helps us innovate faster and provide more ways to collaborate with other developers or just about anyone else who\u2019s interested in the project. You can find more details about project at HKUST\u2019s website (opens in new tab)<\/span><\/a>.<\/p>\n

Improving datacenter efficiency with Vortex<\/h2>\n

In the same spirit of openness, we\u2019ve worked with Professor Byung-Gon Chun from Seoul National University (SNU) to develop Vortex (opens in new tab)<\/span><\/a> in an effort to address the problem of wasted resources at datacenters. Tapping these sometimes vast computing resources \u2014 that remain largely unused outside of peak usage \u2014 represents a huge opportunity to improve datacenter efficiency and save energy.<\/p>\n

Although current resource managers like Google\u2019s Borg system and Apache Mesos attempt to reclaim idle resources for other tasks, they largely fall short when reclaimed resources are inevitably preempted by latency critical tasks. The more aggressively the resources are reclaimed, the more frequently they\u2019re preempted due to conflict, resulting in transient resources. \u00a0The upshot of all this is that current data processing systems that rely on transient resources cannot efficiently complete jobs.<\/p>\n

Vortex, on the other hand, maintains high performance despite frequent preemptions. Developed by SNU grad students, Yunseong Lee and Youngseok Yang during their internship at MSRA, the pair are continuing to work on Vortex after returning to school. Joining the project is SNU undergraduate student Geon-Woo Kim along with contributors from other institutions and Microsoft.<\/p>\n

\"Vortex

Vortex team in SNU (from left to right); Geon-Woo Kim, Youngseok Yang, Byung-Gon Chun, and Yunseong Lee<\/p><\/div>\n

Experimental evaluations have been conducted on Microsoft Azure to measure the Vortex system\u2019s effectiveness.\u00a0The results show that Vortex can scale out much better with frequently preempted transient resources than Apache Spark. In certain cases, Apache Spark failed to complete jobs.<\/p>\n

Hosted on GitHub (opens in new tab)<\/span><\/a>, Vortex has been developed as an application of Apache REEF \u2014 an open source library for big data applications \u2014 in what has since proved to be a mutually beneficial project. \u00a0Vortex is succeeding in leveraging the Apache methods of growing open source projects: Development issues were openly discussed and pull requests were thoroughly reviewed. Meanwhile, the Apache REEF community was able to closely observe how Vortex uses Apache REEF as well as learn about the overall Vortex requirements.<\/p>\n

\"Vortex\"<\/p>\n

Vortex and GeoMission \u2014 as well as other projects like them \u2014 clearly have the potential to succeed in the marketplace. However, we believe that releasing them as open source opens the way to greater long term value for the global community of researchers and developers whose collaborative efforts can sometimes trigger unimaginable breakthroughs. At Microsoft Research Asia, we see a future that includes many more opportunities to collaborate with the open source community \u2014 to the benefit of all.<\/p>\n

Learn more<\/strong><\/p>\n