{"id":170593,"date":"2010-11-08T14:23:00","date_gmt":"2010-11-08T14:23:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/project\/sho-the-net-playground-for-data\/"},"modified":"2019-08-19T15:20:14","modified_gmt":"2019-08-19T22:20:14","slug":"sho-the-net-playground-for-data","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/sho-the-net-playground-for-data\/","title":{"rendered":"Sho: The .NET Playground for Data"},"content":{"rendered":"<p><span id=\"a4df1e13-a82a-4501-971c-adefedc58f59\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagea4df1e13-a82a-4501-971c-adefedc58f59\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-bannerforiguana.png\" alt=\"\" \/><span id=\"ImageCaptiona4df1e13-a82a-4501-971c-adefedc58f59\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><br \/>\nSho is an interactive environment for data analysis and scientific computing that lets you seamlessly connect scripts (in IronPython) with compiled code (in .NET) to enable fast and flexible prototyping. The environment includes powerful and efficient libraries for linear algebra as well as data visualization that can be used from any .NET language, as well as a feature-rich interactive shell for rapid development. Sho is available under the following licenses: <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/\/sho-sholicenseagreement_2_6_2013.rtf\" target=\"_self\" rel=\"noopener noreferrer\">Sho for IronPython 2.6 license<\/a>\u00a0and\u00a0<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/\/sho-sholicenseagreement_2_7_2013.rtf\" target=\"_self\" rel=\"noopener noreferrer\">Sho for IronPython 2.7 license<\/a>.\u00a0See the Installers page for more details about the two versions.<\/p>\n<h1>Links<\/h1>\n\t\t\t<div class=\"moray-accordion\" data-show=\"false\">\n\t\t\t\t\t\t\t<div class=\"clearfix\">\n\t\t\t\t\t<div\n\t\t\t\t\t\tclass=\"btn-group align-items-center mb-g float-sm-right\"\n\t\t\t\t\t\trole=\"group\"\n\t\t\t\t\t\tdata-mount=\"collapse-controls\"\n\t\t\t\t\t\tdata-bi-aN=\"accordion-collapse-controls\"\n\t\t\t\t\t\tdata-target=\"#accordion-1\">\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"expand\" type=\"button\" data-bi-cN=\"Expand all\">Expand all<\/button>\n\t\t\t\t\t\t<span aria-hidden=\"true\"> | <\/span>\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"collapse\" type=\"button\" data-bi-cN=\"Collapse all\">Collapse all<\/button>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<ul class=\"accordion\" id=\"accordion-1\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-2\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-2\">\n\t\tInstallers\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-2\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p>There are two versions of Sho\u00a0available; one built on IronPython 2.6 and one built on IronPython 2.7.\u00a0 Depending on your usage scenario, you will want one or the other; the information below will help you decide.<\/p>\n<h2>Sho 2.1 on IronPython 2.7*<\/h2>\n<p>This is the version you&#8217;ll want if you\u00a0plan to use the Python Tools for Visual Studio IDE for Sho, and\/or you want the latest version of IronPython (2.7). Note that since this latest version of IronPython is distributed from codeplex under a different license, this version of Sho does not ship with the IronPython bits; you&#8217;ll have to <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/ironpython.net\/\" target=\"_self\" rel=\"noopener noreferrer\">install\u00a0IronPython 2.7\u00a0(from codeplex)<\/a> separately before installing Sho.\u00a0An advantage of installing IronPython in this way is that the Python standard libraries will be available to you as well. Note that if you&#8217;re planning to use PTVS you need to uncheck the option in the 2.7 installer for the Visual Studio components; that option conflicts with PTVS.<\/p>\n<p><a href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?displaylang=en&FamilyID=0c841620-8e8e-48d7-9317-2caffe7b9397\"><strong>Installer for Sho 2.1 on IronPython 2.7<\/strong><\/a> (requires separate <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/ironpython.net\/\" target=\"_self\" rel=\"noopener noreferrer\">IronPython 2.7<\/a> install)<\/p>\n<h2>Sho 2.1 on IronPython 2.6*<\/h2>\n<p>This version is fully self-contained, in that the essentials of IronPython 2.6\u00a0are included (just binaries, not the standard libraries; if you\u00a0want the latter you&#8217;ll need to do an install of IronPython 2.6 from codeplex and point your Sho path to the libraries).\u00a0This is the version you&#8217;ll want if the IronPython 2.7 license on codeplex is not compatible with your needs.\u00a0 This is also the version you&#8217;ll want if you want to use the HPC or Azure packages for Sho &#8211; we&#8217;ll be upgrading those in the near future to work with both versions, but for now they will only work with the IronPython 2.6 version.<\/p>\n<p><a href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?displaylang=en&FamilyID=0a776903-d279-4477-85ef-a4bc36e7ceb7\"><strong>Installer for Sho 2.1 on IronPython 2.6<\/strong><\/a> (self-contained)<\/p>\n<h2>Requirements<\/h2>\n<p>OS: Windows 7, Server 2008, Vista, or Server 2003.Dependencies (Sho 2.1 for IronPython 2.6): .NET 4.0 or higher.Dependencies (Sho 2.1 for IronPython 2.7): .NET 4.0 or higher, IronPython 2.7.<\/p>\n<p>*This library includes parts of the Intel Math Kernel Library. For more information about the Intel Math Kernel Library, please visit <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/software.intel.com\/en-us\/intel-mkl\/\">http:\/\/software.intel.com\/en-us\/intel-mkl\/<\/a> .<\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-3\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-3\">\n\t\tIDE (Python Tools for Visual Studio)\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-3\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<h2>Using Sho with Python Tools for Visual Studio (PTVS)<\/h2>\n<\/p>\n<p>If you&#8217;re looking for a great integrated development environment (IDE) for Sho, Python Tools for Visual Studio (PTVS) is a perfect fit.\u00a0 PTVS is a plugin for Visual Studio 2010; you can get the free VS Shell if you don&#8217;t have Visual Studio already. PTVS\u00a0gives you an interactive\u00a0REPL\u00a0with intellisense,\u00a0great editing capabilities such\u00a0syntax highlighting, automatic indents, intellisense, etc., as well as Python debugging, with some project management features to boot.\u00a0 A screenshot of using Sho\u00a0with PTVS is below.<\/p>\n<p><span id=\"cf16b28b-c4da-43f7-874b-64f1b1a1759d\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagecf16b28b-c4da-43f7-874b-64f1b1a1759d\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-ptvs_screenshot.jpg\" alt=\"\" \/><span id=\"ImageCaptioncf16b28b-c4da-43f7-874b-64f1b1a1759d\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Downloading and Installing PTVS and Sho<\/h2>\n<ol>\n<li>You&#8217;ll first need to install IronPython 2.7 from the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/ironpython.net\/\" target=\"_self\" rel=\"noopener noreferrer\">IronPython page<\/a>.\u00a0 When running the installer, be sure to uncheck the &#8220;Visual Studio&#8221; options; those conflict with PTVS.\u00a0 Note that PTVS debugging\/interactive windows will not work properly with earlier versions of IronPython (pre 2.7) &#8211; you can still edit code and do .net debugging, but for the full experience you&#8217;ll want IronPython 2.7.<\/li>\n<li>Next install Sho 2.0.5 (the version targeted to IronPython 2.7).\u00a0You&#8217;ll need to first uninstall any previous Sho installations.<\/li>\n<li>Download and install the\u00a0PTVS plugin from the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/pytools.codeplex.com\/\" target=\"_self\" rel=\"noopener noreferrer\">PTVS project page<\/a>. Note that if you don&#8217;t have Visual Studio 2010 already, you can see various means of getting it (including several free options) in the &#8220;Getting Visual Studio&#8221; section of the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/pytools.codeplex.com\/wikipage?title=Installation - details&referringTitle=Home\" target=\"_self\" rel=\"noopener noreferrer\">PTVS\u00a0install page<\/a>.<\/li>\n<li>Start Visual Studio, and go to Tools->Options->Python Tools->Interpreter Options. Set the &#8220;Default Interpreter&#8221; to be &#8220;IronPython 2.7.&#8221;\u00a0 If you&#8217;re on a 64-bit machine and want to use all of your memory for Sho computations, set it to &#8220;IronPython 64-bit 2.7&#8221;<\/li>\n<\/ol>\n<h2>Setting up Sho for PTVS<\/h2>\n<p>If you&#8217;re a regular Sho user, you&#8217;ll probably want to add the importing of the Sho libraries directly into the startup of the PTVS. To do this, you&#8217;ll first create a startup file, which I like to call &#8220;shoptvs.py&#8221; &#8211; the contents are below:<\/p>\n<pre># shoptvs.py - Sho startup file for PTVS<\/pre>\n<pre>import syssys.path.append(\"c:\/Program Files\/\/sho 2.0\/\/sho\")from sho import *<\/pre>\n<p>Note that you&#8217;ll have to set the path to point to the &#8220;shoshodir&#8221;, which you can find by evaluating shoshodir in the Sho console. To specify this as the startup file for the IronPython interpreter, go to Tools->Options->Python Tools->Interactive Windows, and under &#8220;Startup Script&#8221; specify the full path to shoptvs.py (or whatever you chose to call it). If you don&#8217;t want to create\/use this startup file, you can manually execfile shoptvs.py or just include the lines above in any scripts from which you want to use Sho commands or libraries.<\/p>\n<p>You can now click on Tools->Python Tools->IronPython 64-bit 2.7 Interactive, and you&#8217;ll get a new REPL with all the Sho commands at your fingertips.<\/p>\n<h2>Debugging Sho code from PTVS<\/h2>\n<p>There are two styles of debugging for IronPython &#8211; Python debugging and .NET debugging. Python debugging lets you look inside Python classes and examine variables, just as you would with pdb, but doesn&#8217;t let you step into .NET code. .NET debugging does let you step into and examine .NET code, but gives you a more limited Python debugging experience &#8211; for instance, members of Python classes will appear as they do in .NET, as part of dictionary, as opposed to the class structure you&#8217;d see in Python debugging. In short, if you&#8217;re mostly debugging Python code, you&#8217;ll likely want Python debugging; if you&#8217;re mostly debugging C# (or mixed mode code), you&#8217;ll want to use .NET debugging.<\/p>\n<p>PTVS is focused on Python debugging; if you want to do .NET debugging see &#8220;Using PTVS with the Sho Console&#8221; below. To do Python debugging, simply set breakpoints in your code (F9) and then run commands from the Python Interactive window; when your breakpoints are hit, execution will pause, and you can investigate variables, step through, etc.<\/p>\n<h2>Using PTVS with the Sho Console<\/h2>\n<p>If you&#8217;re doing mixed mode (Python and C#) development or just prefer the Sho Console for your interactive sessions, you can still take advantage of PTVS&#8217;s fantastic editor as well as .NET debugging. As described above, .NET debugging lets you set breakpoints in both Python and C# code, step into C# or Python code, etc., at the cost of not being able to look inside Python objects with the same convenience as Python debugging (above).<\/p>\n<p>To debug code in Sho Console, you&#8217;ll need to first attach to the process via Debug->Attach to Process in Visual Studio. This will bring up a list of processes and their IDs; pick the appropriate &#8220;shoconsole.exe&#8221; process. If you have more than one you can evaluate System.Diagnostics.Process.GetCurrentProcess().Id in the console to find out the Process ID for the one you wish to debug. You can then load up the relevant Python and C# files you wish to debug in Visual Studio, and set breakpoints wherever you wish. When you execute code in the console that results in one of these breakpoints being hit, execution will pause and control will go to Visual Studio; at this point you can step through code, step into code, etc., as you normally would in VS.<\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-4\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-4\">\n\t\tScreenshots and Examples\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-4\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<h2 align=\"left\">Sho Console showing matrix manipulation<\/h2>\n<\/p>\n<p align=\"left\"><span id=\"17dadf24-ac16-4619-9814-34efc874801b\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image17dadf24-ac16-4619-9814-34efc874801b\" title=\"Sho console\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-console.png\" alt=\"Sho console\" \/><span id=\"ImageCaption17dadf24-ac16-4619-9814-34efc874801b\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2 align=\"left\">&#8220;Intellisense&#8221;-like behavior<\/h2>\n<p align=\"left\">Available for class, method, and filename; activated by hitting <tab> in the console<\/tab><\/p>\n<p align=\"left\"><span id=\"2f6b9116-dcf8-4b3b-a7e0-28ba9b49de53\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image2f6b9116-dcf8-4b3b-a7e0-28ba9b49de53\" title=\"Command-completion example\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-consolewithfilecompletion.png\" alt=\"Command-completion example\" \/><span id=\"ImageCaption2f6b9116-dcf8-4b3b-a7e0-28ba9b49de53\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Using doc() to Inspect Objects<\/h2>\n<pre>>>> f = System.Windows.Forms.Form()>>> doc(f)<\/pre>\n<p><span id=\"bb0764af-5439-441c-ba24-8b709aaaa67e\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagebb0764af-5439-441c-ba24-8b709aaaa67e\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-docwindow.png\" alt=\"\" \/><span id=\"ImageCaptionbb0764af-5439-441c-ba24-8b709aaaa67e\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Plotting a Sine Wave<\/h2>\n<pre>>>> x = drange(0, 2*PI, 0.05)>>> plot(x, sin(x))<\/pre>\n<p><span id=\"2f9b6a95-639e-4f63-8c8f-85bb3269230a\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image2f9b6a95-639e-4f63-8c8f-85bb3269230a\" title=\"Sine wave graph\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-sinewave.png\" alt=\"Sine wave graph\" \/><span id=\"ImageCaption2f9b6a95-639e-4f63-8c8f-85bb3269230a\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Computing a Word Histogram<\/h2>\n<pre>>>> txt = System.IO.File.ReadAllText(\".\/declarationofindependence.txt\")>>> table = System.Collections.Hashtable()>>> for word in txt.split():    if table.ContainsKey(word):        table[word] += 1    else:        table[word] = 1>>> pairs = zip(table.Keys, table.Values)>>> elts = sorted(pairs, lambda a, b: a[1] < b&#091;1&#093;)&#091;:10&#093;<\/pre>\n<pre>>>> words, counts = zip(*elts)>>> bar(words, counts)<\/pre>\n<p><span id=\"46b19b28-768e-4707-b78a-155c3a6a2ac6\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image46b19b28-768e-4707-b78a-155c3a6a2ac6\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-wordhistogram.png\" alt=\"\" \/><span id=\"ImageCaption46b19b28-768e-4707-b78a-155c3a6a2ac6\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Creating a Simple GUI<\/h2>\n<pre>>>> f = System.Windows.Forms.Form()>>> b = System.Windows.Forms.Button()>>> f.Controls.Add(b)>>> b.Text = \"Hello World\">>> b.Location = System.Drawing.Point(250,100)>>> f.Width = 561>>> def printhello(sender, evtinfo):    print \"Hello World!\" >>> b.Click += printhello>>> f.ShowDialog()<\/pre>\n<p><span id=\"e7c8b0e6-4f52-4303-88e0-edcbd0bfc00e\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagee7c8b0e6-4f52-4303-88e0-edcbd0bfc00e\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-helloworldform.png\" alt=\"\" \/><span id=\"ImageCaptione7c8b0e6-4f52-4303-88e0-edcbd0bfc00e\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-5\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-5\">\n\t\tVideos\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-5\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<table style=\"height: 955px;\" width=\"747\" cellspacing=\"2\" cellpadding=\"2\">\n<tbody>\n<tr>\n<td style=\"vertical-align: middle;\"><span id=\"d8515c6e-1541-4f45-9324-c3318e3a5520\" class=\"ImageBlock fn\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-sql_demo.wmv\" target=\"_self\" rel=\"noopener noreferrer\"><img decoding=\"async\" id=\"Imaged8515c6e-1541-4f45-9324-c3318e3a5520\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-sql_demo.jpg\" alt=\"\" \/><\/a><span id=\"ImageCaptiond8515c6e-1541-4f45-9324-c3318e3a5520\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/td>\n<td style=\"vertical-align: top;\">\n<h2>Plotting\/Analyzing SQL Data<\/h2>\n<p>This is a short, canonical demo of doing Data-Centric Development from Sho: pulling data from an enterprise source (SQL), visualizing it, and doing some analysis on it. In this case, we fit lines and curves to data of population vs. area for various countries.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: middle;\"><span id=\"bae98195-171b-418a-8635-da889ffc6247\" class=\"ImageBlock fn\"><img loading=\"lazy\" decoding=\"async\" id=\"Imagebae98195-171b-418a-8635-da889ffc6247\" class=\"alignnone\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-speechbrowser.jpg\" alt=\"\" width=\"320\" height=\"238\" \/><span id=\"ImageCaptionbae98195-171b-418a-8635-da889ffc6247\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/td>\n<td style=\"vertical-align: top;\">\n<h2>Web Search with Speech<\/h2>\n<p>This demo shows how a tiny bit of Sho can be used to mix powerful platform components: in this case, we use\u00a0Windows&#8217; speech recognition engine to send queries to bing. [41 lines of code]<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: middle;\"><span id=\"8b67e304-4964-41fe-b0ee-5bd9feec445a\" class=\"ImageBlock fn\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-stickysorter_movie.wmv\" target=\"_self\" rel=\"noopener noreferrer\"><img decoding=\"async\" id=\"Image8b67e304-4964-41fe-b0ee-5bd9feec445a\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-stickysorter_demo.jpg\" alt=\"\" \/><\/a><span id=\"ImageCaption8b67e304-4964-41fe-b0ee-5bd9feec445a\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/td>\n<td>\n<h2>Sticky Sorter Prototype<\/h2>\n<p>This is an\u00a0application built in Sho that allows for arranging and editing sticky notes on a virtual canvas, a key functionality for user experience researchers organizing observation data.\u00a0It was an initial, fully-functioning\u00a0prototype\u00a0that was used as the basis for the\u00a0&#8220;StickySorter&#8221; application now available from\u00a0Office Labs.\u00a0[1000 lines of code]<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: middle;\"><span id=\"ec69e19b-f658-4096-8bd4-f105f0ca48af\" class=\"ImageBlock fn\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-dataoscope.wmv\" target=\"_self\" rel=\"noopener noreferrer\"><img decoding=\"async\" id=\"Imageec69e19b-f658-4096-8bd4-f105f0ca48af\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-surface_big.jpg\" alt=\"\" \/><\/a><span id=\"ImageCaptionec69e19b-f658-4096-8bd4-f105f0ca48af\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/td>\n<td>\n<h2>A Data Oscilloscope for Microsoft Surface<\/h2>\n<p>This is an example of using Sho as a &#8220;data oscilloscope,&#8221; i.e., to visualize the values of live data. in this case, values from a Microsoft Surface application are being shown and updated in real-time using Sho plot objects on a second monitor.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: middle;\"><span id=\"68ae62be-8c27-4c42-9189-e44e32c8234e\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image68ae62be-8c27-4c42-9189-e44e32c8234e\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-debugging_thumbnail.jpg\" alt=\"\" \/><span id=\"ImageCaption68ae62be-8c27-4c42-9189-e44e32c8234e\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/td>\n<td>\n<h2>Debugging Sho Code with Visual Studio 2010<\/h2>\n<p>This video shows how you can edit and debug Sho code with Visual Studio 2010 along with <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.ironpython.net\/tools\/\" target=\"_self\" rel=\"noopener noreferrer\">IronPython Tools<\/a>, available from IronPython.net.\u00a0 The video shows how to set breakpoints, step through code, inspect locals, and more. [highres version, 61MB]<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-6\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-6\">\n\t\tSho Package Installation Instructions\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-6\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<\/p>\n<p><strong>Azure (Cloud Computing)<\/strong> &#8211; [download] [documentation]<\/p>\n<p>Enables\u00a0parametric sweeps on Microsoft Azure directly from\u00a0the Sho console.\u00a0Python functions defined in modules on the local filesystem can be run remotely and return results to the local instance; relevant dependencies and data can be specified for more complex computations.<\/p>\n<p><strong>HPC (Cluster Computing)<\/strong> &#8211; [download] [documentation]<\/p>\n<p>Enables parametric sweeps on Microsoft HPC clusters directly from the Sho console.\u00a0Python functions defined in modules on the local filesystem can be run remotely and return results to the local instance; relevant dependencies and data can be specified for more complex computations.<\/p>\n<p><strong>Package installation instructions:<\/strong><\/p>\n<ol>\n<li>Save the .zip file from the packages page to a temporary directory<\/li>\n<li>Extract the contents of that zip file to your package directory. If you&#8217;re not sure where that is, you can find out from the Sho console by typing>>> shopackagedir<\/li>\n<li>Restart* your Sho console<\/li>\n<\/ol>\n<p>*Note: If you really don&#8217;t want to restart Sho, you can import the name of the package directory name; for instance, for the Optimizer package, import Optimizer.<\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-7\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-7\">\n\t\tHPC Package Documentation\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-7\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<h1>HPC Package Documentation<\/h1>\n<h2>Introduction<\/h2>\n<p>The HPC package lets you run Sho functions on an HPC cluster for parameter sweeps and related scenarios.\u00a0 In its most basic form, you specify a function which takes a couple of inputs, one of which will be the same for all remote instances, one of which will be one element of a list that you&#8217;ll also pass in.\u00a0 Advanced features let you specify data and output directories that will be copied to\/from the cluster, as well as the ability to reconstitute a past cluster session from its output directory.<\/p>\n<p>To use this package, you will have to have access to an HPC cluster, know the name of the headnode, and know\/have read\/write access to your personal working directory on the HPC cluster.\u00a0 Your cluster administrator should be able to give you all of this information.<\/p>\n<h2>Functions<\/h2>\n<ol>\n<li><b>session = clusterrun(cluster, fn, data, paramlist, inDir=None, outDir=None, includeFilter=None, verbose=False)<\/b>session = clusterrun(cluster, fn, data, paramlist, inDir=None, outDir=None, includeFilter=None, verbose=False)FN is a function (defined in some .py file) with two, three, or four arguments. On the cluster, when FN is called, the first argument will be DATA; the second argument will be an element from PARAMLIST. CLUSTERRUN finds the directory which contains the .py file in which FN is defined (we&#8217;ll refer to this as FNMODULE), then copies all .py and .dll files in that directory (and its subdirectories, recursively) to the cluster. Note that DATA and the elements of PARAMLIST must be of a type that is defined for a new instance of Sho that has only imported FNMODULE.INDIR is a local directory that contains additional data your function may need; if INDIR is defined, it will be copied to the cluster; the cluster-accessible location will be the third argument to FN.OUTDIR is a local directory; if specified, FN will get a fourth argument, a directory, to which the instance can write output information. When the command terminates, files in that directory will be copied back to OUTDIR on the local machine, in a subdirectory that is the session ID. Note that files written to this directory should have unique names for each instance, since they will all be copied to one directory.INCLUDEFILTER is a list of additional file types and names (beyond the default .py and .dll) that will be copied from the directory containing the module defining FN, as well as its subdirectories. Example: includeFilter=[&#8220;*.cs&#8221;, &#8220;*.pl&#8221;, &#8220;foo.txt&#8221;]SESSION is an object that identifies this job, and is an argument you can use to the clustergetresults() function once clusterrun has returned.\u00a0 SESSION also contains the working directory for the job (SESSION.WorkDir) as well as the session ID (SESSION.id)<\/li>\n<li><strong>cl = clustersetup(HEADNODE,WORKDIR)\u00a0<\/strong>Sets up a cluster for use, including creating the working directory and installing the current version of Sho. HEADNODE is a string specifying the name of the the head node; WORKDIR in the path to the working directory for your account on the cluster. The running version of Sho will be copied to your working directory in the directory &#8220;sho&#8221;, but if it is already there it will not be recopied.Returns a Cluster object cl, where cl.WorkDir is the working directory<\/li>\n<li><strong>session = clustersession(SESSIONDIR)<\/strong>Reconstitutes a session from a session working directory.\u00a0 For instance, if you run an experiment and want to look at the results later, you can save the working directory (SESSION.WorkDir) from a cluster execution, and then use clustersession() to reconstitute it and examine the results.\u00a0 clustersession does not require access to the cluster as long as you have the<\/li>\n<\/ol>\n<h2>Walkthrough<\/h2>\n<blockquote>\n<p><b># setup<\/b>>>> cl = clustersetup(&#8216;HEADNODENAME&#8217;,&#8217;\/\/FILESERVER\/YOURWORKDIR&#8217;)<strong># note that your actual working directory, &#8220;YOURWORKDIR&#8221; may have your username or other info tacked onto the base workdir, e.g., \/\/FILESERVER\/BASEWORKDIR\/LOCATION\/USERNAME; check with your system administrator for details.<\/strong><\/p>\n<p><b># write a function in a file, for instance, I have a function addstuff in file rctest.py<\/b>>>> addpath(&#8220;blah\/foo\/blah\/foo&#8221;)>>> import rctest>>> rctest.addstuff(3,4)7<\/p>\n<p><b># now run it on the cluster and get the results<\/b>>>> session = clusterrun(cl, rctest.addstuff, 10, [1,2,3,4,5])<a dialog pops up asking for your credentials; it then blocks until execution is complete>>>> session.getresults()Dictionary[int, object]({0 : 11, 1 : 12, 2 : 13, 3 : 14, 4 : 15})<\/a><\/p>\n<p><strong>#\u00a0explore the job&#8217;s working directory to look at\u00a0stdout and stderr and other output<\/strong>>>> session.Workdir&#8216;\/\/FILESERVER\/YOURWORKDIRsession-9dbd860b-e141-4dd0-9896-0cc470640ff7&#8217;<\/p>\n<p># <b>now let\u2019s say you\u2019re recovering a past experiment whose results are in some directory \u2013 note the dir doesn\u2019t have to be on the cluster, you could have copied it somewhere local to keep it around<\/b>>>> session = clustersession(&#8216;\/\/FILESERVER\/YOURWORKDIR\/session-eb821858-b00f-4e6b-8f90-76f92838c9d6&#8217;)>>> session.getdata()10>>> session.getparams()[1,2,3,4,5]>>> session.getresults()Dictionary[int, object]({0 : 11, 1 : 12, 2 : 13, 3 : 14, 4 : 15})<\/p>\n<\/blockquote>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-8\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-8\">\n\t\tAzure Package User Guide\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-8\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<h1>Azure Package User Guide<\/h1>\n<p>The Azure Parameter Sweep package is an easy-to-use package that allows you to parallelize embarrassingly-parallel code over a number of Sho instances running in Azure. It is especially well-suited to parameter sweeps.<\/p>\n<p><span id=\"45ef50ba-3005-4f2a-8267-b19baf343d3e\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image45ef50ba-3005-4f2a-8267-b19baf343d3e\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurecloud.png\" alt=\"\" \/><\/span><\/p>\n<h2>Installation and setup<\/h2>\n<p>In order to use the Azure package, your Azure administrator must first follow the directions in Admin Guide to get everything up and running on the cloud.<\/p>\n<p>To install the package, simple copy the package directory to your Sho packages directory.<\/p>\n<p><span id=\"36b93257-1457-47e6-998d-de5c975feed5\" class=\"ImageBlock fn\"><img loading=\"lazy\" decoding=\"async\" id=\"Image36b93257-1457-47e6-998d-de5c975feed5\" class=\"alignnone\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurepackagedir.png\" alt=\"\" width=\"711\" height=\"396\" \/><\/span><\/p>\n<p>In order to use the package, it will need to know a couple of pieces of information about the Azure environment the Sho instances are running on. The following environment variables should contain the authentication information for your deployment. Ask your administrator for the values to set:<\/p>\n<blockquote>\n<blockquote>\n<pre>SHOAZUREACCOUNT = <your Azure storage account name><\/your><\/pre>\n<pre>SHOAZUREKEY = <the primary key for the storage account><\/the><\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>Usage<\/h2>\n<p>The parameter sweep package uses work functions that take two arguments: one \u201cdata\u201d and one \u201cparameter\u201d value:<\/p>\n<blockquote>\n<pre>def workfn(data, param): ...<\/pre>\n<\/blockquote>\n<p>When you tell the package to run your work function in the cloud, it uses a single \u201cdata\u201d value common to all of the various instances, and a list of parameter values, which are distributed among them. Here is an example showing how to use the package to distribute a very simple work function over a list of parameter values:<\/p>\n<h2>Simple example<\/h2>\n<p>For this simple example, we are going to use a very simple predefined work function that simply adds the data value to the parameter value.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> import paramsweep # import the paramsweep package<\/pre>\n<pre>>>> paramsweep.addDemoDir() # this adds the path to the test functions<\/pre>\n<pre>>>> import paramsweeptest # paramsweeptest contains some demo functions<\/pre>\n<pre>>>> session = paramsweep.run(paramsweeptest.add, 100, [1,2,3,4,5])<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>The run function returns a parameter sweep session object:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> print session<\/pre>\n<pre>ParamSweepSession('2c574185-9cd3-4ba1-a6f3-9668424374a8')<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>This session runs asynchronously on the cloud, leaving your Sho instance running so you can do work while you wait for it to finish. The isDone method will tell you if the session has finished. If it is, we can call getResults to get the results from each of the jobs. Alternately we can call the waitResults function, which blocks until it finishes, and then return the result values.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.isDone()<\/pre>\n<pre>True<\/pre>\n<pre>>>> session.getResults()<\/pre>\n<pre>[101, 102, 103, 104, 105]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>It is good practice to delete the server-side data associated with this session when you\u2019re done.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.cleanup()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Of course, you aren\u2019t limited to using integers as your data or parameter values:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> a = rand(4,4)\u00a0 # This is a Sho array<\/pre>\n<pre>>>> session = paramsweep.run (paramsweeptest.multiply, a, [1, 3, 5])<\/pre>\n<pre>>>> for x in session.waitResults(): print x,\"n\"<\/pre>\n<pre>[ 0.4669 0.1889 0.4706 0.3542<\/pre>\n<pre>0.7747 0.0344 0.3692 0.8417<\/pre>\n<pre>0.4836 0.3800 0.0677 0.2211<\/pre>\n<pre>0.6412 0.3844 0.4372 0.1443]<\/pre>\n<pre>[ 1.4006 0.5668 1.4117 1.0625<\/pre>\n<pre>2.3240 0.1033 1.1076 2.5252<\/pre>\n<pre>1.4509 1.1399 0.2030 0.6633<\/pre>\n<pre>1.9235 1.1532 1.3116 0.4329]<\/pre>\n<pre>[ 2.3344 0.9447 2.3528 1.7709<\/pre>\n<pre>3.8734 0.1722 1.8460 4.2087<\/pre>\n<pre>2.4182 1.8999 0.3384 1.1055<\/pre>\n<pre>3.2058 1.9221 2.1859 0.7215]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>In addition to the return values, you can get back the data and parameter values:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.getData()<\/pre>\n<pre>[ 0.4669 0.1889 0.4706 0.3542<\/pre>\n<pre>0.7747 0.0344 0.3692 0.8417<\/pre>\n<pre>0.4836 0.3800 0.0677 0.2211<\/pre>\n<pre>0.6412 0.3844 0.4372 0.1443]<\/pre>\n<pre>>>> session.getParams()<\/pre>\n<pre>[1, 3, 5]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>If you are planning to run the same work function over and over again in different sessions, you can simplify things further by creating a \u201csweeper\u201d function from it. The sweeper function takes your work function and returns a thing that will automatically sweep your function over the set of parameters, block until they\u2019re done and return the results:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> adder = paramsweep.sweeper(paramsweeptest.add)<\/pre>\n<pre>>>> adder(10, [1,2,3,4,5])<\/pre>\n<pre>[11, 12, 13, 14, 15]<\/pre>\n<pre>>>> adder(100, [1,2,3,4,5])<\/pre>\n<pre>[101, 102, 103, 104, 105]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Note that this is equivalent to:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session = paramsweep.run(paramsweeptest.add, 10, [1,2,3,4,5])<\/pre>\n<pre>>>> session.waitResults()<\/pre>\n<pre>[11, 12, 13, 14, 15]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>More complex examples<\/h2>\n<p>Oftentimes, you will want to run multiple parameter sweep sessions over the same large dataset. Since it can take a long time to upload the data to the cloud, you may want to explicitly upload the data once. Then you can tell the Azure instances to use the persisted cloud data, rather than upload it each time. This example uses the \u201ctopnsvd\u201d test function, which returns the <i>n<\/i> largest singular values from a matrix.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> import azureutils<\/pre>\n<pre>>>> arr = rand(1000, 1000)<\/pre>\n<pre>>>> blob = azureutils.pickleToBlob(arr)<\/pre>\n<pre>>>> session = paramsweep.run(paramsweeptest.topnsvd, blob, [1,2,3,4])<\/pre>\n<pre>>>> session.waitResults()<\/pre>\n<pre>[[ 500.0264], [ 500.0264 18.0877], [ 500.0264 18.0877 18.0564],<\/pre>\n<pre>[ 500.0264 18.0877 18.0564 17.9239]]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>All of the examples we\u2019ve shown so far run work functions that take objects as the parameters and produce output in the form of return values. However, many times it is easier to express your work function in terms of input and\/or output files. For these kinds of applications, we allow you to optionally specify an inDir and outDir argument to your work function. In this case, your work function should be specified as follows:<\/p>\n<blockquote>\n<blockquote>\n<pre>def workfn(data, param, inDir, outDir): ...<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>If you specify an inDir variable, the contents of that local directory will be uploaded to the cloud and then downloaded to your instance. For instance, indirlist is a function that returns a list of the files in inDir (the data and param values are ignored, and set to 0 here):<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session = paramsweep.run(paramsweeptest.indirlist, 0, [0], inDir=\"C:\/tmp\/myindir\")<\/pre>\n<pre>>>> session.waitResults()<\/pre>\n<pre>[['a.txt', 'b.txt']]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>If you want to produce output as files, outdirtest is an example:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session = paramsweep.run(paramsweeptest.outdirtest, \"foo\", [1,2,3], outDir=\"C:\/tmp\/outdir\")<\/pre>\n<pre>>>> session.waitDone()<\/pre>\n<pre>>>> session.downloadOutDir()<\/pre>\n<pre>>>> ls(\"C:\/tmp\/outdir\")<\/pre>\n<pre>1.pickle 27 10\/19\/2010 5:38:33 PM<\/pre>\n<pre>2.pickle 27 10\/19\/2010 5:38:33 PM<\/pre>\n<pre>3.pickle 27 10\/19\/2010 5:38:33 PM<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>You can get a list of session objects that still store data on the cloud using the listSessions function, which will return a handle to all of the sessions started by you:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> for s in paramsweep.listSessions(): print s<\/pre>\n<pre>ParamSweepSession('8aee1717-3d23-44b0-8033-21883a332f39')<\/pre>\n<pre>ParamSweepSession('c6be1ea5-189e-441a-9f31-09a87ffc7102')<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>This is particularly useful if you forgot to clean them up:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> for s in paramsweep.listSessions(): s.cleanup()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Note that your work function <em>must<\/em> conform to one of the following forms, or it will not work:<\/p>\n<blockquote>\n<blockquote>\n<pre>def workFn(data, param): ...<\/pre>\n<pre>def workFn(data, param, inDir): ...<\/pre>\n<pre>def workFn(data, param, inDir, outDir): ...<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Finally, you can retrieve any output text printed by your work function with the getOutputText() method on your session object:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.getOutputText()<\/pre>\n<pre>[...]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>Caching state and data<\/h2>\n<p>In order to make sure that one job doesn\u2019t pollute the Sho environment and cause problems for subsequent jobs running on that Azure instance, the entire Sho environment is restarted between jobs by default. Also by default, the data value is re-downloaded for each job, so that and destructive changes one job makes doesn\u2019t affect another job. Since both of these safeguards are frequently unnecessary and can take a significant amount of time, there are ways to disable them to get your jobs to run faster.<\/p>\n<p>These options are controlled by setting special global variables in the main module of your work function. Setting the \u201ccan_reuse_state\u201d variable to True tells the system that you can re-use the Sho instance between jobs from the same session. The \u201ccan_reuse_data\u201d variable controls whether or not you can reuse the same data object between jobs. Finally, if you want to be able to share state and\/or data values between different sessions, you can set the \u201cstate_token\u201d variable to some unique value. Different sessions with the same \u201cstate_token\u201d value will be able to reuse the Sho environment and data value as if they were jobs from the same session. Here is an example of the settings in the python file that contains the demo job functions:<\/p>\n<blockquote>\n<blockquote>\n<pre>paramsweeptest.py:<\/pre>\n<pre>can_reuse_state = True<\/pre>\n<pre>can_reuse_data = True<\/pre>\n<pre>state_token = \"parametersweeptest\"<\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>Storage Utilities<\/h2>\n<p>The paramsweep and auzreutils modules contain a number of useful utilities for dealing with Azure storage.<\/p>\n<p>To store a data object in the cloud, call the azureutils.pickleToBlob() method:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> arr = <i><some data><\/some><\/i><\/pre>\n<pre>>>> blob = azureutils.pickleToBlob(arr)<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>This stores your data into a new blob with a unique name. If you want to pickle to data into a blob with a particluar name or at a particular URL, you can pass that in:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> azureutils.pickleToBlob(arr, blobname) # blobname is a filename or fully-qualified URL<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>In order to read the data back from the cloud, you can call the unpickle method:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> blob.unpickle()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>If you don\u2019t have a blob object, you can create one given the URI of an object in blob storage. Or, you can directly unpickle from a URI:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> blob = azureutils.BlobLocation(\"http:\/\/<blob-storage-url>\")<\/blob-storage-url><\/pre>\n<pre>>>> mydata = blob.unpickle()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>or<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> mydata = azureutils.unpickleBlob(http:\/\/<blob-storage-url>)<\/blob-storage-url><\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Finally, to delete a pickled blob, call the delete method or call the deleteBlob function:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> blob.delete()<\/pre>\n<pre>>>> azureutils.deleteBlob(\"http:\/\/...\")<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>You can retrieve a list of blobs you\u2019ve pickled via the listPickleBlobs function, which returns a list of BlobLocation objects:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> azureutils.listPickleBlobs()<\/pre>\n<pre>[<azureutils.BlobLocation instance at 0x000000000000002B>, <azureutils.BlobLocation instance at 0x000000000000002C>]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Here are a few other useful functions:<\/p>\n<blockquote>\n<blockquote>\n<pre>uploadFile() \u2013 uploads a file into blob storage<\/pre>\n<pre>downloadFile() \u2013 downloads a blob\u2019s contents to a file<\/pre>\n<pre>uploadDirectory() \u2013 uploads a directory into blob storage<\/pre>\n<pre>downloadBlobDirectory() \u2013 downloads a directory of files from blob storage<\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>Viewing diagnostic information<\/h2>\n<p>If you want to output diagnostic messages from your work function, first import the azureserverutil module, then call the tracemessage function. The message will be saved in Azure diagnostic message storage. Then, you can call the showLog method on your session object in order to view the messages.<\/p>\n<blockquote>\n<blockquote>\n<pre>def tracetest(data, param):<\/pre>\n<blockquote>\n<pre>azureserverutil.tracemessage(str(data) + \": \" + str(param))<\/pre>\n<\/blockquote>\n<\/blockquote>\n<\/blockquote>\n<p>Then, back at the console, call showLog to display the log messages:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session = paramsweep.run(paramsweeptest.tracetest, \"message with param value\", [1,2,3])<\/pre>\n<pre>>>> session.waitDone()<\/pre>\n<pre>(wait a few minutes)<\/pre>\n<pre>>>> session.showLog()<\/pre>\n<pre><span id=\"d19b6f73-fe67-4b47-9082-01fd91e68d37\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imaged19b6f73-fe67-4b47-9082-01fd91e68d37\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurelog.png\" alt=\"\" \/><\/span><\/pre>\n<\/blockquote>\n<\/blockquote>\n<p style=\"text-align: left;\">\u00a0The results will be displayed in a Sho datagrid, with the results from each job grouped together. Note that there is a delay of a couple of minutes between when your work function calls tracemessage and when the corresponding log message is viewable. Passing in True as the optional second parameter to showLog will display internal trace messages that the service records, which can be useful when troubleshooting:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.showLog(True)<\/pre>\n<pre><span id=\"5a7ed872-c891-48e3-99c4-ddd50dbaedbe\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image5a7ed872-c891-48e3-99c4-ddd50dbaedbe\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurelog2.png\" alt=\"\" \/><span id=\"ImageCaption5a7ed872-c891-48e3-99c4-ddd50dbaedbe\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>Alternately, if you want to show all recent diagnostic messages, regardless of which session they belong to, you can all the azureutils.showLog function. This can sometimes help you determine if the service is having problems. Be aware that the format of these diagnostic messages can make them difficult to read:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> azureutils.showLog(10) # show messages from the last 10 minutes<\/pre>\n<pre>>>> azureutils.showLog(TimeSpan.FromHours(12))<\/pre>\n<\/blockquote>\n<\/blockquote>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-9\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-9\">\n\t\tGetting an Azure Account\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-9\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p>Before you can use the Azure package, you\u2019ll need to get an Azure account, so you can upload data and run code in the cloud. To get started, go to the Azure Offers page (<a href=\"http:\/\/www.microsoft.com\/windowsazure\/offers\/\">http:\/\/www.microsoft.com\/windowsazure\/offers\/<\/a>) and select the package that is most appropriate\u00a0for you\u00a0. Note that you do <i>not<\/i> need SQL Azure access to use the Sho Azure package. Please see the following links for information on pricing for academic and NSF-funded work:<\/p>\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.nsf.gov\/cise\/news\/2010_microsoft.jsp\">http:\/\/www.nsf.gov\/cise\/news\/2010_microsoft.jsp<\/a><\/p>\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.azurepilot.com\/\">http:\/\/www.azurepilot.com\/<\/a><\/p>\n<h1>Server-side Installation and setup<\/h1>\n<h2>Creating the Azure storage service<\/h2>\n<p>First, we\u2019ll create the storage service that will be used to store all of our data and results in the cloud. Go to the Windows Azure service portal (<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/windows.azure.com\/\">http:\/\/windows.azure.com<\/a>) and log in.<\/p>\n<p><span id=\"285ed61e-d740-4d69-8661-5c41e80f2026\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image285ed61e-d740-4d69-8661-5c41e80f2026\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azureribbon1.jpg\" alt=\"\" \/><span id=\"ImageCaption285ed61e-d740-4d69-8661-5c41e80f2026\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Then, click on \u201cNew Storage Account\u201d in the ribbon:<\/p>\n<p><span id=\"ec8fcd57-5b01-4a9a-8007-190ef58360f9\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imageec8fcd57-5b01-4a9a-8007-190ef58360f9\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurenewaccount.png\" alt=\"\" \/><span id=\"ImageCaptionec8fcd57-5b01-4a9a-8007-190ef58360f9\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>When prompted, enter a URL to use to access your storage service (e.g., \u201cmyshodata\u201d) and click the \u2018Create\u2019 button. Make a note of the prefix for this URL (the \u201cmyshodata\u201d part) \u2013 users of the package will need to know this account name in order to use the service. The other piece of information is the primary access key associated with this service, which users will need in order to authenticate themselves. To find this, select the new storage service you just created and then click on the \u201cView\u201d button for the primary access key:<\/p>\n<p><span id=\"d721b249-0096-4595-ae70-ad863f175db2\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imaged721b249-0096-4595-ae70-ad863f175db2\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azureviewkey.jpg\" alt=\"\" \/><span id=\"ImageCaptiond721b249-0096-4595-ae70-ad863f175db2\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>This will display the access keys:<\/p>\n<p><span id=\"358b3292-dd4e-47f2-8738-7edae61ed8f5\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image358b3292-dd4e-47f2-8738-7edae61ed8f5\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azureaccesskeys.jpg\" alt=\"\" \/><span id=\"ImageCaption358b3292-dd4e-47f2-8738-7edae61ed8f5\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Users of the service will have to set two environment variables, SHOAZUREACCOUNT, which will get the name of your storage service (e.g., myshodata), and SHOAZUREKEY, which will get the primary access key. You will also need to set these variables on any computer you wish to administer the Sho Azure package from.<\/p>\n<h2>Copying Sho to the Cloud<\/h2>\n<p>Now that you have created the storage service and set your SHOAZUREACCOUNT and SHOAZUREKEY environment variables, you\u2019ll need to copy a Sho installation to the cloud. The simplest way to do that is to run the Sho console and run the uploadSho() function from the Azure package:<\/p>\n<blockquote>\n<blockquote>\n<pre>Sho 2.0.4 on IronPython 2.6.1 (), .NET 4.0.30319.1 and MKL 10.3.<\/pre>\n<pre>Includes parts of the Intel Math Kernel Library for Windows.<\/pre>\n<pre>>>> azureutils.uploadSho()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>This may take quite a long time. When it\u2019s done, you\u2019re ready to create the hosted service and upload the parameter sweeper.<\/p>\n<h2>Creating the Hosted service<\/h2>\n<p>Next, we\u2019ll create the Azure compute service that the actual code will run on. In preparation for this, we must first load the ServiceConfiguration.cscfg file from the Publish directory in the Azure package into a text editor. Then replace <accountname> and <accountkey> with your account name and key in the highlighted lines:<\/accountkey><\/accountname><\/p>\n<blockquote>\n<pre><?xml version=\"1.0\"?><\/pre>\n<pre><serviceconfiguration serviceName=\"ParameterSweep\" xmlns=\"http:\/\/schemas.microsoft.com\/ServiceHosting\/2008\/10\/ServiceConfiguration\"><\/serviceconfiguration><\/pre>\n<blockquote>\n<pre><role name=\"ParamSweepWorkerRole\"><\/role><\/pre>\n<blockquote>\n<pre><instances count=\"1\"><\/instances><\/pre>\n<pre><configurationsettings><\/configurationsettings><\/pre>\n<blockquote>\n<pre><setting name=\"DataConnectionString\" value=\"DefaultEndpointsProtocol=https;AccountName=<<strong>ACCOUNTNAME>;AccountKey=<<strong>ACCOUNTKEY<\/strong>>\" \/><\/setting><\/pre>\n<pre><setting name=\"DiagnosticsConnectionString\" value=\"DefaultEndpointsProtocol=https;AccountName=<<strong>ACCOUNTNAME>;AccountKey=<<strong>ACCOUNTKEY<\/strong>>\" \/><\/setting><\/pre>\n<pre><setting name=\"ResultsContainerName\" value=\"paramsweepresults\"><\/setting><\/pre>\n<pre><setting name=\"JobQueueName\" value=\"paramsweepqueue\"><\/setting><\/pre>\n<\/blockquote>\n<pre><\/pre>\n<pre><certificates><\/certificates><\/pre>\n<pre><\/pre>\n<\/blockquote>\n<pre><\/pre>\n<\/blockquote>\n<pre><\/pre>\n<\/blockquote>\n<p>Now that you have edited the configuration file to be appropriate for your deployment, let\u2019s create a new hosted compute service and upload the deployment. First, click on the \u201cNew Hosted Service\u201d button from the Windows Azure portal:<\/p>\n<p><span id=\"416f874c-3566-484a-a174-9dba1f4c23ae\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image416f874c-3566-484a-a174-9dba1f4c23ae\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurecreatehosted.jpg\" alt=\"\" \/><span id=\"ImageCaption416f874c-3566-484a-a174-9dba1f4c23ae\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>This will bring up the \u201cCreate a new Hosted Service\u201d dialog box. In this screen, you\u2019ll have to make up a name and URL for your hosted service.<\/p>\n<p><span id=\"85ba5adb-d293-4392-af99-3fa36f11bc65\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image85ba5adb-d293-4392-af99-3fa36f11bc65\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurecreatehosted2.jpg\" alt=\"\" \/><span id=\"ImageCaption85ba5adb-d293-4392-af99-3fa36f11bc65\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>In addition, you can upload the deployment package (and its corresponding configuration file that you edited above) now. Choose \u201cBrowse Locally\u2026\u201d and point the file dialog at the .cspkg and .cscfg files in the Publish directory inside you Azure package directory (i.e., SHODIR\/packages\/\/azurePublish, where SHODIR is you Sho installation directory). Note that if you keep the &#8220;Start after successful deployment&#8221; box checked, the deployment start running in the cloud and you will start accruing usage charges!<\/p>\n<h2>Client-side installation and setup<\/h2>\n<p>To install the package, first copy the package directory to your Sho packages directory.<\/p>\n<p><span id=\"54ab8afb-5784-41b5-a656-cb5955123da0\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image54ab8afb-5784-41b5-a656-cb5955123da0\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azurepackagedir.png\" alt=\"\" \/><span id=\"ImageCaption54ab8afb-5784-41b5-a656-cb5955123da0\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Then, set the following environment variables to the values you noted above:<\/p>\n<table class=\" borderColumns borderRows tableBorder\" style=\"height: 78px;\" width=\"643\" cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREACCOUNT<\/td>\n<td valign=\"top\" width=\"319\">The name of the storage account you created (myshodata in the example in this guide)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREKEY<\/td>\n<td valign=\"top\" width=\"319\">The private key for the storage service<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The following are optional, and are needed if you wish to do basic administration (changing the number of active instances) from the Sho console instead of the Azure web portal:<\/p>\n<table class=\" borderColumns borderRows tableBorder\" style=\"height: 251px;\" width=\"642\" cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"299\">SHOAZUREHOST<\/td>\n<td valign=\"top\" width=\"292\">The name of the hosting account<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"299\">SHOAZUREDEPLOYSLOT<\/td>\n<td valign=\"top\" width=\"292\">The deployment slot you used for the Sho worker roles (either \u201cstaging\u201d or \u201cdeployment\u201d)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"299\">SHOAZURECERTNAME<\/td>\n<td valign=\"top\" width=\"292\">The name of the certificate you created (e.g., \u201cWindows Azure Authentication Certificate\u201d)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"299\">SHOAZURESUBID<\/td>\n<td valign=\"top\" width=\"292\">The subscription ID for your Azure subscription. This can be found on the Windows Azure portal screen.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Usage<\/h2>\n<p>Here is a brief example of how to use the package so you can test your installation. Please read the user guide for more complete information.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> paramsweep.addDemoDir()<\/pre>\n<pre>>>> import paramsweeptest<\/pre>\n<pre>>>> session = paramsweep.run(paramsweeptest.add, 100, [1,2,3,4,5])<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>The runsweep function returns a parameter sweep session object:<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> print session<\/pre>\n<pre>ParamSweepSession('2c574185-9cd3-4ba1-a6f3-9668424374a8')<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p>The session runs asynchronously on the cloud. We can check to see if it\u2019s finished by calling isDone(). If it is, we can call getResults to get the results from each of the jobs. Alternately we can call waitResults(), which blocks until it finishes, and then return the result values.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> print session.isDone()<\/pre>\n<pre>True<\/pre>\n<pre>>>> print session.getResults()<\/pre>\n<pre>[101, 102, 103, 104, 105]<\/pre>\n<\/blockquote>\n<\/blockquote>\n<p dir=\"ltr\">It\u2019s a good idea to delete the server-side data associated with this session when you\u2019re done.<\/p>\n<blockquote>\n<blockquote>\n<pre>>>> session.cleanup()<\/pre>\n<\/blockquote>\n<\/blockquote>\n<h2>Service utilities<\/h2>\n<p>The Sho Azure package includes a function that allows you to change the number of Azure worker role instances that are running in the cloud, waiting for jobs. If you want to use this function instead of using the web portal, you\u2019ll have to add a certificate to your service.<\/p>\n<p>First, you\u2019ll need to generate a new self-signed certificate in a .cer file. The following blog post describes one way to do this:<\/p>\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blogs.msdn.com\/b\/ericnel\/archive\/2009\/09\/22\/using-iis-to-generate-a-x509-certificate-for-use-with-the-windows-azure-service-management-api-step-by-step.aspx\">http:\/\/blogs.msdn.com\/b\/ericnel\/archive\/2009\/09\/22\/using-iis-to-generate-a-x509-certificate-for-use-with-the-windows-azure-service-management-api-step-by-step.aspx<\/a><\/p>\n<p>If you are more comfortable with the command line, another method is described here:<\/p>\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/consultingblogs.emc.com\/gracemollison\/archive\/2010\/02\/19\/creating-and-using-self-signed-certificates-for-use-with-azure-service-management-api.aspx\">http:\/\/consultingblogs.emc.com\/gracemollison\/archive\/2010\/02\/19\/creating-and-using-self-signed-certificates-for-use-with-azure-service-management-api.aspx<\/a><\/p>\n<p>To add the certificate to your Azure subscription, select \u201cManagement Certificates\u201d in the sidebar and then click the \u201cAdd Certificate button\u201d<\/p>\n<p><span id=\"f0719549-c61c-48c8-bc66-b577a745b5ce\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagef0719549-c61c-48c8-bc66-b577a745b5ce\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azureaddcert.jpg\" alt=\"\" \/><span id=\"ImageCaptionf0719549-c61c-48c8-bc66-b577a745b5ce\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Then, click the Browse button in the dialog box and find your .cer file:<\/p>\n<p><span id=\"8923dd2d-dedc-4f72-a6ed-f78261ca1492\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image8923dd2d-dedc-4f72-a6ed-f78261ca1492\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2016\/02\/sho-azureaddcert2.jpg\" alt=\"\" \/><span id=\"ImageCaption8923dd2d-dedc-4f72-a6ed-f78261ca1492\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<h2>Using the API from Sho<\/h2>\n<p>In order to change the number of deployment instances, you can call the changeNumWorkerRoles() function:<\/p>\n<p>>>> req = azureservice.changeNumWorkerRoles(<i>num<\/i>)<\/p>\n<p>This function returns a request ID, which you can use to check on the status of the operation. It typically takes several minutes or so to change the number of instances.<\/p>\n<p>>>> azureservice.getOperationStatus(req)<\/p>\n<h2>Setup Summary<\/h2>\n<p>To use the parameter sweep facilities for running jobs in the cloud, your users will need to set the following environment variables on the client computers:<\/p>\n<table class=\" borderColumns borderRows tableBorder\" cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREACCOUNT<\/td>\n<td valign=\"top\" width=\"319\">The name of the storage account you created (myshodata in the example in this guide)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREKEY<\/td>\n<td valign=\"top\" width=\"319\">The private key for the storage service<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>For using the Sho service utilities described in this document, you will also need to set the following environment variables:<\/p>\n<table class=\" borderColumns borderRows tableBorder\" cellspacing=\"0\" cellpadding=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREHOST<\/td>\n<td valign=\"top\" width=\"319\">The name of the hosting account<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZUREDEPLOYSLOT<\/td>\n<td valign=\"top\" width=\"319\">The deployment slot you used for the Sho worker roles (either \u201cstaging\u201d or \u201cdeployment\u201d)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZURECERTNAME<\/td>\n<td valign=\"top\" width=\"319\">The name of the certificate you created (e.g., \u201cWindows Azure Authentication Certificate\u201d)<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"319\">SHOAZURESUBID<\/td>\n<td valign=\"top\" width=\"319\">The subscription ID for your Azure subscription. This can be found on the Windows Azure portal screen.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Thanks! And please send feedback to <a href=\"mailto:shofeedback@microsoft.com\">shofeedback@microsoft.com<\/a><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-10\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-10\">\n\t\tLibraries\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-10\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<h2>Sho External API Guide<\/h2>\n<\/p>\n<p>If you&#8217;d like to use the Sho libraries directly from .NET (C#, Managed C++, F#, etc.), you&#8217;ll need to add references to specific libraries depending on what functionality you wish to use; in some cases you&#8217;ll also need to do a bit of setup so that the references will resolve properly. This guide explains how to do this.<\/p>\n<h2>Installing the Libraries<\/h2>\n<p>To get the libraries, we recommend installing Sho so that you can use the Sho Console as well; however, to build against the libraries, all you need are the contents of the &#8220;bin&#8221; directory from the Sho install; you can get these by copying them from your Sho install (evaluate &#8220;shobindir&#8221; in the Sho console to find the location). Thus, if you&#8217;re putting code into source control, distributing the code, etc., you&#8217;ll only need to distribute the bin directory. Note that if you plan to use EmbeddedSho, you&#8217;ll likely want to copy the entire Sho directory, as you&#8217;ll need to point to it when creating an EmbeddedSho engine.<\/p>\n<h2>Code Setup<\/h2>\n<p>Because the ShoArray classes contain both managed and unmanaged code to achieve maximal computation speed, the array libraries depend on some satellite DLLs. In order for the managed libraries to resolve these references, you&#8217;ll have to do <b>one<\/b> of the following:<\/p>\n<ul>\n<li>[preferred method] Set the environment variable <b>SHODIR<\/b> to the <b>parent<\/b> of Sho&#8217;s bin directory. Alternatively, you can set the variable SHOBINDIR to the bin directory directly. For instance, if you copied the bin directory from the Sho install to C<a class=\"invalidLink\" href=\"http:\/\/research.microsoft.com\/libraries\/\/shobin\">:<\/a>librariesShobin, you should point SHODIR to C<a class=\"invalidLink\" href=\"http:\/\/research.microsoft.com\/libraries\/\/sho\">:<\/a>librariesSho. Note that you can set the variable programatically (ideally as soon as the app starts) viaSystem.Environment.SetEnvironmentVariable(&#8220;SHODIR&#8221;,&#8230;)<\/li>\n<li>Another option is to put your .exe into the Shobin directory, or equivalently copy the Shobin directory to the directory that contains your .exe.<\/li>\n<\/ul>\n<h2>What References Do I Need to Add?<\/h2>\n<p>The table below lists the DLL&#8217;s and namespaces required for various aspects of Sho functionality.<\/p>\n<table class=\" tWiz tableBorder borderRows borderColumns\" style=\"height: 605px;\" width=\"740\" cellspacing=\"3\" cellpadding=\"2\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"213\"><strong>Functionality<\/strong><\/td>\n<td valign=\"top\" width=\"213\"><b>Namespaces<\/b><\/td>\n<td valign=\"top\" width=\"213\"><b>Referenced DLLs<\/b><\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Basic matrix math<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.Array<\/td>\n<td valign=\"top\" width=\"213\">ShoArray.dllMatrixInterf.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Basic matrix + complex numbers<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.ArrayMicrosoft.Scripting.Math<\/td>\n<td valign=\"top\" width=\"213\">ShoArray.dllMatrixInterf.dllMicrosoft.Dynamic.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Extensions for Python matrix code<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.ArrayShoNS.PythonExtensions<\/td>\n<td valign=\"top\" width=\"213\">ShoArray.dllMatrixInterf.dllPythonExt.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Random number generation<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.MathFunc<\/td>\n<td valign=\"top\" width=\"213\">Rand.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Auto-mapping functions<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.ArrayShoNS.MathFunc<\/td>\n<td valign=\"top\" width=\"213\">ShoArray.dllMatrixInterf.dllMathFunc.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Reading\/writing delimited files<\/td>\n<td valign=\"top\" width=\"213\">\n<p>ShoNS.Array<\/p>\n<p>ShoNS.IO<\/p>\n<\/td>\n<td valign=\"top\" width=\"213\">ShoArray.dllMatrixInterf.dlldlmIO.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Plotting\/charting and other visualization<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.Visualization<\/td>\n<td valign=\"top\" width=\"213\">ShoViz.dllSystem.Windows.Forms<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Serialization of objects (including Python objects)<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.PicklingSystem.IO<\/td>\n<td valign=\"top\" width=\"213\">ShoPickleHelper.dll<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"213\">Embedding a Sho interpreter in your C# code<\/td>\n<td valign=\"top\" width=\"213\">ShoNS.Hosting<\/td>\n<td valign=\"top\" width=\"213\">EmbeddedSho.dllIronPython.dllIronPython.Modules.dll<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Sample Projects<\/h2>\n<p>The following projects show examples of how to use the Sho libraries from C#. Each zip file contains a Visual Studio solution file; you should unzip this to {SHODIR}playpen so that the relative references will work properly.<\/p>\n<ul>\n<li><b>CSharpExample<\/b> &#8211; shows how to create an array, do some linear algebra operations, and create a plot window from C#.[<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2010\/11\/CSharpExample40.zip\">Project Zip file<\/a>]<\/li>\n<li><b>EmbeddedShoTest<\/b> &#8211; shows how to create an EmbeddedSho engine, pass variable values back and from C# to Sho, and execute Sho statements (including a plot).][Project Zip file]<\/li>\n<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t\t\n<h1>Contact<\/h1>\n<ul>\n<li><a href=\"mailto:shofeedback@microsoft.com\">shofeedback@microsoft.com<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Sho is an interactive environment for data analysis and scientific computing that lets you seamlessly connect scripts (in IronPython) with compiled code (in .NET) to enable fast and flexible prototyping. The environment includes powerful and efficient libraries for linear algebra as well as data visualization that can be used from any .NET language, as well [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556,13560],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-170593","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-programming-languages-software-engineering","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2010-11-08","related-publications":[],"related-downloads":[234975,234977,234979],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[305975],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Sumit Basu","user_id":33754,"people_section":"Group 1","alias":"sumitb"}],"msr_research_lab":[],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170593"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":4,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170593\/revisions"}],"predecessor-version":[{"id":604380,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170593\/revisions\/604380"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=170593"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=170593"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=170593"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=170593"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=170593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}