{"id":8732,"date":"2023-07-13T07:54:07","date_gmt":"2023-07-13T14:54:07","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=8732"},"modified":"2023-07-13T08:20:22","modified_gmt":"2023-07-13T15:20:22","slug":"microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/","title":{"rendered":"Providing modern data transfer and storage service at Microsoft with Microsoft Azure"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"112\" class=\"size-medium wp-image-7498 alignright\" style=\"margin-top: 0px;\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/ms-digital-technical-stories-300x112.png\" alt=\"Microsoft Digital technical stories\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/ms-digital-technical-stories-300x112.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/ms-digital-technical-stories.png 500w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/>Companies all over the world have launched their cloud adoption journey. While some are just starting, others are further along the path and are now researching the best options for moving their largest, most complex workflows to the cloud. It can take time for companies to address legacy tools and systems that have on-premises infrastructure dependencies.<\/p>\n<p>Our Microsoft Digital Employee Experience (MDEE) team has been running our company as mostly cloud-only since 2018, and continues to design cloud-only solutions to help fulfill our Internet First and <a href=\"https:\/\/www.microsoft.com\/en-us\/security\/business\/zero-trust#:~:text=Microsoft%20deployed%20Zero%20Trust%20to%20secure%20corporate%20and,secure%2C%20least-privilege%20access%20to%20corporate%20resources%20and%20services.\" target=\"_blank\" rel=\"noopener\">Microsoft Zero Trust<\/a> goals.<\/p>\n<p>In MDEE, we designed a Modern Data Transfer Service (MDTS), an enterprise-scale solution that allows the transfer of large files to and from partners outside the firewall and removes the need for an extranet.<\/p>\n<p>MDTS makes cloud adoption easier for teams inside Microsoft and encourages the use of Microsoft Azure for all of their data transfer and storage scenarios. As a result, engineering teams can focus on building software and shipping products instead of dealing with the management overhead of Microsoft Azure subscriptions and becoming subject matter experts on infrastructure.<\/p>\n<p><em>[<a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/simplifying-microsofts-royalty-ecosystem-with-connected-data-service\/\">Unpack simplifying Microsoft\u2019s royalty ecosystem with connected data service<\/a>. | <a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/how-microsoft-employees-are-leveraging-the-cloud-for-file-storage-with-onedrive-folder-backup\/\">Check out how Microsoft employees are leveraging the cloud for file storage with OneDrive Folder Backup<\/a>. | <a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/simplifying-compliance-evidence-management-with-microsoft-azure-confidential-ledger\/\">Read more on simplifying compliance evidence management with Microsoft Azure confidential ledger<\/a>.]<\/em><\/p>\n<h2>Leveraging our knowledge and experience<\/h2>\n<p>As part of Microsoft\u2019s cloud adoption journey, we have been continuously looking for opportunities to help other organizations move data and remaining legacy workflows to the cloud. With more than 220,000 employees and over 150 partners that data is shared with, not every team had a clear path for converting their transfer and storage patterns into successful cloud scenarios.<\/p>\n<p>We have a high level of Microsoft Azure service knowledge and expertise when it comes to storage and data transfer. We also have a long history with legacy on-premises storage designs and hybrid third-party cloud designs. Over the past decade, we engineered several data transfer and storage services to facilitate the needs of Microsoft engineering teams. Those services traditionally leveraged either on-premises designs or hybrid designs with some cloud storage. In 2019, we began to seriously look at replacing our hybrid model, which included a mix of on-premises resources, third party software, and Microsoft Azure services, with one modern service that would completely satisfy our customer scenarios using only Azure\u2014thanks to new capabilities in Azure making it possible and it being the right time.<\/p>\n<p>MDTS uses out of the box Microsoft Azure storage configurations and capabilities to help us address legacy on-premises storage patterns and support Microsoft core commitments to fully adopt Azure in a way that satisfies security requirements. Managed by a dedicated team of service engineers, program managers, and software developers, MDTS offers performance, security, and is available to any engineering team at Microsoft that needs to move their data storage and transfer to the cloud.<\/p>\n<h2>Designing a Modern Data Transfer and Storage Service<\/h2>\n<p>The design goal for MDTS was to create a single storage service offering entirely in Microsoft Azure, that would be flexible enough to meet the needs of most engineering teams at Microsoft. The service needed to be sustainable as a long-term solution, continue to support ongoing Internet First and Zero Trust Network security designs, and have the capability to adapt to evolving technology and security requirements.<\/p>\n<h3>Identifying use cases<\/h3>\n<p>First, we needed to identify the top use cases we wanted to solve and evaluate which combination of Microsoft Azure services would help us meet our requirements. The primary use cases we identified for our design included:<\/p>\n<ul class=\"c-list\">\n<li><strong>Sharing and\/or distribution of complex payloads:<\/strong> We not only had to provide storage for corporate sharing needs, but also share those same materials externally. The variety of file sizes and different payload characteristics can be challenging because they don\u2019t always fit a standard profile for files (e.g., Office docs, etc.).<\/li>\n<li><strong>Cloud storage adoption (shifting from on-premises to cloud):<\/strong> We wanted to ensure that engineering teams across Microsoft that needed a path to the cloud would have a roadmap. This need could arise because of expiring on-premises infrastructure, corporate direction, or other modernization initiatives like ours.<\/li>\n<li><strong>Consolidation of multiple storage solutions into one service, to reduce security risks and administrative overhead:<\/strong> Having to place data and content in multiple storage datastores for the purposes of specific sharing or performance needs is both cumbersome and can introduce additional risk. Because there wasn\u2019t yet a single service that could meet all their sharing needs and performance requirements, employees and teams at Microsoft were using a variety of locations and services to store and share data.<\/li>\n<\/ul>\n<h3>Security, performance, and user experience design requirements<\/h3>\n<p>After identifying the use cases for MDTS, we focused on our primary design requirements. They fell into three high-level categories: security, performance, and user experience.<\/p>\n<h4>Security<\/h4>\n<p>The data transfer and storage design needed to follow our Internet First and Zero Trust network design principles. Accomplishing parity with Zero Trust meant leveraging best practices for encryption, standard ports, and authentication. At Microsoft, we already have standard design patterns that define how these pieces should be delivered.<\/p>\n<ul class=\"c-list\">\n<li><strong>Encryption:<\/strong> Data is encrypted both in transit and at rest.<\/li>\n<li><strong>Authentication:<\/strong> Microsoft Azure Active Directory supports both corporate synced domain accounts, external business-to-business accounts, as well as corporate and external security groups. Leveraging Azure Active Directory allows teams to remove dependencies on corporate domain controllers for authentication.<\/li>\n<li><strong>Authorization:<\/strong> Microsoft Azure Data Lake Gen2 storage provides fine grained access to containers and subfolders. This is possible because of many new capabilities, most notably the support for OAuth, hierarchical name space, and POSIX permissions. These capabilities are necessities of a Zero Trust network security design.<\/li>\n<li><strong>No non-standard ports:<\/strong> Opening non-standard ports can present a security risk. Using only HTTPS and TCP 443 as the mechanisms for transport and communication prevents opening non-standard ports. This includes having software capable of transport that maximizes the ingress\/egress capabilities of the storage platform. Microsoft Azure Storage Explorer, AzCopy, and Microsoft Azure Data Factory meet the no non-standard ports requirement.<\/li>\n<\/ul>\n<h4>Performance<\/h4>\n<p>Payloads can range from being comprised of one very large file, millions of small files, and every combination in between. Scenarios across the payload spectrum have their own computing and storage performance considerations and challenges. Microsoft Azure has optimized software solutions for achieving the best possible <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/scalability-targets-standard-account#:~:text=The%20ingress%20limit%20refers%20to%20all%20data%20that,data%20that%20is%20received%20from%20a%20storage%20account.\" target=\"_blank\" rel=\"noopener\">storage ingress and egress<\/a>. MDTS helps ensure that customers know what optimized solutions are available to them, provides configuration best practices, and shares the learnings with Azure Engineering to enable robust enterprise scale scenarios.<\/p>\n<ul class=\"c-list\">\n<li><strong>Data transfer speeds:<\/strong> Having software capable of maximizing the ingress\/egress capabilities of the storage platform is preferable for engineering-type workloads. It\u2019s common for these workloads to have complex payloads, payloads with several large files (10-500 GB) or millions of small files.<\/li>\n<li><strong>Ingress and egress:<\/strong> Support for ingress upwards of 10 Gbps and egress of 50 Gbps. Furthermore, client and server software that can consume the maximum amount of bandwidth possible up to the maximum amount in ingress\/egress possible on client and storage.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<table style=\"width: 100%;\" border=\"1\" cellpadding=\"5\">\n<tbody>\n<tr>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>Data size\/ bandwidth<\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>50 Mbps <\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>100 Mbps<\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>500 Mbps<\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>1 Gbps<\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>5 Gbps<\/strong><\/th>\n<th style=\"border: 1px solid black; padding: 5px; text-align: center;\"><strong>10 Gbps<\/strong><\/th>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid black; padding: 5px;\">1 GB<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">2.7 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">1.4 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.3 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.1 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.03 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.010 minutes<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid black; padding: 5px;\">10 GB<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">27.3 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">13.7 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">2.7 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">1.3 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.3 minutes<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">&gt;0.1 minutes<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid black; padding: 5px;\">100 GB<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">4.6 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">2.3 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.5 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.2 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.05 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.02 hours<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid black; padding: 5px;\">1 TB<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">46.6 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">23.3 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">4.7 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">2.3 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.5 hours<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.2 hours<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid black; padding: 5px;\">10 TB<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">19.4 days<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">9.7 days<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">1.9 days<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.9 days<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.2 days<\/td>\n<td style=\"border: 1px solid black; padding: 5px;\">0.1 days<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><em>Copy duration calculations based on data size and the bandwidth limit for the environment.<\/em><\/p>\n<h4>User experience<\/h4>\n<p>Users and systems need a way to perform manual and automated storage actions with graphical, command line, or API-initiated experiences.<\/p>\n<ul class=\"c-list\">\n<li><strong>Graphical user experience:<\/strong> <a href=\"https:\/\/azure.microsoft.com\/en-us\/features\/storage-explorer\/\" target=\"_blank\" rel=\"noopener\">Microsoft Azure Storage Explorer<\/a> provides Storage Admins the ability to graphically manage storage. It also has storage consumer features for those who don\u2019t have permissions for Administrative actions, and simply need to perform common storage actions like uploading, downloading, etc.<\/li>\n<li><strong>Command line experience:<\/strong> <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/storage-use-azcopy-v10\" target=\"_blank\" rel=\"noopener\">AzCopy<\/a> provides developers with an easy way to automate common storage actions through CLI or scheduled tasks.<\/li>\n<li><strong>Automated experiences:<\/strong> Both <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/data-factory\/\" target=\"_blank\" rel=\"noopener\">Microsoft Azure Data Factory<\/a> and AzCopy provide the ability for applications to use Azure Data Lake Gen2 storage as its primary storage source and destination.<\/li>\n<\/ul>\n<h4><strong>Identifying personas<\/strong><\/h4>\n<p>Because a diverse set of personas utilize storage for different purposes, we need to design storage experiences that satisfy the range of business needs. Through the process of development, we identified these custom persona experiences relevant to both storage and data transfer:<\/p>\n<ul class=\"c-list\">\n<li><strong>Storage Admins:<\/strong> The Storage Admins are Microsoft Azure subscription owners. Within the Azure subscription they create, manage, and maintain all aspects of MDTS: Storage Accounts, Data Factories, Storage Actions Service, and Self-Service Portal. Storage Admins also resolve requests and incidents that are not handled via Self-Service.<\/li>\n<li><strong>Data Owners: <\/strong>The Data Owner personas are those requesting storage who have the authority to create shares and authorize storage. Data Owners also perform the initial steps of creating automated distributions of data to and from private sites. Data Owners are essentially the decision makers of the storage following handoff of a storage account from Storage Admins.<\/li>\n<li><strong>Storage Consumers:<\/strong> At Microsoft, storage consumers represent a broad set of disciplines, from engineers and developers to project managers and marketing professionals. Storage Consumers can use Microsoft Azure Storage Explorer to perform storage actions to and from authorized storage paths (aka Shares). Within the MDTS Self Service Portal, a storage consumer can be given authorization to create distributions. A distribution can automate the transfer of data from a source to one or multiple destinations.<\/li>\n<\/ul>\n<h2>Implementing and enhancing the solution architecture<\/h2>\n<p>After considering multiple Microsoft Azure storage types and complimentary Azure Services, the MDTS team chose the following Microsoft Azure services and software as the foundation for offering a storage and data transfer service to Microsoft Engineering Groups.<\/p>\n<ul class=\"c-list\">\n<li><strong>Microsoft Azure Active Directory:<\/strong> Meets the requirements for authentication and access.<\/li>\n<li><strong>Microsoft Azure Data Lake Gen2:<\/strong> Meets security and performance requirements by providing encryption, OAuth, Hierarical namespace, fine grained authorization to Azure Active Directory entities, and 10+ GB per sec ingress and egress.<\/li>\n<li><strong>Microsoft Azure Storage Explorer:<\/strong> Meets security, performance, and user experience requirements by providing a graphical experience to perform storage administrative tasks and storage consumer tasks without needing a storage account key or role based access (RBAC) on an Azure resource. Azure Storage Explorer also has AzCopy embedded to satisfy performance for complex payloads.<\/li>\n<li><strong>AzCopy:<\/strong> Provides a robust and highly performant command line interface.<\/li>\n<li><strong>Microsoft Azure Data Factory:<\/strong> Meets the requirements for orchestrating and automating data copies between private networks and Azure Data Lake Gen2 storage paths. Azure Data Factory copy activities are equally as performant as AzCopy and satisfy security requriements.<\/li>\n<\/ul>\n<h3>Enabling Storage and Orchestration<\/h3>\n<p>As illustrated below, the first MDTS design was comprised entirely of Microsoft Azure Services with no additional investment from us other than people to manage the Microsoft Azure subscription and perform routine requests. MDTS was offered as a commodity service to engineering teams at Microsoft in January 2020. Within a few months we saw a reduction of third-party software and on-premises file server storage, which provided significant savings. This migration also contributed progress towards the company-wide objectives of Internet First and Zero Trust design patterns.<\/p>\n<figure id=\"attachment_8775\" aria-describedby=\"caption-attachment-8775\" style=\"width: 1200px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8775 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-001.png\" alt=\"The first design of MDTS provides storage and orchestration using out of the box Microsoft Azure services.\" width=\"1200\" height=\"396\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-001.png 1200w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-001-300x99.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-001-1024x338.png 1024w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-001-768x253.png 768w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><figcaption id=\"caption-attachment-8775\" class=\"wp-caption-text\">The first design of MDTS provides storage and orchestration using out of the box Microsoft Azure services.<\/figcaption><\/figure>\n<p>We initially onboarded 35 engineering teams which included 10,000 Microsoft Azure Storage Explorer users (internal and external accounts), and 600 TB per month of Microsoft Azure storage uploads and downloads. By offering the MDTS service, we saved engineering teams from having to run Azure subscriptions themselves and needing to learn the technical details of implementing a modern cloud storage solution.<\/p>\n<h4>Creating access control models<\/h4>\n<p>As a team, we quickly discovered that having specific repeatable implementation strategies was essential when configuring public facing Microsoft Azure storage. Our initial time investment was in standardizing an access control process which would simplify complexity and ensure a correct security posture before handing off storage to customers. To do this, we constructed onboarding processes for identifying the type of share for which we standardized the implementation steps.<\/p>\n<p>We implemented standard access control models for two types of shares: container shares and sub-shares.<\/p>\n<h4>Container share access control model<\/h4>\n<p>The container share access control model is used for scenarios where the data owner prefers users to have access to a broad set of data. As illustrated in the graphic below, container shares supply access to the root, or parent, of a folder hierarchy. The container is the parent. Any member of the security group will gain access to the top level. When creating a container share, we also make it possible to convert to a sub-share access control model if desired.<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8779\" aria-describedby=\"caption-attachment-8779\" style=\"width: 561px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8779 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-002.png\" alt=\"Microsoft Azure Storage Explorer grants access to the root of a folder hierarchy using the container share access control model.\" width=\"561\" height=\"955\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-002.png 561w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-002-176x300.png 176w\" sizes=\"(max-width: 561px) 100vw, 561px\" \/><figcaption id=\"caption-attachment-8779\" class=\"wp-caption-text\">Microsoft Azure Storage Explorer grants access to the root, or parent, of a folder hierarchy using the container share access control model. Both engineering and marketing are containers. Each has a specific Microsoft Azure Active Directory Security group. A top-level Microsoft Azure AD Security group is also added to minimize effort for users who should get access to all containers added to the storage account.<\/figcaption><\/figure>\n<p>This model fits scenarios where group members get Read, Write, and Execute permissions to an entire container. The authorization allows users to upload, download, create, and\/or delete folders and files. Making changes to the Access Control restricts access. For example, to create access permissions for download only, select Read and Execute.<\/p>\n<h4>Sub-share access control model<\/h4>\n<p>The sub-share access control model is used for scenarios where the data owner prefers users have explicit access to folders only. As illustrated in the graphic below, folders are hierarchically created under the container. In cases where several folders exist, a security group access control can be implemented on a specific folder. Access is granted to the folder where the access control is applied. This prevents users from seeing or navigating folders under the container other than the folders where an explicit access control is applied. When users attempt to browse the container, authorization will fail.<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8782\" aria-describedby=\"caption-attachment-8782\" style=\"width: 1200px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8782 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003.png\" alt=\"Microsoft Azure Storage Explorer grants access to sub-folder only using the sub-share access control model.\" width=\"1200\" height=\"1290\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003.png 1200w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003-279x300.png 279w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003-953x1024.png 953w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003-768x826.png 768w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-003-1005x1080.png 1005w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><figcaption id=\"caption-attachment-8782\" class=\"wp-caption-text\">Microsoft Azure Storage Explorer grants access to sub-folder only using the sub-share access control model. Members are added to the sub-share group, not the container group. The sub-share group is nested in the container group with execute permissions to allow for Read and Write on the sub-share.<\/figcaption><\/figure>\n<p>This model fits for scenarios where group members get Read, Write, and Execute permissions to a sub-folder only. The authorization allows users to upload, download, create folders\/files, and delete folders\/files. The access control is specific to the folder \u201cproject1.\u201d In this model you can have multiple folders under the container, but only provide authorization to a specific folder.<\/p>\n<p>The sub-share process is only applied if a sub-share is needed.<\/p>\n<ul class=\"c-list\">\n<li>Any folder needing explicit authorization is considered a sub-share.<\/li>\n<li>We apply a sub-share security group access control with Read, Write, and Execute on the folder.<\/li>\n<li>We nest the sub-share security group in the parent share security group used for Execute only. This allows members who do not have access to the container enough authorization to Read, Write, and Execute the specific sub-share folder without having Read or Write permissions to any other folders in the container.<\/li>\n<\/ul>\n<h4>Applying access controls for each type of share (container and or sub-share)<\/h4>\n<p>The parent share process is standard for each storage account.<\/p>\n<ul class=\"c-list\">\n<li>Each storge account has a unique security group. This security group will have access control applied for any containers. This allows data owners to add members and effectively give access to all containers (current and future) by simply changing the membership of one group.<\/li>\n<li>Each container will have a unique security group for Read, Write, and Execute. This security group is used to isolate authorization to a single container.<\/li>\n<li>Each container will have a unique group for execute. This security group is needed in the event sub-shares are created. Sub-shares are folder-specific shares in the hierarchical namespace.<\/li>\n<li>We always use the default access control option. This is a feature that automatically applies the parent permissions to all new child folders (sub-folders).<\/li>\n<\/ul>\n<p>The first design enabled us to offer MDTS while our engineers defined, designed, and developed an improved experience for all the personas. It quickly became evident that Storage Admins needed the ability to see an aggregate view of all storage actions in near real-time to successfully operate the service. It was important for our administrators to easily discover the most active accounts and which user, service principle, or managed service identity was making storage requests or performing storage actions. In July 2020, we added the Aggregate Storage Actions service.<\/p>\n<h3>Adding aggregate storage actions<\/h3>\n<p>For our second MDTS design, we augmented the out of the box Microsoft Azure Storage capabilities used in our first design with the capabilities of Microsoft Azure Monitor, Event Hubs, Stream Analytics, Function Apps, and Microsoft Azure Data Explorer to provide aggregate storage actions. Once the Aggregate Storage Actions capability was deployed and configured within MDTS, storage admins were able to aggregate the storage actions of all their storage accounts and see them in a single pane view.<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8783\" aria-describedby=\"caption-attachment-8783\" style=\"width: 1200px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8783 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-004.png\" alt=\"The second design of MDTS introduces aggregate storage actions.\" width=\"1200\" height=\"544\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-004.png 1200w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-004-300x136.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-004-1024x464.png 1024w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-004-768x348.png 768w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><figcaption id=\"caption-attachment-8783\" class=\"wp-caption-text\">The second design of MDTS introduces aggregate storage actions.<\/figcaption><\/figure>\n<p>The Microsoft Azure Storage Diagnostic settings in Microsoft Azure Portal makes it possible for us to configure specific settings for blob actions. Combining this feature with other Azure Services and some custom data manipulation gives MDTS the ability to see which users are performing storage actions, what those storage actions are, and when those actions were performed. The data visualizations are near real-time and aggregated across all the storage accounts.<\/p>\n<p>Storage accounts are configured to route logs from Microsoft Azure Monitor to Event Hub. We currently have 45+ storage accounts that generate around five million logs each day. Data filtering, manipulation, and grouping is performed by Stream Analytics. Function Apps are responsible for fetching UPNs using Graph API, then pushing logs to Microsoft Azure Data Explorer. Microsoft Power BI and our modern self-service portal query Microsoft Azure Data Explorer and provide the visualizations, including dashboards with drill down functionality. The data available in our dashboard includes the following information aggregated across all customers (currently 35 storage accounts).<\/p>\n<ul class=\"c-list\">\n<li>Aggregate view of most active accounts based on log activity.<\/li>\n<li>Aggregate total of GB uploaded and download per storage account.<\/li>\n<li>Top users who uploaded showing the user principal name (both external and internal).<\/li>\n<li>Top users who downloaded showing the user principal name (both external and internal).<\/li>\n<li>Top Accounts uploaded data.<\/li>\n<li>Top Accounts downloaded data.<\/li>\n<\/ul>\n<p>The only setting required to onboard new storage accounts is to configure them to route logs to the Event Hub. Because we can have an aggregate store of all the storage account activities, we are able to offer MDTS customers an account view into their storage account specific data.<\/p>\n<p>Following the release of Aggregate Storage Actions, the MDTS team, along with feedback from customers, identified another area of investment\u2014the need for storage customers to \u201cself-service\u201d and view account specific insights without having role-based access to the subscription or storage accounts.<\/p>\n<h3>Providing a self-service experience<\/h3>\n<p>To enhance the experience of the other personas, MDTS is now focused on the creation of a Microsoft Azure web portal where customers can self-service different storage and transfer capabilities without having to provide any Microsoft Azure Role Based Access (RBAC) to the underlying subscription that hosts the MDTS service.<\/p>\n<p>When designing MDTS self-service capabilities we focused on meeting these primary goals:<\/p>\n<ul class=\"c-list\">\n<li>Make it possible for Microsoft Azure Subscription owners (Storage Admins) to provide the platform and services while not needing to be in the middle of making changes to storage and transfer services.<\/li>\n<li>The ability to create custom persona experiences so customers can achieve their storage and transfer goals through a single portal experience in a secure and intuitive way. Some of the new enterprise scale capabilities include:\n<ul class=\"c-list\">\n<li>Onboarding.<\/li>\n<li>Creating storage shares.<\/li>\n<li>Authorization changes.<\/li>\n<li>Distributions. Automating the distribution of data from one source to one or multiple destinations.<\/li>\n<li>Provide insights into storage actions (based on the data provided in Storage Actions enabled in our second MDTS release).<\/li>\n<li>Reporting basic consumption data, like the number of users, groups, and shares on a particular account.<\/li>\n<li>Reporting the cost of the account<\/li>\n<\/ul>\n<\/li>\n<li>As Azure services and customer scenarios change, the portal can also change.<\/li>\n<li>If customers want to \u201cself-host\u201d (essentially take our investments and do it themselves), we will easily be able to accommodate.<\/li>\n<\/ul>\n<figure id=\"attachment_8784\" aria-describedby=\"caption-attachment-8784\" style=\"width: 1200px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8784 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-005.png\" alt=\"Our next design of MDTS introduces a self-service portal.\" width=\"1200\" height=\"750\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-005.png 1200w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-005-300x188.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-005-1024x640.png 1024w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-005-768x480.png 768w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><figcaption id=\"caption-attachment-8784\" class=\"wp-caption-text\">Our next design of MDTS introduces a self-service portal.<\/figcaption><\/figure>\n<h2>Storage consumer user experiences<\/h2>\n<p>After storage is created and configured, data owners can then share steps for storage consumers to start using storage. Upload and download are the most common storage actions, and Microsoft Azure provides software and services needed to perform both actions for <em>manual<\/em> and <em>automated<\/em> scenarios.<\/p>\n<p>Microsoft Azure Storage Explorer is recommended for manual scenarios where users can connect and perform high speed uploads and downloads manually. Both Microsoft Azure Data Factory and AzCopy can be used in scenarios where automation is needed. AzCopy is heavily preferred in scenarios where synchronization is required. Microsoft Azure Data Factory doesn\u2019t provide synchronization but does provide robust data copy and data move. Azure Data Factory is also a managed service and better suited in enterprise scenarios where flexible triggering options, uptime, auto scale, monitoring, and metrics are required.<\/p>\n<h3>Using Microsoft Azure Storage Explorer for manual storage actions<\/h3>\n<p>Developers and Storage Admins are accustomed to using Microsoft Azure Storage Explorer for both storage administration and routine storage actions (e.g., uploading and downloading). Non-storage admin, otherwise known as Storage Consumers, can also use Microsoft Azure Storage Explorer to connect and perform storage actions without needing any role-based access control or access keys to the storage account. Once the storage is authorized, members of authorized groups can perform routine steps to attach the storage they are authorized for, authenticating with their work email, and leveraging the options based on their authorization.<\/p>\n<p>The processes for sign-in and adding a resource via Microsoft Azure Active Directory are found in the <strong>Manage Accounts<\/strong> and <strong>Open Connect Dialogue<\/strong> options of Microsoft Azure Storage Explorer.<\/p>\n<p>After signing in and selecting the option to add the resource via Microsoft Azure Active Directory, you can supply the storage URL and connect. Once connected, it only requires a few clicks to upload and download data.<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8785\" aria-describedby=\"caption-attachment-8785\" style=\"width: 789px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8785 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-006.png\" alt=\"Microsoft Azure Storage Explorer Local and Attached module.\" width=\"789\" height=\"224\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-006.png 789w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-006-300x85.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-006-768x218.png 768w\" sizes=\"(max-width: 789px) 100vw, 789px\" \/><figcaption id=\"caption-attachment-8785\" class=\"wp-caption-text\">Microsoft Azure Storage Explorer Local and Attached module. After following the add resource via Microsoft Azure AD process, the Azure AD group itshowcase-engineering is authorized to Read, Write, and Edit (rwe) and members of the group can perform storage actions.<\/figcaption><\/figure>\n<p>To learn more about using Microsoft Azure Storage Explorer, <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/vs-azure-tools-storage-manage-with-storage-explorer?tabs=windows#sign-in-to-azure\" target=\"_blank\" rel=\"noopener\">Get started with Storage Explorer<\/a>. There are additional links in the <em>More Information<\/em> section at the end of this document.<\/p>\n<p><strong>Note<\/strong>: Microsoft Azure Storage Explorer uses AzCopy. Having AzCopy as the transport allows storage consumers to benefit from high-speed transfers. If desired, AzCopy can be used as a stand-alone command line application.<\/p>\n<h3>Using AzCopy for manual or automated storage actions<\/h3>\n<p>AzCopy is a command line interface used to perform storage actions on authorized paths. AzCopy is used in Microsoft Azure Storage Explorer but can also be used as a standalone executable to automate storage actions. It\u2019s a multi-stream TCP based transport capable of optimizing throughput based on the bandwidth available. MDTS customers use AzCopy in scenarios which require synchronization or cases where Microsoft Azure Storage Explorer or Microsoft Azure Data Factory copy activity doesn\u2019t meet the requirements for data transfer. For more information about using AzCopy please see the <em>More Information<\/em> section at the end of this document.<\/p>\n<p>AzCopy is a great match for standalone and synchronization scenarios. It also has options that are useful when seeking to automate or build applications. Because AzCopy is a single executable running on either a single client or server system, it isn\u2019t always ideal for enterprise scenarios. Microsoft Azure Data Factory is a more robust Microsoft Azure service that meets the most enterprise needs.<\/p>\n<h3>Using Microsoft Azure Data Factory for automated copy activity<\/h3>\n<p>Some of the teams that use MDTS require the ability to orchestrate and operationalize storage uploads and downloads. Before MDTS, we would have either built a custom service or licensed a third-party solution, which can be expensive and\/or time consuming.<\/p>\n<p>Microsoft Azure Data Factory, a cloud-based ETL and data integration service, allows us to create data-driven workflows for orchestrating data movement. Including Azure Data Factory in our storage hosting service model provided customers with a way to automate data copy activities. MDTS\u2019s most common data movement scenarios are distributing builds from a single source to multiple destinations (3-5 destinations are common).<\/p>\n<p>Another requirement for MSTS was to leverage private data stores as a source or destination. Microsoft Azure Data Factory provides the capability to use a private system, also known as a self-hosted integration runtime. When configured this system can be used in copy activity communicating with on-premises file systems. The on-premises file system can then be used as a source and\/or destination datastore.<\/p>\n<p>In the situation where on-premises file system data needs to be stored in Microsoft Azure or shared with external partners, Microsoft Azure Data Factory provides the ability to orchestrate pipelines which perform one or multiple copy activities in sequence. These activities result in end-to-end data movement from one on-premises file systems to Microsoft Azure Storage, and then to another private system if desired.<\/p>\n<p>The graphic below provides an example of a pipeline orchestrated to copy builds from a single source to several private destinations.<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_8787\" aria-describedby=\"caption-attachment-8787\" style=\"width: 736px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8787 size-full\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-007.png\" alt=\"Detailed example of a Microsoft Azure Data Factory pipeline, including the build system source.\" width=\"736\" height=\"597\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-007.png 736w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146-img-007-300x243.png 300w\" sizes=\"(max-width: 736px) 100vw, 736px\" \/><figcaption id=\"caption-attachment-8787\" class=\"wp-caption-text\">Microsoft Azure Data Factory pipeline example. Private site 1 is the build system source. Build system will build, load the source file system, then trigger the Microsoft Azure Data Factory pipeline. Build is then uploaded, then Private sites 2, 3, 4 will download. Function apps are used for sending email notifications to site owners and additional validation.<\/figcaption><\/figure>\n<p>For more information on Azure Data Factory, please see Introduction to <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/data-factory\/introduction\" target=\"_blank\" rel=\"noopener\">Microsoft Azure Data Factory<\/a>. There are additional links in the <em>More Information<\/em> section at the end of this document.<\/p>\n<p>If you are thinking about using Microsoft Azure to develop a modern data transfer and storage solution for your organization, here are some of the best practices we gathered while developing MDTS.<\/p>\n<h3>Close the technical gap for storage consumers with a white glove approach to onboarding<\/h3>\n<p>Be prepared to spend time with customers who are initially overwhelmed with using Azure Storage Explorer or AzCopy. At Microsoft, storage consumers represent a broad set of disciplines\u2014from engineers and developers to project managers and marketing professionals. Azure Storage Explorer provides an excellent experience for engineers and developers but can be a little challenging for less technical roles.<\/p>\n<h3>Have a standard access control model<\/h3>\n<p>Use Microsoft Azure Active Directory security groups and group nesting to manage authorization; Microsoft Azure Data Lake Gen2 storage has a limit to the number of Access Controls you can apply. To avoid reaching this limit, and to simplify administration, we recommend using Microsoft Azure Active Directory security groups. We apply the access control to the security group only, and in some cases, we nest other security groups within the access control group. We nest Member Security Groups within Access Control Security Groups to manage access. These group types don\u2019t exist in Microsoft Azure Active Directory but do exist within our MDTS service as a process to differentiate the purpose of a group. We can easily determine this differentiation by the name of the group.<\/p>\n<ul class=\"c-list\">\n<li><strong>Access Control Security Groups:<\/strong> We use this group type for applying Access Control on ADLS Gen2 storage containers and\/or folders.<\/li>\n<li><strong>Member Security Groups:<\/strong> We use these to satisfy cases where access to containers and\/or folders will constantly change for members.<\/li>\n<\/ul>\n<p>When there are large numbers of members, nesting prevents the need to add members individually to the Access Control Security Groups. When access is no longer needed, we can remove the Member Group(s) from the Access Control Security Group and no further action is needed on storage objects.<\/p>\n<p>Along with using Microsoft Azure Active Directory security groups, make sure to have a documented process for applying access controls. Be consistent and have a way of tracking where access controls are applied.<\/p>\n<h3>Use descriptive display names for your Microsoft Azure AD security groups<\/h3>\n<p>Because Microsoft Azure AD doesn&#8217;t currently organize groups by owners, we recommend using naming conventions that capture the group\u2019s purpose and type to allow for easier searches.<\/p>\n<ul class=\"c-list\">\n<li><strong>Example 1:<\/strong> mdts-ac-storageacct1-rwe. This group name uses our service standard naming convention for Access Control group type on Storage Account 1, with access control Read, Write, and Execute. mdts = Service, ac = Access Control Type, storageacct1 = ADLS Gen2 Storage Account Name, rwe = permission of the access control.<\/li>\n<li><strong>Example 2:<\/strong> mdts-mg-storageacct1-project1. This group name uses our service standard naming convention for Member Group type on Storage Account 1. This group does not have an explicit access control on storage, but it is nested in mdts-ac-storageacct1-rwe where any member of this group has the Read, Write, and Execute access to storage account1 because it&#8217;s nested in mdts-ac-storageacct1-rwe.<\/li>\n<\/ul>\n<h3>Remember to propagate any changes to access controls<\/h3>\n<p>Microsoft Azure Data Lake Gen2 storage, by default, doesn\u2019t automatically propagate any access control changes. As such, when removing, adding, or changing an access control, you need to follow an additional step to propagate the access control list. This option is available in Microsoft Azure Storage Explorer.<\/p>\n<h3>Storage Consumers can attempt Administrative options<\/h3>\n<p>Storage Consumers use Microsoft Azure Storage Explorer and are authenticated with their Microsoft Azure Active Directory user profile. Since Azure Storage Explorer is primarily developed for Storage Admin and Developer personas, all administrative actions are visible. It is common for storage consumers to attempt administrative actions, like managing access or deleting a container. Those actions will fail due to only being accessed via access control lists (ACLs). There isn\u2019t a way to provide administration actions via ACL\u2019s. If administrative actions are needed, then users will become a Storage Admin which has access via Azure\u2019s Role Based Access Control (RBAC).<\/p>\n<h3>Microsoft Azure Storage Explorer and AzCopy are throughput intensive<\/h3>\n<p>As stated above, AzCopy is leveraged by Microsoft Azure Storage Explorer for transport actions. When using Azure Storage Explorer or AzCopy it\u2019s important to understand that transfer performance is its specialty. Because of this, some clients and\/or networks may benefit from throttling AzCopy\u2019s performance. In circumstances where you don\u2019t want AzCopy to consume too much network bandwidth, there are configurations available. In Microsoft Azure Storage Explorer use the <strong>Settings<\/strong> option and select the <strong>Transfers<\/strong> section to configure <strong>Network Concurrency<\/strong> and\/or <strong>File Concurrency<\/strong>. In the <strong>Network<\/strong> <strong>Concurrency<\/strong> section, <strong>Adjust<\/strong> <strong>Dynamically<\/strong> is a default option. For AzCopy, there are flags and environment variables available to optimize performance.<\/p>\n<p>For more information, visit <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/storage-use-azcopy-configure\" target=\"_blank\" rel=\"noopener\">Configure, optimize, and troubleshoot AzCopy<\/a>.<\/p>\n<h3>Microsoft Azure Storage Explorer sign-in with MSAL<\/h3>\n<p>Microsoft Authentication Library, currently in product preview, provides enhanced single sign-on, multi-factor authentication, and conditional access support. In some situations, users won\u2019t authenticate unless MSAL is selected. To enable MSAL, select the <strong>Setting<\/strong> option from Microsoft Azure Storage Explorer\u2019s navigation pane. Then in the application section, select the option to enable <strong>Microsoft Authentication Library<\/strong>.<\/p>\n<h3>B2B invites are needed for external accounts (guest user access)<\/h3>\n<p>When there is a Microsoft business need to work with external partners, leveraging guest user access in Microsoft Azure Active Directory is necessary. Once the B2B invite process is followed, external accounts can be authorized by managing group membership. For more information, read <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/active-directory\/external-identities\/what-is-b2b\" target=\"_blank\" rel=\"noopener\">What is B2B collaboration in Azure Active Directory?<\/a><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"74\" class=\"alignnone size-medium wp-image-7448\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/key-takeaways-300x74.png\" alt=\"Key Takeaways\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/key-takeaways-300x74.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/key-takeaways.png 500w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<p>We used Microsoft Azure products and services to create an end-to-end modern data transfer and storage service that can be used by any group at Microsoft that desires cloud data storage. The release of Microsoft Azure Data Lake Gen 2, Microsoft Azure Data Factory, and the improvements in the latest release of Azure Storage Explorer made it possible for us to offer MDTS as a fully native Microsoft Azure service.<\/p>\n<p>One of the many strengths of using Microsoft Azure is the ability to use only what we needed, as we needed it. For MDTS, we started by simply creating storage accounts, requesting Microsoft Azure Active Directory Security Groups, applying an access control to storage URLs, and releasing the storage to customers for use. We then invested in adding storage actions and developed self-service capabilities that make MDTS a true enterprise-scale solution for data transfer and storage in the cloud.<\/p>\n<p>We are actively encouraging the adoption of our MDTS storage design to all Microsoft engineering teams that still rely on legacy storage hosted in the Microsoft Corporate network. We are also encouraging any Microsoft Azure consumers to consider this design when evaluating options for storage and file sharing scenarios. Our design has proven to be scalable, compliant, and performant with the Microsoft Zero Trust security initiative, handling extreme payloads with high throughput and no constraints on the size or number of files.<\/p>\n<p>By eliminating our dependency on third-party software, we have been able to eliminate third-party licensing, consulting, and hosting costs for many on-premises storage systems.<\/p>\n<p>Are you ready to learn more? <a href=\"https:\/\/portal.azure.com\/\" target=\"_blank\" rel=\"noopener\">Sign up for your own Microsoft Azure subscription and get started today<\/a>.<\/p>\n<p>To receive the latest updates on Azure storage products and features to meet your cloud investment needs, visit <a href=\"https:\/\/azure.microsoft.com\/en-us\/updates\/?category=storage\" target=\"_blank\" rel=\"noopener\">Microsoft Azure updates<\/a>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"81\" class=\"alignnone size-medium wp-image-7482\" src=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/related_links-300x81.png\" alt=\"Related links\" srcset=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/related_links-300x81.png 300w, https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2021\/10\/related_links.png 500w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<ul class=\"c-list\">\n<li><a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/simplifying-microsofts-royalty-ecosystem-with-connected-data-service\/\">Unpack simplifying Microsoft\u2019s royalty ecosystem with connected data service<\/a>.<\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/how-microsoft-employees-are-leveraging-the-cloud-for-file-storage-with-onedrive-folder-backup\/\">Check out how Microsoft employees are leveraging the cloud for file storage with OneDrive Folder Backup<\/a>.<\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/simplifying-compliance-evidence-management-with-microsoft-azure-confidential-ledger\/\">Read more on simplifying compliance evidence management with Microsoft Azure confidential ledger<\/a>.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Companies all over the world have launched their cloud adoption journey. While some are just starting, others are further along the path and are now researching the best options for moving their largest, most complex workflows to the cloud. It can take time for companies to address legacy tools and systems that have on-premises infrastructure [&hellip;]<\/p>\n","protected":false},"author":133,"featured_media":8733,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"_hide_featured_on_single":false,"_show_featured_caption_on_single":true,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[1],"tags":[373,361,331],"coauthors":[646],"class_list":["post-8732","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-azure-data-and-storage","tag-azure-identity-and-security","tag-data-and-ai","program-microsoft-digital-technical-stories","m-blog-post"],"jetpack_publicize_connections":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog<\/title>\n<meta name=\"description\" content=\"Learn how Microsoft retooled the way it transfers large files in and out of the company.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog\" \/>\n<meta property=\"og:description\" content=\"Learn how Microsoft retooled the way it transfers large files in and out of the company.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/\" \/>\n<meta property=\"og:site_name\" content=\"Inside Track Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-07-13T14:54:07+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-07-13T15:20:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1040\" \/>\n\t<meta property=\"og:image:height\" content=\"585\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Inside Track staff\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Inside Track staff\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"27 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/\",\"url\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/\",\"name\":\"Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg\",\"datePublished\":\"2023-07-13T14:54:07+00:00\",\"dateModified\":\"2023-07-13T15:20:22+00:00\",\"author\":{\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/d87d5d73eed76dd055086299c842c17e\"},\"description\":\"Learn how Microsoft retooled the way it transfers large files in and out of the company.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage\",\"url\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg\",\"contentUrl\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg\",\"width\":1040,\"height\":585,\"caption\":\"Deploying a Modern Data Transfer Service internally at Microsoft is providing the company with an enterprise-scale system for transferring large files in and out of its firewall.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Providing modern data transfer and storage service at Microsoft with Microsoft Azure\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/#website\",\"url\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/\",\"name\":\"Inside Track Blog\",\"description\":\"How Microsoft does IT\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/d87d5d73eed76dd055086299c842c17e\",\"name\":\"Inside Track staff\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/image\/fb338dea806293f999356f7d8a399348\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/3788032f4762be09ee17b2ec39f3d75b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/3788032f4762be09ee17b2ec39f3d75b?s=96&d=mm&r=g\",\"caption\":\"Inside Track staff\"},\"description\":\"Questions? Send us a note: msitstaff@microsoft.com\",\"url\":\"https:\/\/www.microsoft.com\/insidetrack\/blog\/author\/insidetrack\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog","description":"Learn how Microsoft retooled the way it transfers large files in and out of the company.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/","og_locale":"en_US","og_type":"article","og_title":"Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog","og_description":"Learn how Microsoft retooled the way it transfers large files in and out of the company.","og_url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/","og_site_name":"Inside Track Blog","article_published_time":"2023-07-13T14:54:07+00:00","article_modified_time":"2023-07-13T15:20:22+00:00","og_image":[{"width":1040,"height":585,"url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg","type":"image\/jpeg"}],"author":"Inside Track staff","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Inside Track staff","Est. reading time":"27 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/","url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/","name":"Providing modern data transfer and storage service at Microsoft with Microsoft Azure - Inside Track Blog","isPartOf":{"@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage"},"image":{"@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg","datePublished":"2023-07-13T14:54:07+00:00","dateModified":"2023-07-13T15:20:22+00:00","author":{"@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/d87d5d73eed76dd055086299c842c17e"},"description":"Learn how Microsoft retooled the way it transfers large files in and out of the company.","breadcrumb":{"@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#primaryimage","url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg","contentUrl":"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg","width":1040,"height":585,"caption":"Deploying a Modern Data Transfer Service internally at Microsoft is providing the company with an enterprise-scale system for transferring large files in and out of its firewall."},{"@type":"BreadcrumbList","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-uses-azure-to-provide-a-modern-data-transfer-and-storage-service\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.microsoft.com\/insidetrack\/blog\/"},{"@type":"ListItem","position":2,"name":"Providing modern data transfer and storage service at Microsoft with Microsoft Azure"}]},{"@type":"WebSite","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/#website","url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/","name":"Inside Track Blog","description":"How Microsoft does IT","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/d87d5d73eed76dd055086299c842c17e","name":"Inside Track staff","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/insidetrack\/blog\/#\/schema\/person\/image\/fb338dea806293f999356f7d8a399348","url":"https:\/\/secure.gravatar.com\/avatar\/3788032f4762be09ee17b2ec39f3d75b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3788032f4762be09ee17b2ec39f3d75b?s=96&d=mm&r=g","caption":"Inside Track staff"},"description":"Questions? Send us a note: msitstaff@microsoft.com","url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/author\/insidetrack\/"}]}},"jetpack_featured_media_url":"https:\/\/www.microsoft.com\/insidetrack\/blog\/uploads\/prod\/2022\/11\/10146_hero.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9hcZA-2gQ","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/posts\/8732"}],"collection":[{"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/users\/133"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/comments?post=8732"}],"version-history":[{"count":24,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/posts\/8732\/revisions"}],"predecessor-version":[{"id":8844,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/posts\/8732\/revisions\/8844"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/media\/8733"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/media?parent=8732"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/categories?post=8732"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/tags?post=8732"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.microsoft.com\/insidetrack\/blog\/wp-json\/wp\/v2\/coauthors?post=8732"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}