Microsoft SQL Server 2025, now in preview, is an enterprise AI-ready database from ground to cloud that tackles these challenges by bringing AI to customers’ data. This release continues SQL Server’s three decades of innovation in performance and security, adding new AI capabilities. With Microsoft Fabric integration, customers can bring their data into the next generation of data analytics. The release supports hybrid environments across clouds, on-premises datacenters, and edge, leveraging Microsoft Azure innovation for customers’ databases.

Over the years, SQL Server has transcended well beyond a traditional relational database. With the latest release of SQL Server, we’re enabling customers to build AI applications deeply integrated with the SQL engine. SQL Server 2025 is transforming into a vector database in its own right, using the built-in filtering capabilities along with a vector search, with great performance and is easily consumable by developers using T-SQL.

AI built-in

This new version has AI built-in, simplifying AI application development and retrieval-augmented generation (RAG) patterns with secure, performant, and easy-to-use vector support, leveraging familiar T-SQL syntax. With this new capability, you can combine vectors with your SQL data for a hybrid AI vector search.

Build AI applications with your enterprise database

SQL Server 2025 is an enterprise-ready vector database with built-in security and compliance, bringing enterprise AI to your data. It features a native vector store and index powered by DiskANN, a vector search technology using disk storage to efficiently find similar data points in large datasets. These databases efficiently support chunking and enable accurate data retrieval through semantic searching. In this latest SQL Server version, flexible AI model management within the engine using Representational State Transfer (REST) interfaces allows you to use AI models from ground to cloud.

In addition, whether customers are working on data preprocessing, model training, or RAG patterns, extensible, low-code tools offer flexible model interfaces within the SQL engine, supported by T-SQL and external REST endpoints. These tools enhance developers’ ability to create various AI applications through seamless integration with popular AI frameworks like LangChain, Semantic Kernel, and Entity Framework Core.

Boost developer productivity

When building data-intensive applications such as AI applications, it’s critical to focus on extensibility, frameworks, and data enrichment to enhance developers’ productivity. We ensure SQL will provide a best-in-class experience for developers by incorporating features such as REST API support, GraphQL integration through Data API Builder, and Regular Expression enablement. Additionally, native JSON support enables developers to more effectively deal with frequently changing schema and hierarchical data, facilitating the creation of more dynamic applications. Overall, we’re enhancing SQL development to be more extensible, performant, and user-friendly. All functionalities are underpinned by the security provided by the SQL Server engine, making it a truly enterprise-ready platform for AI.

Best-in-class security and performance

SQL Server 2025 is an industry leader in database security and performance. Support for Microsoft Entra managed identities improves credential management, reduces potential vulnerabilities, and provides compliance and auditing capabilities. SQL Server 2025 introduces outbound authentication support for MSI (Managed Service Identity) for SQL Server enabled by Azure Arc.

We’re also introducing performance and availability enhancements, extensively battle-tested on Microsoft Azure SQL, to SQL Server. In the new version you can boost workload performance and reduce troubleshooting with enhanced query optimization and query performance execution. Optional Parameter Plan Optimization (OPPO) is designed to enable SQL Server to choose the optimal execution plan based on customer-provided runtime parameter values and to significantly reduce bad parameter sniffing problems that can exist in workloads. Persisted statistics on secondary replicas prevent the loss of statistics during a restart or failover, thereby avoiding potential performance degradation. Regarding query execution, the improvements in batch mode processing and columnstore indexing further establish SQL Server as a mission-critical database for analytical workloads.   

Optimized locking reduces lock memory consumption and minimizes blocking for concurrent transactions through Transaction ID (TID) Locking and Lock After Qualification (LAQ). This capability enables customers to increase uptime and enhance concurrency and scale for SQL Server applications. 

Change event streaming for SQL Server brings real-time application integration with event driven architectures, command query responsibility segregation, and real-time intelligence. This will add new database engine capabilities to capture and publish incremental changes to data and schema to a provided destination such as Azure Event Hubs and Kafka in near real-time.

Microsoft Fabric and Azure Arc connected

In traditional data warehouse and data lake scenarios, integrating all your data involves designing, monitoring, and managing complex ETL (Extract, Transform, Load) processes to transfer operational data from SQL Server. These traditional methods do not support real-time data transfer, resulting in latency that prevents the creation of real-time analytics. Microsoft Fabric offers comprehensive, integrated, and AI-enhanced data analytics services designed to meet modern requirements of analytical workloads. Mirrored SQL Server Database in Fabric is a fully managed, resilient process that simplifies SQL Server data replication to Microsoft OneLake in near real-time. Mirroring will enable customers to continuously replicate data from SQL Server databases running on Azure virtual machines or outside of Azure, serving online transaction processing (OLTP) or operational store workloads directly into OneLake in order to facilitate analytics and insights on the unified Fabric data platform.

Azure continues to be a critical component of SQL Server. With Azure Arc, SQL Server 2025 will continue to offer cloud capabilities to enable customers better manage, secure, and govern their SQL estate at scale across on-premises and cloud. Capabilities like automatic patching, automatic backups, monitoring, and Best Practices Assessment offer customers more ways to streamline routine tasks and further enhance their business continuity. In addition, Azure Arc simplifies SQL Server licensing by offering a pay-as-you-go option, bringing flexibility and licensing visibility to our customers.

Sign up for the preview today

We’re currently onboarding customers and partners to SQL Server 2025 preview, in advance of general availability in the coming year. 

Register today to apply for the SQL Server 2025 Community Technology Preview (CTP)¹ and stay informed about SQL Server 2025 updates.

Microsoft just announced the upcoming release of SQL Server Management Studio (SSMS) 21 Preview 1. This release integrates Microsoft Copilot capabilities into SSMS. The Copilot experience streamlines SQL development by offering real-time suggestions, code completions, and best practice recommendations. If you would like to take part and have an early hands-on experience with this new capability, please use this link to indicate your interest.

¹Some of the new capabilities covered in this blog may not be available in the first CTP version.

The post Announcing Microsoft SQL Server 2025: Enterprise AI-ready database from ground to cloud appeared first on Microsoft SQL Server Blog.

This week, we are proud to be back in front of our community at SQLBits, together with our partner AMD to share the latest innovations from ground to cloud and beyond. From SQL Server, Azure SQL, to our powerful solutions like Microsoft Fabric and Microsoft Copilot, these product enhancements are designed to help make application migration, modernization, and development easier so you can power what’s next.

Better performance with the next generation of Azure SQL Managed Instance ​ 

What sets the SQL Server family apart from other operational databases is that it’s all built on the same SQL engine. So, whether you’re running at the edge or in the cloud, the power to unlock the potential of data and AI is always there. We’re announcing the public preview of Azure SQL Managed Instance Next-gen GP, now even more powerful and performant. Accelerate your migrations and efficiently manage your unique workload demands. Enhance your productivity with superior performance and adaptable compute and storage choices. Join the preview to maximize your efficiency.  

We now have a way customers can get stated with Azure SQL Managed Instance. Our free offer gives you:

• A General Purpose instance with up to 100 databases
• 720 vCore hours of compute every month
• 64 GB of storage

Azure SQL Managed Instance

Enhance your productivity with superior performance and more

Try it now free

AI-power your Azure SQL Database experience with Copilot 

We are bringing the power of Copilot to Azure SQL Database, now in private preview. Copilot in Azure SQL Databases delivers a set of AI-enhanced experiences built to help streamline design, operation, optimization of Azure SQL Database-driven applications, and improve productivity in the Azure Portal. This new functionality introduces two new Azure portal experiences: 

• Natural language to SQL: This experience within the Azure portal query editor for Azure SQL Database translates natural language queries into SQL, making database interactions more intuitive.  
• Microsoft Copilot for Azure integration: This experience adds Azure SQL Database skills into Copilot for Azure, customers with self-guided assistance, empowering them to manage their databases and solve issues independently.  

Sign up for the preview access.

Simplify your journey to Azure with SQL Server enabled by Azure Arc 

Earlier this month, we introduced a way to streamline migration to Azure SQL with SQL Server enabled by Azure Arc. Migration assessment removes some of the complexity around cloud migration by helping you better assess your SQL Server readiness for Azure SQL. Through the Azure Arc agent, customers can get help with:

• Streamlining discovery and migration readiness assessments.
• Evaluating and measuring the readiness of SQL Server instance and databases.
• Getting best-fit recommendations.   

Learn about this assessment “SQL Server enabled by Azure Arc, now assists in selecting the best Azure SQL target.” 

Deliver better value and power AI with Flexible Server in Azure Database for PostgreSQL 

In November 2023, we announced the preview of the new Azure AI extension, enabling you to integrate Azure AI services with your operational data in Azure Database for PostgreSQL. Now, we’re sharing that Flexible Server in Azure Database for PostgreSQL is now directly integrated with Azure OpenAI Service. Learn how to use Azure AI with Azure Database for PostgreSQL.

We were excited to have our AMD partners join us to co-sponsor SQLBits to showcase how best in class technology partnerships can help customers achieve their business outcomes for both SQL Server and PostgreSQL workloads. A recently commissioned Principled Technologies report found that for customers who migrated to Azure Databases for PostgreSQL—flexible server, backed by AMD EPYC™ processors—saw significantly faster online transaction processing (OLTP) performance, in fact, 4.71 times new orders per minute when compared to single server. Customers also achieved better value, 3.88 times the performance per dollar. Read the full report.

We look forward to the week ahead and connecting with you in person. 

The post Expand the limits of innovation with Azure data appeared first on Microsoft SQL Server Blog.

You have told us that in today’s demanding world of massive data, wide variety of data sources, and expectations of near real-time application and query performance you need more than just a database engine. You need a modern data platform.

SQL Server 2019 encompasses all of this in one product, SQL Server 2019 brings enhancements in the core SQL engine, offers a scale-up and scale-out system with built in support for Big Data (Apache Spark, Data Lake), state of the art data virtualization technology, and with built-in machine learning capabilities.

SQL Server 2019 is designed to solve challenges of the modern data professional including:

• Store enterprise data in a data lake and offer SQL and Spark query capability overall data
• Reduce the need for Extract, Transform, and Load (ETL) applications by eliminating data movement
• Integrate and secure machine learning applications with scalable performance
• Reduce the need for application and query changes to gain a boost in performance
• Increase confidential computing of data through hardware enclaves
• Increase application and database uptime and availability through features like ADR (Advanced Database Recovery)
• Extend the power of the T-SQL language in a secure and robust fashion
• Run applications and deploy databases across multiple operating systems and platforms with compatibility
• Reduce the risk of upgrades while using new SQL capabilities when you are ready though inbuilt database compatibility levels

Unified data platform

Enterprises today use multiple data platforms to meet their business needs. This ranges from operational databases, to data-marts, and to Big Data platforms. These platforms have different security models and tools ecosystems, which often require different skill sets and domain expertise. SQL Server 2019 offers all these capabilities as a unified data platform.

Customers can choose to deploy SQL Server in a traditional pattern. SQL Server 2019 also supports big data cluster deployment that comes with additional capabilities of big data, data virtualization, data-mart and enterprise data lake.

Big Data Clusters come with an easy to use deployment and management experience through the azdata command line tool, or the GUI tool Azure Data Studio. Azure Data Studio is a cross platform, modern tool built on top of VS Code. ADS provides experiences to access Big Data Clusters through interactive dashboards, and also includes rich SQL and Jupyter Notebooks experiences.

Azure Data Studio offers a unified view across all enterprise data both on-premises and in the cloud. You can view all your SQL Servers, data marts, data lakes and any external data sources that you wish to virtualize.

In addition to supporting connections to many data sources, Azure Data Studio has a built-in Jupyter notebook experience. You can use notebooks to access, explore, manipulate, visualize, and model against the built-in Apache Spark experience for SQL Server Big Data Clusters, but what I am most excited about is how we also light up all these capabilities on top of the native SQL Server engine. Using the SQL kernel, you can create rich interactive experiences using T-SQL against any version or edition of SQL Server, on-premises on in the cloud. You can program in Python, SQL, Powershell, Scala, and R. It is such a powerful concept that we have started to convert our documentation, deployment experiences, and troubleshooting manuals into interactive notebooks. We will continue to invest in innovative ways to incorporate this new paradigm into our data experience.

Azure Data Studio comes with many other rich features that make it the most powerful single tool to work across relational databases, data virtualization, big data, and built-in virtualization. Through custom dashboards, intelligent visualizations, and a highly extensible platform, we can continue to bring innovative and modern data experiences into Azure Data Studio such as the SandDance visualizer seen below.

Azure Data Studio is on GitHub so that you can see the innovative progress being made in this evolving tool. To learn more about the capabilities of Azure Data Studio, read our documentation.

Intelligence over all of your data

Enterprises today have data scattered across a variety of platforms and data sources but need to access this data in a consistent manner. This has led to the need to build complex and expensive Extract, Transform, and Load (ETL) applications often moving the data into a central database platform like SQL Server.

SQL Server 2019 has solutions for these challenges by providing data virtualization. Applications and developers can use the familiar and consistent language of T-SQL through external tables to access structured and unstructured data from sources like Oracle, MongoDB, Azure SQL, Teradata, and HDFS. SQL Server 2019 eases the burden of establishing these data sources with built-in driver support. Polybase extends this functionality allowing you to access a host of other data sources using the ODBC driver of your choice. Because external tables act and look like tables you can join data across these sources with local SQL Server tables to provide a seamless experience. In addition, external tables provide some of the same capabilities as local SQL Server tables including using familiar security and object management techniques.

SQL Server 2019 extends the functionality of Polybase by providing a complete analytics platform powered by Big Data Clusters. In addition, a Big Data Cluster deployment includes a Data Pool which can be used to build a data mart of cached results from queries from external tables across or outside the cluster or data directly ingested from sources such as IoT data.

As mentioned earlier, Big Data Clusters come with Apache Spark built-in providing an end-to-end, secure machine learning platform using technologies such as SparkML and SQL Server Machine Learning Services. Data Scientists now have a complete system to train and prepare machine learning models using external table data sources in and outside the cluster. These models can then be deployed and consumed as applications using a RESTful Web Service compliant with Swagger applications.

“Building and deploying our vertical AI-solution for clinical radiology combines very diverse implementation paradigms, data formats and regulatory requirements. SQL Server 2019 big data clusters allowed us to accommodate and integrate all aspects from one shared platform – for our data scientists with their deep learning as well as for our software engineers who wire up workflows, security and scalability. At runtime, our healthcare customers benefit from simple containerized deployment and maintenance while being able to move our solution between on-prem and the cloud easily.”  Read more in our Balzano customer story.

René Balzano, Founder and CEO, Balzano

Industry leading intelligent performance

SQL Server provides one of the most powerful query engines proven with industry leading benchmarks. Today new TPC benchmarks have been announced continuing to prove SQL Server has unparalleled performance in the industry.

Developers and data professionals need more. They need a database engine that can adapt to their query workloads and reduce time for expensing performance tuning. Users expect to migrate to the latest release of SQL Server and gain performance without having to make major application changes. When they have to analyze query performance they need deep insights anytime and anywhere.

We recently performed a performance test on a SQL Server 2019 instance running on a Windows Server 2016 running on an 8-Socket (224 cores, 12TB RAM and 200TB+ SSD storage) Lenovo Server (ThinkSystem SR950) using Intel Cascade Lake processors. We generated data and loaded a single table (LINEITEM, TPCH database schema with 145TB+ raw data). This table had 54TB compressed data with 1 Trillion+ rows. Q1 query (as defined by TPCH) which scans the whole table and selects nearly all the rows for computation was run in both cold (all data read from storage) and warm (data in memory) scenario. SQL Server 2019 was able to provide unparalleled performance processing over a trillion rows with the query completing in under 2 mins (107 secs) in warm cache and under 4 mins (238 secs) with all data read from storage. For the warm cache, this translates to processing over 6 billion rows/sec and read throughput of around 50GB/sec. You can find the Azure Data Studio Notebooks that showcase both scenarios on GitHub.

SQL Server 2019 includes built-in query processing capabilities called Intelligent Query Processing. By updating your database compatibility level to 150 (the default level for SQL Server 2019), the query processor in the SQL Server engine can enhance performance through capabilities like batch-mode on row store, scalar UDF inlining,or table variable deferred compilation. It can automatically correct memory-related query execution issues through memory grant feedback. No query or application changes are required for a boost in performance.

SQL Server also provides optimized in-memory capabilities without changing your application through the support of persistent memory and optimized tempdb metadata access. Tempdb now just runs faster.

Because in some situations you need immediate access to query performance insights, SQL Server 2019 enables lightweight query profiling by default including the ability to access the last actual query execution plan. For deeper insights over time, enable the query store and enable historical query plan performance analysis including the ability to have SQL Server automatically correct query plan regressions.

A list of all the performance improvements in SQL Server 2019 is available.

Mission critical security and availability

A modern data platform must provide confidential computing through software that does not expose vulnerabilities and features to secure your data. SQL Server 2019 for the last 9 years has been the least vulnerable database product in the industry according to NIST (National Institute of Standards and Technology Comprehensive Vulnerability Database).

Built on the foundation of security capabilities like row-level security and dynamic data masking, SQL Server 2019 provides features that meet the security needs of modern applications through always encrypted with secure enclaves. Enclaves allow for data to be only available in an unencrypted state in the enclave memory space on the server. This provides control over the unencrypted data while enabling rich query computing and indexing. SQL Server 2019 also provides a new T-SQL interface to classify your data assisting you to meet compliance standards such as GDPR. Because classification is now built into the engine, SQL Server audit can be used to track users who access classified data.

Keeping your data available at all times is critical to any data application. Data professionals for years have struggled with managing transaction log growth and application availability due to long running or long open active transactions. SQL Server 2019 now provides accelerated database recovery to overcome these challenges with no application changes required. Using a database option, SQL Server will use a Persisted Version Store to track changes allowing for rollback and undo recovery to execute faster than it takes to observe changes. Transaction log truncation is no longer dependent on active transactions. Accelerated database recovery is based on the work done for Azure SQL Database (read the whitepaper on the topic recently presented at the VLDB 2019 conference) and is one of the key technologies behind the amazing performance of Hyperscale in Azure.

Platform and language of choice with compatibility

SQL Server rocked the industry by bringing SQL Server to Linux with SQL Server 2017. Using the innovative technology of the SQLPAL, SQL Server on Linux has application and database compatibility with Windows. Backup a database on Windows and restore it on Linux with no application changes. SQL Server 2019 continues the Linux journey by offering new capabilities such as Replication, Distributed Transactions, Polybase, and Machine Learning Services.

Because SQL Server has embraced Linux, the community has already seen the benefits of using SQL Server containers including simplified software patching, consistency, and easy integration into Continuous Integration/Continuous Deployment (CI/CD) pipelines. Customers now see the benefit of using SQL Server with containers in production. SQL Server 2019 embraces this need by offering containers based on Red Hat Enterprise Linux images and running by default as non-root. This enables SQL Server containers to be officially supported on the popular Kubernetes platform RedHat OpenShift.

Developers also need to use the language of their choice with a compatible data platform. SQL Server supports a variety of popular languages and providers such as C#, Java, node.js, PHP, Ruby, and Go. In addition, SQL Server 2019 also allows developers to extend the T-SQL language through Java classes with SQL Server Language Extensions using the extensibility framework, the same architecture that powers SQL Server Machine Learning Services.

The voice of the customer

Everything poured into SQL Server is based on customer experiences. Listening to our customers is the only way to provide solutions that meet real-world challenges. SQL Server 2019 addresses key customer feedback areas such as:

• Improved context for errors such as string truncation
• New diagnostics for supportability
• Query store enhancements
• Deep engine performance improvements.

Read about all the SQL Server 2019 improvements in our documentation.

The post SQL Server 2019 is now generally available appeared first on Microsoft SQL Server Blog.

With this release of SQL Server 2019 community technology preview 3.2, we are announcing the preview of Big Data Clusters for SQL Server 2019. Big Data Clusters for SQL Server enables big data analytics within SQL Server. It brings HDFS and Apache Spark™ into SQL Server for scale out compute and storage.

Big data clusters allow you to deploy scalable clusters of SQL Server, Apache Spark™, and HDFS running on Kubernetes. It provides all the tools and systems to ingest, store, and prepare data for analysis as well as to train and operationalize machine learning models. It allows you to query external data sources through data virtualization and combine and analyze your high-value relational data with high-volume big data. You will be also be able to build and deploy scalable and productive data-driven applications in big data clusters.

Our early adopter customers are already using Big Data Clusters for SQL Server 2019 for their production workloads. Check out what they have to say below:

Systems Imagination Inc.

“With SQL Server 2019 big data clusters, we can solve for on-demand big data experiments. We can analyze cancer research data coming from dozens of different data sources, mine interesting graph features, and carry out analysis at scale.” – Pieter Derdeyn, Knowledge Engineer, Systems Imagination Inc

Aginity

“We are excited to see Microsoft up the game in modern databases with the richest feature set we have seen to date. With SQL Server 2019 big data clusters’ management and performance architecture, our customers will have the ability to scale analytics across our portfolio of analytic management solutions, Aginity Pro Team, and Enterprise and build a trusted analytic layer to operationalize analytics in a highly consistent and efficient way both on-premises and in the cloud.”- Paul Schaut, CEO, Aginity

There are several scenarios in which big data clusters lets you interact with your big data. Please visit the SQL Server big data clusters documentation to learn more.

To download the preview bits for Big Data Clusters for SQL Server 2019, please review the documentation to learn more.

In addition to the preview of Big Data Clusters for SQL Server 2019, we are pleased to announce the integration of Azul System’s Zulu Embedded for all scenarios where Java is used in SQL Server – in PolyBase,  Apache Spark™, Java extensibility, and more. There is no additional cost beyond what you pay for SQL Server.

Customers are encouraged to deploy this free preview, try out features in this release, and provide feedback to the engineering team. Your feedback is very helpful in refining features to be the most useful.

Check out the What’s new in SQL Server 2019 preview documentation to learn more.

Ready to learn more?

• If you want to run SQL Server 2019 community technology preview 3.2 in production with full support from Microsoft support and you’re already enrolled in the Early Adoption Program (EAP), please contact your assigned program manager to get set up for production support.
• To get started with SQL Server 2019 community technology preview 3.2, you can find the download instructions on the SQL Server 2019 web page or use a SQL Server 2019 community technology preview virtual machine in Microsoft Azure.
• Check out what features were introduced in SQL Server community technology preview 3.1.
• Watch the recent webinar on Run SQL Server on Linux and Kubernetes.
• Watch this free on-demand webinar to discover how SQL Server 2019 big data clusters can help you gain business insight.

The post SQL Server 2019 community technology preview 3.2 is now available appeared first on Microsoft SQL Server Blog.

Now, for the first time ever, SQL Server 2019 creates a unified data platform with Apache Spark^TM and Hadoop Distributed File System (HDFS) packaged together with SQL Server as a single, integrated solution. Through the ability to create big data clusters, SQL Server 2019 delivers an incredible expansion of database management capabilities, further redefining SQL Server beyond a traditional relational database. And as with every release, SQL Server 2019 continues to push the boundaries of security, availability, and performance for every workload with Intelligent Query Processing, data compliance tools and support for persistent memory. With SQL Server 2019, you can take on any data project, from traditional SQL Server workloads like OLTP, Data Warehousing and BI, to AI and advanced analytics over big data.

SQL Server provides a true hybrid platform, with a consistent SQL Server surface area from your data center to public cloud—making it easy to run in the location of your choice. Because SQL Server 2019 big data clusters are deployed as containers on Kubernetes with a built-in management service, customers can get a consistent management and deployment experience on a variety of supported platforms on-premises and in the cloud: OpenShift or Kubernetes on premises, Azure Kubernetes Service (AKS), Azure Stack (on AKS) and OpenShift on Azure. With Azure Hybrid Benefit license portability, you can choose to run SQL Server workloads on-premises or in Azure, at a fraction of the cost of any other cloud provider.

SQL Server – Insights over all your data

SQL Server continues to embrace open source, from SQL Server 2017 support for Linux and containers to SQL Server 2019 now embracing Spark and HDFS to bring you a unified data platform. With SQL Server 2019, all the components needed to perform analytics over your data are built into a managed cluster, which is easy to deploy and it can scale as per your business needs. HDFS, Spark, Knox, Ranger, Livy, all come packaged together with SQL Server and are quickly and easily deployed as Linux containers on Kubernetes. SQL Server simplifies the management of all your enterprise data by removing any barriers that currently exist between structured and unstructured data.

Here’s how we make it easy for you to break down barriers to realized insights across all your data, providing one view of your data across the organization:

• Simplify big data analytics for SQL Server users. SQL Server 2019 makes it easier to manage big data environments. It comes with everything you need to create a data lake, including HDFS and Spark provided by Microsoft and analytics tools, all deeply integrated with SQL Server and fully supported by Microsoft. Now, you can run apps, analytics, and AI over structured and unstructured data – using familiar T-SQL queries or people familiar with Spark can use Python, R, Scala, or Java to run Spark jobs for data preparation or analytics – all in the same, integrated cluster.
• Give developers, data analysts, and data engineers a single source for all your data – structured and unstructured – using their favorite tools. With SQL Server 2019, data scientists can easily analyze data in SQL Server and HDFS through Spark jobs. Analysts can run advanced analytics over big data using SQL Server Machine Learning Services: train over large datasets in Hadoop and operationalize in SQL Server. Data scientists can use a brand new notebook experience running on the Jupyter notebooks engine in a new extension of Azure Data Studio to interactively perform advanced analysis of data and easily share the analysis with their colleagues.
• Break down data silos and deliver one view across all of your data using data virtualization. Starting in SQL Server 2016, PolyBase has enabled you to run a T-SQL query inside SQL Server to pull data from your data lake and return it in a structured format—all without moving or copying the data. Now in SQL Server 2019, we’re expanding that concept of data virtualization to additional data sources, including Oracle, Teradata, MongoDB, PostgreSQL, and others. Using the new PolyBase, you can break down data silos and easily combine data from many sources using virtualization to avoid the time, effort, security risks and duplicate data created by data movement and replication. New elastically scalable “data pools” and “compute pools” make querying virtualized data lighting fast by caching data and distributing query execution across many instances of SQL Server.

“From its inception, the Sloan Digital Sky Survey database has run on SQL Server, and SQL Server also stores object catalogs from large cosmological simulations. We are delighted with the promise of SQL Server 2019 big data clusters, which will allow us to enhance our databases to include all our big data sets. The distributed nature of SQL Server 2019 allows us to expand our efforts to new types of simulations and to the next generation of astronomical surveys with datasets up to 10PB or more, well beyond the limits of our current database solutions.”- Dr. Gerard Lemson, Institute for Data Intensive Engineering and Science, Johns Hopkins University.

Enhanced performance, security, and availability

The SQL Server 2019 relational engine will deliver new and enhanced features in the areas of mission-critical performance, security and compliance, and database availability, as well as additional features for developers, SQL Server on Linux and containers, and general engine enhancements.

Industry-leading performance – The Intelligent Database

• The Intelligent Query Processing family of features builds on hands-free performance tuning features of Adaptive Query Processing in SQL Server 2017 including Row mode memory grant feedback, approximate COUNT DISTINCT, Batch mode on rowstore, and table variable deferred compilation.
• Persistent memory support is improved in this release with a new, optimized I/O path available for interacting with persistent memory storage.
• The Lightweight query profiling infrastructure is now enabled by default to provide per query operator statistics anytime and anywhere you need it.

Advanced security – Confidential Computing

• Always Encrypted with secure enclaves extends the client-side encryption technology introduced in SQL Server 2016. Secure enclaves protect sensitive data in a hardware or software-created enclave inside the database, securing it from malware and privileged users while enabling advanced operations on encrypted data.
• SQL Data Discovery and Classification is now built into the SQL Server engine with new metadata and auditing support to help with GDPR and other compliance needs.
• Certification Management is now easier using SQL Server Configuration Manager.

Mission-critical availability – High uptime

• Always On Availability Groups have been enhanced to include automatic redirection of connections to the primary based on read/write intent.
• High availability configurations for SQL Server running in containers can be enabled with Always On Availability Groups using Kubernetes.
• Resumable online indexes now support create operations and include database scoped defaults.

Developer experience

• Enhancements to SQL Graph include match support with T-SQL MERGE and edge constraints.
• New UTF-8 support gives customers the ability to reduce SQL Server’s storage footprint for character data.
• The new Java language extension will allow you to call a pre-compiled Java program and securely execute Java code on the same server with SQL Server. This reduces the need to move data and improves application performance by bringing your workloads closer to your data.
• Machine Learning Services has several enhancements including Windows Failover cluster support, partitioned models, and support for SQL Server on Linux.

Platform of choice

• Additional capabilities for SQL Server on Linux include distributed transactions, replication, Polybase, Machine Learning Services, memory notifications, and OpenLDAP support.
• Containers have new enhancements including use of the new Microsoft Container Registry with support for RedHat Enterprise Linux images and Always On Availability Groups for Kubernetes.
  You can read more about what’s new in SQL Server 2019 in our documentation.

SQL Server 2019 support in Azure Data Studio

Expanded support for more data workloads in SQL Server requires expanded tooling. As Microsoft has worked with users of its data platform we have seen the coming together of previously disparate personas: database administrators, data scientists, data developers, data analysts, and new roles still being defined. These users increasingly want to use the same tools to work together, seamlessly, across on-premises and cloud, using relational and unstructured data, working with OLTP, ETL, analytics, and streaming workloads.

Azure Data Studio offers a modern editor experience with lightning fast IntelliSense, code snippets, source control integration, and an integrated terminal. It is engineered with the data platform user in mind, with built-in charting of query result sets, an integrated notebook, and customizable dashboards. Azure Data Studio currently offers built-in support for SQL Server on-premises and Azure SQL Database, along with preview support for Azure SQL Managed Instance and Azure SQL Data Warehouse.

Azure Data Studio is today shipping a new SQL Server 2019 Preview Extension to add support for select SQL Server 2019 features. The extension offers connectivity and tooling for SQL Server big data clusters, including a preview of the first ever notebook experience in the SQL Server toolset, and a new PolyBase Create External Table wizard that makes accessing data from remote SQL Server and Oracle instances easy and fast.

Getting started

Find additional resources and get started today by visiting the links below:

• Preview SQL Server 2019 for Windows, Linux, or Docker.
• Sign up for the Early Adoption Program to get advice and support for your project from SQL Server engineering, or to try SQL Server 2019 big data clusters.
• Download Azure Data Studio to get started with new SQL Server big data features like data virtualizations and notebooks.

The post SQL Server 2019 preview combines SQL Server and Apache Spark to create a unified data platform appeared first on Microsoft SQL Server Blog.