Organisations that can connect and use their data are more resilient and adaptable, driving sustainable growth. But how? We’ve identified three ways your organisation can leverage data insights to turn into action.

The hybrid world of work

We all have a lived experience of hybrid working, and it’s here to stay. In our latest Work Trend Index, we found that:

_{53% of people are likely to consider transitioning to hybrid in the year ahead if they haven’t already}

_{53% of employees are more likely to prioritise health and wellbeing over work.}

This means organisations need a new digital fabric for collaboration that brings together both digital and physical spaces. One that connects people and empowers them to balance their career and their wellbeing.  Organisations won’t be able to scale to this transition without a strong understanding of data.

Unilever provides their people – including individuals, managers, and leaders – with data-driven, privacy-protected visibility with Microsoft Viva. These data insights help Unilever improve the employee experience and promote greater work-life balance.

The hyper-connected business

We need that next level of real-time hyper-connectivity between businesses, and between consumers and businesses, where data and intelligence flow freely to tackle the challenges of supply and demand.​

According to our research, 80 percent of companies suffer with significant data silos. This prevents them from gaining meaningful insight to make business decisions. But by ensuring your data strategy combines the right capabilities and the right culture, you can identify opportunities, better serve customers, transform your products, empower employees, drive sustainable results and optimise operations. 

Access and unify your data

The more siloed your data, the harder it is to accomplish data governance. When you harness the streams of data being created on a secure platform, enabling better decision making and transformative processes.

Analyse, predict and orchestrate

Once you have unified your data you can leverage AI and Machine Learning. Run big data analytics to predict customer intent to purchase and identify segments that are at risk of churning. This can help identify new, or even protect revenue streams, improve operational efficiencies, create sustainable supply chains and drive a better overall quality of service.

Activate and measure

Take these insights and democratise access through your organisation. The people who will best put the data to use are the ones who deal with it day-to-day. By allowing both front and back-end employees access to that data, they can create low/no code apps that streamline operations and deliver better customer experiences.

Heineken gives their frontline employees customer data insights directly on a unified platform with Azure Synapses and Dynamics 365. This enables their sellers to gain much richer insights about their customer’s preferences to deliver the best possible purchase recommendations and provide a much more tailored buying experience.

Omnichannel customer experience

Technology has shaped both the online and offline experience for customers. And the more data silos organisations have, the more frustrated the customer becomes.

A hyper-connected business can link all the customer touchpoints together to create a 360-degree view. Employees can access this, meaning they can provide the best experience to the customer, no matter the point of they journey they are in, or how they’re getting in touch. AI and Machine Learning can then help drive richer data insights that can be used to delight and build trust.

Alpha XR Boots Alliance balances data and privacy to deliver more engaging and personalised experiences, to their patients and customers. By dramatically enhancing their customer personalisation, they can deliver the best tailored offers and content to the right customer, in the right context, at the right time and through the right channels across the entire journey.

Build sustainable growth with data insights

The events over the past several years have shown us that organisations that are able to connect and use their data are more robust and able to adapt to changing environments, harness potential and drive competitive advantage.​

Through empowering employees with the right culture, unifying and optimising your data, and building the omnichannel customer experience, you can turn data insights into action.

Find out more

Imagine business powered by data

Put your most important asset to work

The post How to turn data insights into action appeared first on Microsoft Industry Blogs - United Kingdom.

I believe that when you harness the streams of data being created, tie this to your existing data and apply AI to it, you enable better decision making and transformative processes. You will:

• Engage customers in new, meaningful ways​.
• Empower employees with the real-time insights they need to make the right decisions and be agile​.
• Optimise operations by better anticipating customer demands and supply chain disruptions​.
• Transform your products based on the signals from end users or the products themselves​.
• Meet and exceed sustainability commitments to have a better impact on the world around us.

You can see success in the way Marks and Spencer use their data. They own every part of their supply chain and collect data from across all touchpoints. They needed an agile solution that would scale, store and analyse their massive amounts of data. By replacing their on-premise data warehouse with Microsoft Azure-based data platform built around Azure Synapses Analytics, they gave teams access to valuable data they didn’t have before.

“By democratising access, we’re allowing more people to have ideas, and these ideas add incremental value to the business. Our retail support team is already reshaping how we report stock and stock loss to managers. Since they work closest to the retail side of the business, they know exactly how to consolidate and present information in ways that lead to real improvements, ” says Aaronpal Dhanda, Head of Data Technology.

The reality of data

As you can see, you can bring so much value to your organisation when you connect and manage your data properly. However, true modernisation comes by innovating the processes that run your business. This is where data realities begin to hit both IT and business leaders.

Most leaders agree that provisioning an end-to-end analytics platform is not a simple task. They need to make sure they can trust their data, that they can get deeper insights from it and it is bias-free. And, they need to ensure compliance along every step of the analytics journey.

All while trying to create agility and more rapid decision making by democratising data and tools to everyone and providing the digital upskilling they need to do this effectively.

The analytics paradox

More and more tools, systems and things become connected. As a result, companies must figure out how to manage and analyse new classes of data. Additionally, different lines of businesses see different values in data and analytics.

And this is what creates the paradox of analytics. Although analytics systems are intended to be centralised to provide “the single version of truth”, the more we apply new technology to integrate and analyse data in different ways, the more silos we can recreate.

Sometimes, through hard work to dissolve operational data silos we end up creating more. And not just data silos, but siloed teams and people. When we implement new technology for specific types of data, we are creating a more fractured approach to data integration. Then, this must be put back into the overall platform during the analytics lifestyle.

This approach is often sold as a centralised solution. But it is siloed architectures that creates siloed data teams. As a result, data governance becomes extremely difficult to accomplish. All counter to our objective to enable deeper analytics collaboration between teams.

Bring your vision to life with analytics

Every organisation has experts in both data and analytics technology. When they collaborate over data with the same efficiency they do for productivity applications, their expertise is utilised across team and organisational boundaries.​

Bristol City Council serves more than 400,000 people, with social care playing a huge role in their services. The children services team wanted to see a holistic view of the citizens they work with. This meant they needed to create a secure common data platform. By using Azure Data Lakes and Azure Data Factory, they unified that data to create a single view of each child across the disparate systems.

“The Microsoft Azure solution has been revolutionary for Bristol City Council,“ says Simon Oliver, Director of Digital Transformation. “We are now able to see the outcomes of the decisions we give. And by being able to do that at scale we’re able to make decisions based on what will happen.”

By connecting, managing and governing your data assets in a cohesive fashion, you have the foundation for successful digital modernisation. A simplified framework allows all of your data and all types of data to exist on the same platform and be governed in the same way.​

Azure Synapse and Azure Purview provide a single cloud service with a single interface for development, management, monitoring, security and governance of your data. You can link to your Dataverse to run advanced analytics on data from Dynamics 365 and Power Platform simply. This allows you to:​

• Be agile to the needs of the business or react to unpredictable ​external changes.
• Enable quicker insights for decision making​.
• Impact business models and overall value chains​.
• Get more granular, deeper insights.

And, most importantly, it is done in a secure, compliant and governed way.

Drive analytics value

Free your people to focus on real business value by managing the complexity of architecture on a single unified platform. This gives every person and every team in your organisation powerful analytics and insight. So, you can go from trying to manage the analytics paradox to driving real change and value to your organisation.

Find out more

Building a data-driven organisation 

About the author

As an advocate of data-driven decisions, Robin has spent over two decades at Microsoft ensuring organisations have the tools to leverage the zettabytes of data available today to achieve their digital transformation vision.

Microsoft has been on its own digital transformation journey for several years and data has been a central part of that journey. Robin focuses on creating a data-driven culture across the business at Microsoft. This includes ensuring that we are considering data across our internal processes, as well as how we are helping our customers and partners succeed with data.

Robin is passionate about learning and collaborating with our customers and partners about how to truly leverage data and AI to create new solutions.

Prior to working at Microsoft, she served in the US Military. She strives to bring her best in all aspects of work and personal life. From obtaining two law degrees and multiple professional certifications – all while working full time, parenting her daughters and balancing personal commitments (including training for an IronMan), she believes anything is possible.

The post Empower employees by unifying your analytics and data architecture appeared first on Microsoft Industry Blogs - United Kingdom.

Let’s dissect this a little. Firstly, what is culture? It’s quite an ethereal term and one I have often struggled with. Someone once shared a simple definition that resonated with me: Culture is “what you do when your boss isn’t watching”. Culture is something ingrained into how you work and think, which is important. You can’t simply say you have a culture as an organisation. You must live and breathe the culture.

And what do we mean when we say data-driven? It’s not about collecting all data. In fact, lack of data isn’t a problem for most organisations! However, what they often struggle with is extracting value from that data. Therefore, what we are really talking about is decisions that are driven from data. Because we use the data to inform and justify our decisions, it needs to be good quality.

So, a data-driven culture is one where the organisational norm is that decision making is driven by data. How can leaders successfully build this culture? If we look at the journey to a data-driven culture, I think of four steps.

1. Create the right mindset for a data-driven culture

To me this is the most crucial step – leadership must be clear. I don’t just mean that leaders need to talk about using data. Leaders need to demonstrate how they place data at the heart of what the organisation is trying to achieve every day.

In order to thrive, leaders must be clear about what their organisation’s purpose and outcomes are. A great way to create accountability and direction is to tie those purposes and outcomes to measures of success.

At Microsoft, we use an approach called Objectives and Key Results (OKRs) to organise and align our activities to transform. The focus on key results inspires a data-driven mindset across the organisation. It also provides a common data driven focus and language for everyone in the business – we all start to think about the measures that matter.

2. Find organisational and individual value in a data-driven culture

When looking at driving change I have to say that unfortunately we, as humans, can be a selfish bunch. Often, one of the biggest drivers of successful change is understanding what is in it for the individual. Within Microsoft we apply the PROSCI change methodology. At the heart of this is the ADKAR change model. There is the adage: organisations don’t change, people do. ADKAR is an acronym for five elements of change for individuals:

• Awareness of the need to change.
• Desire to participate and support the change.
• Knowledge on how to change.
• Ability to implement desired skills and behaviours.
• Reinforcement to sustain the change.

To embed the change within our people and therefore to drive change in the organisation, we really need to create the desire to change. If people are told the future is a data-driven culture they simply won’t buy into it and commit to it. Therefore, demonstrating change and demonstrating value from data fast is important. When people see that the change works and is more effective, they’ll want to change.

3. Build your and your employee’s skills

If we are working on changing our mindsets, we also need to prepare our people with the right skills and tools. Everyone needs basic data literacy skills and we all different levels of knowledge. Some people have inherent data literacy skills. Others may need support to be able to understand and assimilate data then interpret and analyse it. Then, once we have the basics in place, we need to progress to understand how we can use the tools at our disposal to answer the business question we have. However, we can’t just throw tools like Excel, Tableau and PowerBI at our people and expect them to be able to optimise and transform our organisations.

Leaders need to help their employees on their learning journey by democratising data access, building learning opportunities and give employees the time to take those opportunities. One way you can do this is to build re- and upskilling into employee KPIs. In our data journey we move from a data consumer to data analyst, citizen data scientist and beyond. Not everyone starts in the same place. Everyone’s learning path is different and the KPIs need to reflect that.

Microsoft provides access to great learning tools to support you and your employee’s individual journey. These include Microsoft Learn – the front door to all your training needs whether you are just starting out or an experienced professional, with role-based learning paths. You can also explore how to use AI in your organisation with the Microsoft AI Business School.

4. Empower employees with the right tools

So, now you’ve changed your mindset and the mindset of your organisation. You’ve seen the value of a data-driven organisation and are building relevant skills. But what tools do leaders need to get insights?

Firstly, organisations need quality, curated data that is easily accessible. Not everyone in the business is a data engineer who can find, cleanse and prepare data for analytics. You need an easy way for everyone in an organisation to find the business data that they need. It also needs to be presented in a manner that is easily understandable – using the language they understand. This is where a data marketplace or data catalogue is invaluable. At Microsoft we have Azure Purview, our unified data governance platform. This is a platform that automatically discovers data wherever it lives in your organisation. It can classify data and identify data lineage; but importantly it also presents a data catalogue of your data using business language. The data catalogue is a core element of a successful self-service strategy.

Using self-service data insights tools like PowerBI provides easy access to pre-prepared and certified datasets. This enables your people to be confident in the quality of the data source and empowers them to discover new insights from the data. It also allows the data owners can enable controls to ensure colleagues can only see the data they need to.

A continuous journey to a data-driven culture

These four steps will help you build a data-driven culture. I also want to remind you of the final step in ADKAR: Reinforcement! It’s critical that this is not seen as a one-off initiative. You need to work hard at reinforcing the change to build a successful data-driven culture. If people don’t use these new skills, mindset and tools, it is the case of use it or lose it. This can be tough – but creating a champion network focused on data is a wonderful way to organically drive and embed the culture.

Find out more

Build a data-driven organisation

Peer to peer interview: Unite your data strategy and culture

Create a data culture

About the author

James is a Digital Advisor in Microsoft Consulting Services. He is focussed on helping customers realise their business outcomes and purpose by enabling their digital transformation with advanced cloud technologies – with a particular focus on data, AI, automation and sustainability. Prior to joining Microsoft in 2014, James held several roles across financial services (HSBC, Schroders), public sector (Scottish Water) and consulting (PwC).

The post The role of leadership in a successful data-driven culture appeared first on Microsoft Industry Blogs - United Kingdom.

To build and maintain trust with citizens we – as a data community – have an obligation to address these ethical issues. Previously, I’ve talked about how to build and effective data strategy and culture. A critical aspect of both strategy and culture is to ensure the ethical and responsible use of AI and data. We need to empower organisations to use data with a sense of responsibility. The EU recently released their Artificial Intelligence Act, the first legal framework for AI. In it, they take a risk-based approach to protect EU citizen’s rights while ensuring they can still foster innovation. As we saw with GDPR, the AI Act includes fines for infringements of up to four percent of global annual turnover (or €20M, if greater). Therefore, it is more important than ever to focus on the responsible use of data and AI.

Build your responsible AI strategy with the right question

Are you using AI technology to do the right things? Is it answering the right problems in the right way? AI shouldn’t be implemented because it’s a shiny new piece of technology. It should be used to help solve a problem. And to work properly, it needs to reflect the community you serve. To do this you need to build your data and AI solutions on ethical principles that put people first.

At Microsoft, one of my focusses as Chief Data Officer (CDO) is to ensure our use of data and AI remains ethical and responsible. What I have found is this is just as much a culture shift as much as a technological process. In a recent webinar, when I spoke with other data leaders across the industry, they also agreed.

What was clear across the board is that organisations need to take a very practical approach to responsible data and AI principles. Below are six principles that organisations can use to build their own responsible AI governance.

1.      Fairness

Although our society is diverse, it is unfortunately unfair and bias. It is our role to ensure that the systems we develop and deploy reduce this unfairness. However, fairness doesn’t just relate to the technical components of the system. It also about the societal context in which it is used.

“Ensuring the biases are taken care of is important. We think about how data is being increasingly used across platforms and avoiding any disproportional impact as a result,” says Sudip Trivedi, Head of Data and Analytics at London Borough of Camden.

How can leaders ensure fairness? We need diverse teams that question the data and models we are using at every step along the journey. We need to think critically about the implications and unintended consequences more broadly. Having checklists to continually monitor data and AI processes is a great way to ensure we stay fair. Leverage tools and learnings to validate fairness regularly.

Fairness tools:

AI fairness checklist

Datasheet fairness checklist

Fairlearn open-source toolkit

2.      Inclusiveness

Our aim at Microsoft is to empower everyone to achieve more. We are intentionally inclusive and intentionally diverse in the paths we take. AI needs to be built with everyone in mind. Because when you design solutions that everyone can access, the data you collect will be fairer.

This is where your diverse organisation becomes a huge benefit to you. By ensuring that your data and AI teams are diverse you will be building for everyone. And don’t forget to include a diverse audience for your testing to ensure that your systems remain accessible for all.

“It takes having that diversity within your organisation or stakeholder group to spot issues,” says Nina Monckton, Head of Data Strategy, Advancing Analytics & Data Science at AXA Health.

Inclusive tools:

Inclusive design guidelines

Design with accessibility in mind

3.      Reliable and safe

Our data and AI processes need to be consistent with our values and principles. As owners of these models, we need to continuously check that they’re not causing harm to society. And if they are, we need to have processes to fix them. We’re also transparent with our users on these issues.

Building reliable and safe AI isn’t limited to just physical systems that affect human life. For example, self-driving cars or AI in healthcare. It’s also about ensuring that every model you create stays reliable and safe no matter how big it gets or how many people work on it.

Reliable and safe tools:

Accelerate the pace of machine learning while meeting governance and control objectives with MLOps

Preserve privacy with Project Laplace

4.      Transparency

Transparency can help us reduce unfairness in AI systems; it can help developers debug systems, and it helps us build trust with our customers.

Those who are creating the AI systems should be transparent about how and why they’re using AI. They should be open about the limitations of their systems. People should also be able to understand the behaviour of AI systems.

“Being transparent is critical to doing good data work. If you don’t have the transparency, it’s very difficult to know if it’s doing its job well,” says Daniel Gilbert, Director of Data at News UK.

To truly understand AI, we need to democratise through digital skilling. This is not just within your organisation, but within society too. We need to work together to help encourage skills growth across our communities with digital skilling programmes. This will help further increase diversity in our organisations as we introduce people to the opportunities of technology careers.

“A lot of the data we are collecting and using are from people who are digital literate. There’s a real hard question: Is the data we’re collecting really representative of the people we’re trying to provide services for?” says Nina.

Transparency tools:

Microsoft Learn

Improve digital skills

Bridging the digital divide

5.      Privacy and security

Privacy is a fundamental right, and it must be built in to all our systems and products. With AI, machine learning and the reliance on data, we add new complexities to those systems. This adds new requirements to keep systems secure and to ensure data is governed and protected.

You must think about where and how the data is coming from. Is it coming from a user or a public source? How can your organisation prevent corruption and keep the data secure?

Privacy and security tools:

Learn about confidential computing 

6.      Accountability

As leaders, we are accountable for how our systems impact the world. Let’s look at facial recognition. There’s a lot of good uses for it, but only if we stick to principles that guide on how we develop, sell, and advocate for regulation on facial recognition.

Accountability includes internal and external factors. We need to keep key stakeholders informed across the whole cycle of AI systems. And we need to ensure we stay accountable to society.

Mahesh Bharadhwaj, Head of Europe Analytics at Funding Circle talks about asking the right questions at the right time: “Are we using the AI to do the right things? Do we check the models are being built correctly? Are we making sure the model is being deployed on the context it is built?”

Accountability tools:

Explore interaction guidelines 

Responsible AI builds trust

To build trust, a balance between culture and data capabilities is key. We need to make sure we are encouraging people to leverage data in ethical and responsible ways. These six principles should help you build AI-systems while building a diverse and inclusive culture. By doing this, we will ensure we’re serving our community in the best way possible.

Find out more

Discover our approach to responsible and ethical AI

Build a modern data strategy

Resources to empower your development team

Register for Microsoft Build on 25-27 May 

About the author

As an advocate of data-driven decisions, Robin has spent over two decades at Microsoft ensuring organisations have the tools to leverage the zettabytes of data available today to achieve their digital transformation vision.

Microsoft has been on its own digital transformation journey for several years and data has been a central part of that journey. Robin focuses on creating a data-driven culture across the business at Microsoft. This includes ensuring that we are considering data across our internal processes, as well as how we are helping our customers and partners succeed with data.

Robin is passionate about learning and collaborating with our customers and partners about how to truly leverage data and AI to create new solutions.

Prior to working at Microsoft, she served in the US Military. She strives to bring her best in all aspects of work and personal life. From obtaining two law degrees and multiple professional certifications – all while working full time, parenting her daughters and balancing personal commitments (including training for an IronMan), she believes anything is possible.

The post 6 ways leaders can build responsible AI and data systems and the tools that can help appeared first on Microsoft Industry Blogs - United Kingdom.

Let’s start with the obvious: data is nothing without a context. It is inert. Only when it is analysed in a specific framework does it become information. This is then processed and internalised by individuals to create knowledge.

In the digital world, this relationship doesn’t change. It’s up to your organisation to transform this data into knowledge, then action. That’s how you get a positive return on investment. Make your data relevant and useful by starting with these steps to create organisational change.

1.      Set your goals

To harness the full value of data, you need to ensure it is captured accurately, with the end goal in mind. So, the first step should be setting your goals.

Your data strategy will depend on the business problem you’re trying to solve and the environment you work in. This will lead to better understanding the factors and variables of your analysis. Executive buy-in will be the key to success. Ensure you on-board your organisation’s leaders early so you have the right engagement from all stakeholders along the process.

Nationwide’s business savings team’s initial goals were to gain a holistic view of the customer. With Microsoft Dynamics 365, they can manage the whole lifecycle of a customer account while streamlining account services.

“We’re spending less time on admin tasks which gives us the capacity to spend more time on value added activities which our clients appreciate. We now have a wealth of information at our fingertips.”

Anthony Pooley, Customer Relationship Manager, Business Savings at Nationwide Building Society

2.      Plan, plan, plan

As you grow your ambition of leveraging insights to benefit the business, you need to ensure you have an effective data strategy and management plan. This includes how the data will be stored and managed. You need to ensure accessibility without sacrificing security, all while aligning to current policies and regulations.

Also, you need to define a data governance maturity model, including data classification and compliance policies. You can get a head start on this in our data governance guide. Ensure you connect with all business stakeholders and empower them to define the strategy.

Having a maturity model will help ask the right questions and make sure you use the best practice standards and capabilities. This approach is fundamental, and it’s at the core of our Microsoft 365 cloud strategy.

3.      Reduce data silos

Data can be found in different parts of your organisation – marketing, sales, operations and more. The only way to deliver true value is to connect your data together. A solution like Microsoft Dynamics 365 can give you a 360-degree view of information, creating a single source of truth. What does this mean for your data insights? Real-time and predictive results that help deliver better services and products while making it easier for employees to access the information they need.

Electrical ecommerce site AO.com chose Microsoft Dynamics 365 for its scalability and stability. They also benefit from a single source of data and can extract information to deliver better business and customer value.

“What we get with Microsoft Dynamics 365 is a single view of inventory, a single view of customers. Through some of the advanced analytics and Power BI… [we gain] … the ability to make better and faster actionable decisions about how to trade.”

Carl Phillips: Director of Technology, AO.com

4.      Keep security at the heart

As mentioned, you need to ensure your data is kept safe when stored, managed and in transit. With intelligent security, you can empower your organisation against cyberthreats with machine learning and automation. A 2019 Forrester study commissioned by Microsoft, found our security technology automatically remediates 97 percent of endpoint attacks detected.

With built-in security across platforms, including Microsoft Dynamics 365, you can ensure you’re protected and compliant while employees can focus on high-value tasks, securely accessing the information they need, no matter where they are.

5.      Democratise data skills

You must ensure your whole organisation has the skills to understand data. To take advantage of new information, insights and resources, everyone needs to understand how to use data effectively. This will help build an accessible knowledge network, plus improve services and customer experiences.

It’s not an easy task. But building a data skills learning plan is vital to get the best out of data and to empower employees. The Forrester report on Why Digital Literacy Matters goes in-depth on how to spread data literacy. Dynamics 365 also makes it easy by giving employees a single source of truth. Microsoft Viva Topics uses AI across Microsoft 365 to recognise content, extract important information and automatically organises it into shared topics like projects, processes and customers. It then creates a knowledge network based on relationships, making it easy for employees to find what they need, when they need it.

6.      Build a feedback loop

To ensure your data insights stay relevant and agile, implement a change control process consistent across the organisation. Keep verifying your data strategy against your business plans, goals and with your executive sponsors to ensure you’re heading in the right direction. Be ready to adjust. Flexibility and speed of change will determine the difference between frustration and success.

Weave data throughout your organisation

Ultimately the power of data is directly correlated to your strategy and your people. Data doesn’t belong in a different realm. It doesn’t belong to one team or silo. On the contrary, it needs to intertwine with business processes, stakeholders and policies. It needs to reach across business silos and connect together to surface the right information, at the right time. Make sure you embark on the journey as an organisation. The response from this empowerment will surprise you.

Find out more

Learn how to harness the power of data

Find out how the power of data can benefit everyone

Build an effective data strategy

Use data and analytics to innovate

Resources to empower your development team

How to implement a data streaming solution with Azure Streaming Analytics 

About the author

Passionate about marketing and digital business strategy Krizia joined Microsoft to help customers achieving their business goals while getting the most out of their IT investments. She is focussed on raising the awareness around cloud solutions and security best practices. Yoga lover, book worm and travel enthusiast, so far Krizia’s work experience has taught her the importance of teamwork, exemplary leadership and communications skills.

The post Unleash the power of data with knowledge and insights appeared first on Microsoft Industry Blogs - United Kingdom.

Take a principled approach to your data strategy

With data governance in mind, the following diagram shows the essential stages for data lifecycle, please consult the attached document on “A Guide to Data Governance”. An effective data strategy must have provisions for data governance, but they are mutually inclusive, but not the same thing. 

For the purpose of this blog, let’s take a closer look at the considerations to take when building design principles for the four layers that are key in your data strategy, focused towards delivering business outcome and value from data. 

1. Data ingestion

A key consideration for data ingestion is the ability to build a data pipeline extremely fast, from requirements to production, in a secure and compliant manner. Elements such as metadata driven, self-service, low-code technologies to hydrating your data lake are key. When building a pipeline, consider design, the ability to do data wrangling, scale compute and also data distribution capabilities. In addition, having the right DevOps support for the continuous integration and delivery of your pipeline is critical. Tools such as Azure Data Factory support a plethora of on-premises, Software as a Service and data sources from other public clouds. Having such agility from the get-go is always helpful.

2. Storage

Data needs to be tagged and organised in physical and logical layers. Data lakes are key in all modern data analytics architectures. The Hitchhikers guide to the data lake is an excellent resource to understand different considerations that companies need to make. Organisations need to apply appropriate data privacy, security and compliance requirements based on data classification and the industry compliance requirements they operate in. The other key considerations are cataloguing and self-service to aid organisation level data democratisation to fuel innovation, guided by appreciate access control.

Choosing the right storage based on the workload is key, but even if you don’t get it right the first time, the cloud provides you options to failover quickly and provide options to restart the journey reasonably easily. You can choose the right database based on your application requirements. When choosing an analytics platform, make sure you consider the ability to process batch and streaming data.

3. Data processing

Your data processing needs will vary according to workload, for example most big data processing has elements of both real-time and batch processing. Most enterprises also have elements of time series processing requirements and the need to process free-form text for enterprise search capabilities.
The most popular organisational processing requirements come from Online Transaction Processing (OLTP). Certain workloads need specialised processing such as High Performance Computing (HPC), also called ‘Big Compute’. These use many CPU or GPU-based computers to solve complex mathematical tasks.

For certain specialised workloads, customers can secure execution environments such as Azure confidential computing which helps users secure data while it’s ‘in use’ on public cloud platforms (a state required for efficient processing). The data is protected inside a Trusted Execution Environment (TEE), also known as an enclave. This protects the code and data against viewing and modification from outside of the TEE.  This can provide the ability to train AI models using data sources from different organisations without sacrificing data confidentiality.

4. Analytics

Extract, transform, load construct or otherwise called ETL (or ELT depending on where the transform happens) relates to online analytical processing (OLAP), and data warehousing needs. One of the useful capabilities here is the ability to automatically detect schema drift. Consider end-to-end architectures like automated enterprise BI with Azure Synapse Analytics and Azure Data Factory. To support advanced analytics including Machine Learning and AI capabilities, it is key to consider the reusability of platform technologies already in use for other processing requirements of similar nature. Here is a quick start guide with working example for end-to-end processing.

Batch processing on Databricks, R, Python or for deep learning models are common examples. Compute, storage, networking, orchestration and DevOps/MLOps are key considerations here. For super large models, look at distributed training of deep learning models on Azure or the Turning Project. You also need to consider the ability to deal with data and model drift too.

Azure Enterprise Ready Analytics Architecture helps collate all the four layers together with people, process, security, and compliance. We also suggest using the recommended architectures from Azure Landing Zones to get started. It uses the Microsoft Cloud Adoption Framework and culminates our experience working through thousands of large scale enterprise deployments.

Now that we have covered the four stages, the following representation shows the key capabilities needed on top of your data platform to provide end to end data governance capability.  

Here are some resources to help you build a data strategy that accounts for governance and also allows for seamless innovation. 

• A proactive approach to privacy includes capabilities such as data catalog and stewardship, data lineage,  
• Map business glossary and industry models to fuel data pipelines,  
• A common data model (CDM) is often the key to success.  
• Industry accelerators contribute to agility in an organisation’s ability to deliver transformations and sustain innovations. 

After making all the capability provisions, and taking a principled architectural view as discussed in this section, you will most likely end up with the building blocks required for your cloud strategy journey which may look something like the below:

An equivalent architectural representation on Azure may look like this:

A good data strategy values ethics and responsibility

Taking a principled approach on additional, but very relevant considerations, such as data governance and responsible AI will pay dividends later.

At Microsoft, we use four core principles of fairness, reliability and safety, privacy and security, and inclusiveness. Underpinning this is two foundational principles of transparency and accountability. As we move from principle to practice, we’re sharing our learnings to help you on your journey. We put responsible AI and our principles into practice through the development of resources and a system of governance. Some of our guidelines focus on human-AI interaction, conversational AI, inclusive design, an AI fairness checklist, and a datasheet for datasets.

In addition to practices, we’ve developed a set of tools to help others understand, protect, and control AI at every stage of innovation. Our tools are a result of collaboration across various disciplines to strengthen and accelerate responsible AI, spanning software engineering and development, to social sciences, user research, law, and policy.

To further collaboration, we also open-sourced many tools such as Interpret ML and Fair Learn that others can use to contribute and build upon alongside democratising tools through Azure Machine Learning.

Capture the data opportunity

The pivot to becoming a data-driven organisation is fundamental to deliver competitive advantage in the new normal. We want to help our customers shift from an application only approach to an application and data-led approach, helping to create an end-to-end Data Strategy that can ensure repeatability and scalability across current and future use cases that impact business outcomes.

Working with 1000s of customers across different industries of varying complexities we can pick out optimal patterns and tailor transformation plans to accelerate time to value. Be it retail, financial services, manufacturing, health care or public sector, we have the industry knowledge, and deep domain expertise to help you build a resilient data strategy and culture.

Let’s get started. Contact your account team at Microsoft or connect with a Microsoft Partner.

Find out more

Download: A Guide to Data Governance

How to build and deliver an effective data strategy: part 1

How to build and deliver an effective data strategy: part 2

Driving effective data governance for improved quality and analytics

Powering digital transformation at Microsoft with Modern Data Foundations

Designing a modern data catalog at Microsoft to enable business insights

Resources for your development team

Learn how to manage identities and governance in Azure

About the author

Pratim Das is the Director of Data & AI Architecture and CDO Advisory at Microsoft UK. Pratim and his team’s mission is to work alongside their customers in delivering insights and most importantly value from data, in achieving great business outcomes. Be it retail, financial services, manufacturing, health care or public sector, they have industry knowledge, and deep domain expertise to build a resilient data culture, and customer capability. Pratim’s special interests are around operational excellence for petabyte scale analytics, and design patterns covering “good data architecture” including governance, catalogue, privacy and data democratisation in a secure and compliant manner. Pratim brings over 20 years of experience both as a customer, and also working as a technology vendor building Data & AI services, with a key focus on building capability, products and solutions, that leads into fostering a data driven culture. 

The post How to build and deliver an effective data strategy: part 3 appeared first on Microsoft Industry Blogs - United Kingdom.

1. Building a capability organisation-wide and project-wide

Now that you have your key projects mapped with business outcomes and calibrated with impact and complexity, and your baseline, you can start looking at building the capability to deliver them.

The first step would be to look at all capabilities you need, either at an organisational level holistically, or at a project level. Start by mapping what you have.

As a next step, look at Azure native services, and start mapping what you need to deliver success.

2. Culture is a key part of a successful data strategy

To build a successful data strategy, you need a data-driven culture. One that fosters open, collaborative participation consistently. This is so the entire workforce can learn, communicate and improve the organisation’s business outcomes. It will also improve an employee’s own ability to generate impact or influence, backed by data. Where you start on the journey will depend on your organisation, your industry, and where you in the maturity curve. Let’s look at what a maturity curve looks like:

Level 0

Data is not exploited programmatically and consistently. The data focus within the company is from an application development perspective. On this level, we commonly see ad-hoc analytics projects. Additionally, each application is highly specialised to the unique data and stakeholder needs. Each has significant code bases and engineering teams, with many being engineered outside of IT as well. Finally, use case enablement – as well as analytics – are very siloed. 

Level 1

Here, we see teams being formed, strategy being created, but analytics still is departmentalised. At this level, organisations tend to be good at traditional data capture and analytics. They may also have a level of commitment to cloud-based approaches; for example, they may already be accessing data from the cloud. 

Level 2

The innovation platform is almost ready, with workflows in place to deal with data quality, and the organisation is able to answer a few ‘why’ questions. At this level, organisations are actively looking for an end-to-end data strategy with centrally governed data lake stores controlling data store sprawl and improving data discoverability. They are ready for smart and intelligent apps that bring compute to the centrally governed data lake(s), reducing the need for federated copies of key data, reducing GDPR and privacy risks as well as reducing compute costs. They are also ready for multi-tenantable ,centrally hosted shared data services for common data computing tasks and recognise the value of this to enable the speed of insights from data science driven Intelligence Services. 

Level 3

Some of the characteristics of this level are a holistic approach to data and projects related to data being deeply integrated with business outcomes. We would also see predictions being done using analytics platforms. At this level, organisations are unlocking digital innovation from both a data estate and application development perspective. They have the foundational data services including data lake(s) and shared data services in place. Multiple teams across the company are successfully delivering on critical business workloads, key business use cases, and measurable outcomes. Telemetry is being utilised to identify new shared data services. IT is a trusted advisor to teams across the company to help improve critical business processes through the end-to-end trusted and connected data strategy. 

Level 4

Here we see the entire company using a data-driven culture, frameworks and standards enterprise. We also see automation, centres of excellence around analytics and/or automation, and data-driven feedback loops in action. One of the outcomes of a data-driven culture, is the use of AI in a meaningful way, and here it is easy to define a maturity model as the one shown below.

3. Focus on architecture

When considering every data product or service, it’s important to focus on the architectural principals. Think about whether you want to continue to manage and maintain your current service or products, or undertake new ones. The five architectural constructs are detailed in the Azure Well Architected Framework and summarised below.  

1. Security 

This is about the confidentiality and integrity of data, including privilege management, data privacy and establishing appropriate controls. For all data products and services, consider network isolation, end-to-end encryption, auditing and polices at platform level. For identity, consider single sign on integration, multi-factor authentication backed conditional access and managed service identities. It is essential to focus on separation of concerns, such as control pane versus data place, role-based access control (RBAC), and where possible, attribute-based access control (ABAC). Security and data management must be baked into the architectural process at layers for every application and workload. In general, set up processes around regular or continuous vulnerability assessment, threat protection and compliance monitoring. 

2. Reliability

Everything has the potential to break and data pipelines are no exception. Hence, great architectures are designed with availability and resiliency in mind. The key considerations are how quickly you can detect change, and how quickly you can resume operations. When building your data platform, consider resilient architectures, cross region redundancies, service level SLAs and critical support. Set up auditing, monitoring, and alerting by using integrated monitoring, and a notification framework. 

3. Performance efficiency

User delight comes from the architectural constructs of performance and scalability. Performance can vary based on external factors. It is key to continuously gather performance telemetry and react as quickly as possible, i.e. using the architectural constructs for management and monitoring. The key considerations here are storage and compute abstraction, dynamic scaling, partitioning, storage pruning, enhanced drivers, and multi-layer cache. Take advantage of hardware acceleration such as FPGA network where possible. 

4. Cost optimisation

Every bit of your platform investment must yield value. It is critical to architect with the right tool for the right solution in mind. This will help you analyse spend over time and the ability to scale out versus scale in when needed. For your data and analytics platform, consider reusability, on-demand scaling, reduced data duplication and certainly take advantage of the Azure advisor service.  

5. Operational excellence

This is about making the operational management of your data products and service as seamless as possible through automation and your ability to quickly respond to events. Focus on data ops though process automation, automated testing, and consistency. For AI, considering building in a MLOps framework as part of your normal release cycle. 

4. Tools and technology to power your data strategy

The right set of tools and technologies will be the backbone for your data products and services. Here are some of the key considerations to take.

Do not get stuck in a never-ending learning or design loop, otherwise known as analysis paralysis, or building PoC after PoC. Beyond a certain point, additional time spent in this cycle does not add equivalent value to your organisation’s business objectives.

5. Think big, start small, and act fast.

Even if you don’t have 100 percent of the features from the get-go, it is more important to get started in delivering business value iteratively. Leave the rest to product innovation from vendors and the capability you going to build with each iteration. Growth mindset is cultivated best when we accomplish more with less. This balance is an art, it fosters creativity and innovation.

Simplification is key. At Microsoft, we have been innovating on behalf of our customers. We have many services for data procurement and many more for storage – depending on the volume, variety, velocity, and veracity. Similarly, an array of services for analytics, visualisation, and data science. Despite the flexibility and options, we understand that simplicity is important. A holistic solution that you can get started with immediately makes it easy to see return of investment quicker. For example, we see Azure Synapse Analytics as a category in its own right. It has ample integration options in your current estate, as well as ISV solutions.

In a nutshell, what we have created is a single integrated platform for BI, AI and Continuous Intelligence. This is wrapped under four foundational capabilities of: management, security, monitoring and a metastore. Underpinning this is a decoupled storage layer, data integration layer, analytics runtimes (either on-demand as serverless, or provisioned). The runtimes provide choice, such as SQL with T-SQL for batch and interactive processing, or Spark for big data, and support of most languages such as SQL, Python, .NET, Java, Scala and are all made available through a single interface called Synapse Analytics Studio.

The start of a successful journey to a modern data strategy

This principled approach will help you shift from an application-only approach to an application and data-led approach. This will help your organisation build an end-to-end data strategy that can ensure repeatability and scalability across current and future use cases that impact business outcomes. Our final blog in this three part series takes a deep dive into how you can execute your data strategy successfully.

[msce_cta layout=”image_center” align=”center” linktype=”blue” imageurl=”http://approjects.co.za/?big=en-us/industry/blog/wp-content/uploads/sites/22/2020/09/PreviewImage-2.png” linkurl=”http://approjects.co.za/?big=en-gb/industry/blog/?p=40439&preview=true” linkscreenreadertext=”How to build and deliver an effective data strategy: part 3″ linktext=”How to build and deliver an effective data strategy: part 3″ imageid=”40718″ ][/msce_cta]

Find out more

How to build and deliver an effective data strategy: part 1

Explore the Azure Well Architected Framework

Driving effective data governance for improved quality and analytics 

Designing a modern data catalog at Microsoft to enable business insights 

Download: A Guide to Data Governance

Resources for your development team

Encourage your developers to explore modern data warehouse analytics in Azure

About the author

Pratim Das is the Director of Data & AI Architecture and CDO Advisory at Microsoft UK. Pratim and his team’s mission is to work alongside their customers in delivering insights and most importantly value from data, in achieving great business outcomes. Be it retail, financial services, manufacturing, health care or public sector, they have industry knowledge, and deep domain expertise to build a resilient data culture, and customer capability. Pratim’s special interests are around operational excellence for petabyte scale analytics, and design patterns covering “good data architecture” including governance, catalogue, privacy and data democratisation in a secure and compliant manner. Pratim brings over 20 years of experience both as a customer, and also working as a technology vendor building Data & AI services, with a key focus on building capability, products and solutions, that leads into fostering a data driven culture. 

The post How to build and deliver an effective data strategy: part 2 appeared first on Microsoft Industry Blogs - United Kingdom.

In this three part blog series, we’re going to guide you along the journey to build a strong data strategy. This first blog is designed as an introduction and will explain the core characteristics of data and some of the ways you can build competitive advantage through your data strategy.

Identify potential challenges and roadblocks to overcome

It can be difficult to harness the power of data in a secure and compliant manner. Sometimes, you can run into challenges like organisational silos, building a data-driven culture, and a never-ending challenge coming out of running multiple tools and technologies across the organisation. Time to market is one of the most critical factors these days for all businesses. Organisations can have great ideas and data can be an enabler, but with so many challenges it can take weeks, if not months, before they can start getting insights and ultimately deliver business value from data.  

“An effective data strategy that delivers value to the business is based on four pillars: People with the right skills and data culture; Technology enabling speed of access to data; Processes to ensure security and repeatable patterns; Data the final golden ingredient that is trusted and understood. Get these right and the data strategy will fly.” 

Peter Jackson, Director, Group Data Sciences at Legal & General, author of Chief Data Officers Playbook and Data Driven Business Transformation.

Understand the 7 attributes of data for a strong data strategy

To build a strong data strategy, you first need to understand how data works. Understanding these core characteristics will help you build a principled practice around how to deal with data.

1. Data travels fast, but the velocity of data movement cannot defy the laws of physics. It must conform to the laws of the land or the industry that created it. 

2. Data never changes by itself, but it is prone to changes and accidental loss, unless explicit measures are in place to mitigate such challenges. Ensure controls, databases and storage anti-corruption measures, monitoring, audits, alerts and downstream processes are in place to deal planned or unforeseen changes. 

3. Data by itself, and simply though the act of storing it, does not produce any insights or yield any value. In order to discover insights or extract value, most (if not all) data, independent of the volume, velocity, variety and veracity, has to go through four discrete steps: ingestion, storage, processing, and analytics. These each have their own set of principles, processes, tools, and technologies. Withholding data assets and related insights may affect socio-economic, political, research and investment decisions, hence it is of paramount importance for organisations to build the capability to provide insights in a secure and responsible manner. 

4. All data generated or procured must go through a data classification exercise, unless otherwise explicitly stated. Where needed, the gold standard for dealing with confidential data is encryption at rest and in transit. 

5. Data has gravity. This means that data, applications, and services all have their own gravitational pull. But data is the heaviest here, and therefore has the most gravitational pull. Unlike Newton’s apple, data doesn’t have a physical mass to draw in surrounding objects; instead, latency and throughput act as accelerants to the analytics process. 

6. Latency, throughput, and ease of access often warrants that data is duplicated even when that is not the desired outcome. Set up people, processes, tools, and technologies appropriately to balance such requirements against organisation’s data polices. 

7. The speed at which data can be processed is governed by architectural constructs, and facilitated through innovations in software, hardware, and networking. Some of the key architectural considerations are: setting up data distribution, partitioning, cache technologies, batch vs stream-processing, and balancing backend vs client-side processing. 

Prioritise the business outcomes you want your data strategy to achieve

It is key to align your data strategy with your business outcomes. Having a successful data strategy will give you competitive advantage. In essence, most business outcomes can be classified under one or more of these four umbrella categories:

1. Empower your employees  

Consider enabling your workforce with real-time knowledge of customers/devices/machines, efficiently collaborating to meet customer or business needs with agility. 

2. Engage with customers 

Deliver a personalised, rich, connected experience, inspired through your brand. Drive loyalty along every step of the customer journey by harnessing the power of data and insights. 

3. Optimise operations  

Increase the flow of information across your entire business operation. Keep your business processes synchronised and make every interaction valuable through a data driven approach. 

4. Transform your products and development lifecycle

Gather telemetry data about your services and offerings. Use the data to prioritise a release or create a new feature, and evaluate effectiveness and adoption continuously. 

Once you have prioritised your business outcomes, it is key to look at current projects, long-term strategic initiatives and classify them accordingly. Consider combining the 4 business outcomes in a matrix format shown below, based on complexity and impact. Also, think about adding the architectural pillars to help you dive deeper into the scenario.

Benchmark your current data estate and capability

Benchmarking your current estate is critical to measure success. This allows you to quantify the exact investment needed in terms of people, process and technology. You’ll be able to see where you are on the Maturity Model, and the gaps you need to bridge.

Our teams work regularly with many customers on these and can help you get started. Here is a table showing the thought process:

Category	Action areas	Current Status	Focus on 5 pillars of Architecture
Preparation	There are about 8-10 topics here starting from Strategy, Charter, Ethics, etc.
Agility	Depending on the organisation there could be 5 to 10 topics here, such as strategy around Data Lake, Catalogue, Common Data Model etc.
Resilience	10-20 topics are here starting from discovery, recovery and anomaly detection.

Figure 2: Assessing your current estate and capability is key in ensuring you can track progress and celebrate success. Using the maturity level detailed in part 2 of this blog, you can rate your current capability 0-4 and use the 5 pillars of architecture to guide those benchmarks.

Once you’re clear about the business outcomes you’re looking to drive through your data strategy, the next step is to build the capability to deliver them. The second blog in this three part series takes you through how to build your data strategy with practical steps to map your capabilities, deliver a data-driven culture, and evaluate products and services. In part three, we dive deeper into how to execute on your data strategy.

[msce_cta layout=”image_center” align=”center” linktype=”blue” imageurl=”http://approjects.co.za/?big=en-us/industry/blog/wp-content/uploads/sites/22/2020/09/PreviewImage-2.png” linkurl=”http://approjects.co.za/?big=en-gb/industry/blog/?p=40388&preview=true” linkscreenreadertext=”How to build and deliver an effective data strategy: part 2 ” linktext=”How to build and deliver an effective data strategy: part 2 ” imageid=”40718″ ][/msce_cta]

Contribution thankfully received from Nick Hurt (Senior Solutions Architect, Big Data Analytics), John Mallinder (Principal Enterprise Data Architect), Blesson John (Senior Solutions Architect, Data & AI), Dave Lusty (Senior Solutions Architect, Data & AI), Susan Meldahl (Director Business Programs), and Sumi Venkitaraman (Senior Product Marketing Manager, Data & AI)

Find out more

Driving effective data governance for improved quality and analytics 

Powering digital transformation at Microsoft with Modern Data Foundations  

Designing a modern data catalog at Microsoft to enable business insights 

Download: A Guide to Data Governance

Resources for your development team

Encourage your developers to get started with Microsoft data analytics

About the author

Pratim Das is the Director of Data & AI Architecture and CDO Advisory at Microsoft UK. Pratim and his team’s mission is to work alongside their customers in delivering insights and most importantly value from data, in achieving great business outcomes. Be it retail, financial services, manufacturing, health care or public sector, they have industry knowledge, and deep domain expertise to build a resilient data culture, and customer capability. Pratim’s special interests are around operational excellence for petabyte scale analytics, and design patterns covering “good data architecture” including governance, catalogue, privacy and data democratisation in a secure and compliant manner. Pratim brings over 20 years of experience both as a customer, and also working as a technology vendor building Data & AI services, with a key focus on building capability, products and solutions, that leads into fostering a data driven culture.

The post How to build and deliver an effective data strategy: part 1 appeared first on Microsoft Industry Blogs - United Kingdom.