<v ->Hello, everyone, my name is Deepak Agrawal,</v> and I have a unique opportunity today to talk about a product that has changed the course of big data analytics and that product is Azure Data Explorer, and today we are going to look at one of the key scenario around enterprise IT monitoring. Telemetry, when we look at telemetry, that is the key to digital transformation. You have got smart buildings, data coming in, you've got smart factories, connected devices. All of this data that is coming in is making companies change how they look at their data driven decisions today. And today, with how the data is flowing in, we expect that by 2030, they're going to be 50 billion connected devices around us, that will be emitting a significant amount of data that is going to go and make the change of how the digital transformation is taking place today. With that, I want to introduce Azure Data Explorer. It's a big data analytics cloud platform, optimized for interactive, ad-hoc queries. It's a big data platform, so you can imagine the data that is coming in high velocity, data that comes in high volume, and can have different structure. You can have semi-structured data like JSON, XML that you can ingest in this, you can bring your free text data. A typical data that is of append-only nature, the data that does not change over time, typically of the telemetry data. It's an analytics engine, so you can imagine all the relational query models, filters, aggregates, joints, you can create calculated columns, and there are tons of out of the box connectors and operators that can be used in order to make your analytical job very simple. It's a cloud platform, that means it's fully managed 24/7 supported within Azure. And it's a platform, so you get a vanilla database, you do not need to conform to any specific schema or a structure. You can design your own database using the schema that is right for your business, right for your scenario. It is not on any existing technology. It is purposefully built from ground up to enable rapid exploration of your data in a very ad-hoc manner. What ADX offers to its customers is you've got local disk cache, which allows you to run faster query, and you also have a colder tier that allows you to store data in a queryable manner, and that makes it very cost optimal for your scenario. We also support external tables, which lets you query data that is sitting in a blob container or in ADLS Gen2 and directly make a schema and be able to query the data. Platform out of the box provide indexing and compression capabilities. So everything that you bring in this platform is indexed by default. You do not need to worry about which column I can query. You can query almost every single column that is available in your database. By default, we provide caching and retention, which is helping you with your data life cycle. You can define these at the database or at the table level to take the control of how you want to retain your data. This platform also offers cross cluster cross databases query. So you are not confined to deploy this service in a specific geo. You can actually deploy your service within its own geo and be able to still query our data across the geo in a single query, and be able to look at a very comprehensive view of your own telemetry. All the auto scale capabilities that are all native to this platform. You can even define rule-based auto-scaling, which can change based on certain metrics that are already available as part of your cluster. This platform provides native data exploration capability, amazing text search capability, and very unique JSON parsing capability that allows you to index your JSON data in its native shape and allows you to query that with very easy. And that's something that is very unique to this platform. Out of the box time series, machine learning capabilities, you can run your Python and R code in a distributed manner. We have geospatial support as well. That allows you to run your special queries Very easy on top of this platform. You can bring your data that can come from multiple sources, data that is of different format. you bring your CSV data, TXT data or JSON data. You can bring all of that as part of this platform. This platform enables high throughput manage ingestion pipeline and supports both batch and streaming ingestion. A streaming ingestion offers a latency of around like less than 10 seconds. That really makes it a near real-time analytical solution that is available today. The platform is highly performant allows you to run many concurrent queries on top of your data, and you can really use it in order for doing a high skill performance analytics job On top of this. All the enterprise capabilities are breaking. AAD authentication is native to this. You can build low-level security, private link, You can bring your own key in order to secure your data. We support customer managed keys and all of those capabilities are natively available in the product. And it's all being leveraged. This platform is leveraging Kusto query language that is native to this. And that language is very simple yet very powerful for data exploration scenario. And last but not the least, this product allows you to focus on your business not operations. You deploy this service in cloud, it's 24/7 supported by by Azure. Auto-scale is enabled. Everything is indexed. You don't need to worry about what do I need to do in order to make this platform faster. All of that is done within the service fabric. You just need to focus on your business operations. This platform has been around since quite some time within Microsoft. It was first launched in 2015 for internal Microsoft workload. And it was GA in fact 2019 for external customers. If you look at almost every single Microsoft service is using this platform today for their telemetry analytics. Whether you're talk about Azure, you talk about windows, you talk about SQL. They all have been using this platform to run their telemetry, both proactive and reactive. This platform is also the analytical data engine for many of the other SaaS offerings that are available in Azure today. Then you look at all the offerings that are available within Azure monitor, being Log Analytics or application insight, they are leveraging ADX as the underlying data platform. There are other security products like Microsoft Defender, ATP and Microsoft Sentiment. They're all using ADX as their data platform. There are other services like IoT times-series insight, in gaming domain, you've got PlayFab. They all have been leveraging ADX as their data platform. And since it became a GA service, there are a number of customers. They are leveraging ADX to build their own data platform, to help their customers. So it's been quite mature in terms of how it is currently being used across the market, across multiple domains. Few key scenarios that I like to call out, which are some of the niche scenario for ADX is building near real-time big data analytics on customer events or log data. So you can bring your custom log, custom events data in ADX and be able to build your real time dashboards on top of this data. So that allow you to really build this near real-time analytical engine. The second scenario, which is actually one of the one that's been very popular used by many of the customers is to build centralized near real-time observability solution, where you can bring your on-prem data, your Azure data or data from other cloud providers into one single unified analytics platform, and be able to enable both visualize and alerting capabilities along with analytical capabilities. So that's sort of the key scenario where most customers are using ADX today. We have got other two scenarios where you can build your security log analytics product on top of ADX and also your IoT analytics solution. Or later in the slide, I've got few architecture that really talks about how you can leverage ADX in building these solutions and helping customers achieve their potential. Moving on. Today we are talking about ADX and how it is being used in IT monitoring space for large or small enterprises. When you look at enterprise telemetric data, typically the data can be categorized in four different categories. One is security logs where you want to bring the real time monitoring value from the security log. You've got system logs where you can bring your system counters data, your IFX logs, your syslog data and other audit data. Then you've got your application log, which can come from customer applications that can have varying schema, different retention for each one of them, multiple sources. It could be both real-time or it could be patched data. And then you have got another variety of data, which is around networking. You know where you have NETFLOW data coming in from your network devices, both modern routers, switches and whatnot in order to be able to look at your network analysis. In order to bring all of this data together, typically most customers are using this data to do data exploration. They bring this data into a log platform and the query this data for ad-hoc patterns, they also build their visualization reports to be able to share this data both within their operations as well as with their leadership team. Then other key scenarios around alerting, they want to be able to alert when something doesn't look right, or if something that is not really giving the right output. So that's one of the key scenario for this data. The other scenario is around security value, where you want to build your SIEM and SOAR capabilities. You want to be able to alert if something doesn't look right in terms of how someone is trying to access the data and not just that you also be able to create an incident, be able to manage the incident life cycle. The last but not the least that we have seen that how customers are using this data is for IT monitoring. I've seen many customers building their observability platform on top of this and be able to, again, integrate with ServiceNow and other incident management services, where they can look at the IT monitoring data, be able to alert on it and be able to manage through the incident life cycle. When you look at IT monitoring in Azure, this is how you know it's been stabbed. You've got Azure Data Explorer that is the underlying platform that is being used within Azure for most of the analytical workload, it offers all the agent support for all the open source agent support. You have got update policies, ingestion mapping that allows you to really perform your ETL as the data is being ingested. We have got ADX dashboard, that's a native dashboarding capability, Power BI, Grafana, and few other dashboarding support to be able to visualize your data. And also we have native support for logic apps that allows you to put the alerting layer on top of your data. That's a path, a service that is available for custom application logs or custom even data. on top of ADX now we have Azure monitor that has got both log analytics and application insight. And it has us offering currently being built for building observability platform, be able to provide it monitoring solutions. And on top of log analytics, there is Azure Sentinel which is using Azure log analytics as the underlying data platform. And it's another SaaS offering that is being built to provide security value. Now, the one thing that's common to the stack is the Kusto query language that is native to ADX. That's the same language being used in Azure Data Explorer. That is the same language being used in Log Analytics and Sentinel. And with this unique service ADX proxy now customers can also query cross service data. What it means is that the data that is sitting in a Sentinel workspace, you can directly query the data from workspace into the data that is sitting in ADX natively. So on the right hand side, I've got two screenshots that really shows when you are Sentinel hunting experience, how would you query the data that is sitting in ADX? You do not need to duplicate this data. You can directly equally this data that is available on ADX natively within Sentinel experience. And same thing is true for Log Analytics workspace. You can directly call into your ADX cluster and be able to join your data that is sitting in ADX as well as in Log Analytics workspace. So that's something that is very unique capability that offers, and that makes it very simple for you to build a solution that provides multiple values. And then top of this you've got no end to end monitoring solution that can be deployed in Azure. which is leveraging both SaaS and PaaS offerings to build and scale cloud native end-to-end IT monitoring platform for enterprises. So this is something where you can bring all variety of data. You've got security logs, you've got system log data, networking log, and other different application logs that can be ingested. And now the key USP for this architecture is that you're not duplicating any data. The data that is of security value is lending incentive where you have the most out of the box value from the platform. The data around IT monitoring and observability is lending into Azure monitor, which is the default service to provide native visualization alert capability. Everything else where you do not have any out of the box capability that is available today, you can leverage Azure Data Explorer. You can bring data from on-prem. You can bring data from your network devices natively within ADX by leveraging many of the managed pipeline that are already built in and be able to provide this comprehensive unified analytics platform. And again, once the data is in Sentinel or in Log Analytics workspace, or even sitting in ADX with ADX proxy, you can query this data across all of these three stacks, and that makes it very easy to build your entire unified analytics platform on top of your IT monitoring data. And once your data is in Sentinel or LA or in ADX, you can leverage all the SDKs that are available today, or the web UI for a docket analytics, or you can even leverage logic apps to build your alert. All of these capabilities are a lot of whether your data is sitting in either one of these platforms. And with this architecture, it's an hybrid cloud solution. You can bring your other cloud data as well in ADX. It's an Azure cloud native solution, wholly supported, you've got all the auto-scale capabilities, so you can scale your solution as your business grows, as you start seeing more and more data coming in. there is absolutely no data duplication in this that allows you to really build a very cost optimal solution, which allows you to run cross queries between all of these services. You can also bring your on-prem data as part of this architecture and all of the services that support KQL which makes it very easy for anyone to learn one language and be able to use that language across all of these pack of services. And with that, you can analyze your data. You can build your visualization layer, you can even alert on top of your data. And many times I've seen this question that comes in, why do we choose one platform versus another? What is the capability that I miss when I used the SaaS offering versus a PaaS offering? Now, there's a very simple answer to this. What is your customer is looking for? If you're looking for a solution that is fit for a purpose and provides every out of the box capability, that is what you're looking for, then you should be leveraging analytical SaaS offering that is available today. You have got Azure monitor which provides out of the box value for app and in for monitoring, you have one security which provides SIEM and SOAR capabilities out of the box. And you've got other services that really offers very unique domain capabilities for you to leverage. Anytime when you are constrained by scaling your service in a way that you want to take full control of your data sources, full control of your schema and how you bring data, how long you retain this data, or if you need not a multi-tenant service, but a single tenant service where you have the reserve costing model, where you have got resources that are reserved for you, or if you want to be able to leverage KQL for capabilities that are time series or telemetry data. Then you should be building your analytical solution on top of ADX, where you can have a flexible platform providing all of these capabilities to you. So that's typically how I decide whether I'm looking for something out of the box with no engineering costs on my side, I go with analytical SaaS capabilities that are available in Azure today. Otherwise I build my own solution by leveraging ADX natively. We have a customer case study where SAP did something very similar on top of Azure, where they build their entire platform. Top of a combination of these services. You've got data that is coming in from their infra. You've got application data that is coming in from SAP, and they leverage their interrogation pipeline where they're pushing some data for their application monitoring into application insights. Some data that is coming in from their Infra monitoring is going into Log Analytics and the rest of the data that is coming in from their custom applications, custom logs, that is all being ingested into Azure Data Explorer, and together, they provide a very comprehensive view into their entire telemetry by correlating data across all of these stacks. And then they're also leveraging Azure Data Lake for their long-term cold data storage, where they are building an archival tier by continuously exporting data that is going into Azure Data Explorer. With that, as I promise, these are some of the reference architectures that are available today in documentation, where you can leverage ADX along with other Azure services, to be able to build a variety of solutions for your customers. You can use ADX for interactive queries. You can leverage ADX for building monitoring solution the one that we discussed today, you can even leverage ADX to build your content delivery network analytics. I've got many customers who are leveraging ADX to bring their high throughput ingestion of their CDN logs and building near real-time analytical dashboards. To be able to look at the screen performance and few other scenarios like that. You can leverage ADX to bring high volume streaming data from IoT devices, sensors, connected buildings into ADX, to be able to provide this analytical ability, to be able to look at how these devices are interacting with the rest of the ecosystem. And another use case, which is currently being pushed on ADX is to be able to build your security data lake on top of ADX. You've got Sentinel, Windows Defender, ATP. These are the services currently being used in these domain. And they offer limited default retention on top of security data with the native capability of ADX being the underlying platform, you can very easily export this data from this platform and store this data into Azure Data Explorer for long-term retention, and be able to provide your SOC analysis, the unique capability of writing a single query, to be able to look at your data both in short term and in longterm view in a single analysis. with the new announcement that's happening around Azure Synapse analytics. Now, ADX is also going to be offered as another engine that is going to enable or enrich Synapse to really offer a new class of log and time series and analytical workload on top of Synapse. And with that Synapse will be a single enterprise platform for all analytics workload that is going to be offered to its customers. With that, I just want to take a quick recap, again, Azure Data Explorer, it's an interactive analytical service for fast-growing data. There are four pillars. What we discuss, you get instant big data insight on top of your high volume, high velocity data. It's a fully managed platform that allows you to scale natively as part of your growth. We have got everything indexed, by default you do not need to really spend a lot of time in operations. You can focus on your business, this platform democratizing data analytics. That means you no longer building cubes. You're no longer building definition of questions that are being asked for someone. You are actually taking these raw logs in front of developers, in front of your data science team, to be able to really, answer new questions, be able to discover new patterns. And this platform has been proven in order to scale from gigabytes to petabytes very easily. You need not to worry about, bringing your service down or be able to really have challenges when you see business growth. Very seamlessly using optimized auto-scale capabilities, you can grow your platform Very easily on top of ADX. And with that I really thank you all. some of these links related to product and out social handle and our forum are available on the slide. Please, please connect with us and thank you again for joining me here today. Have a great day.
