>> Hey friends, did you know with Azure Arc enabled data
services that you can bring SQL Managed instance
and Postgres Hyperscale to any Kubernetes infrastructure
no matter where that may be. Lior Kamrat is going to tell
me how today on Azure Friday. [MUSIC] >> Hey friends, I'm Scott
Hanselman and it's Azure Friday. I'm here with Lior Kamrat and
we're going to talk about Azure Arc enabled data
services today. How are you? >> I'm goods, Scott,
it's good to be back. >> It is, it's nice to see you. I always like to see
what's on your shelf and I always feel like my
office is cool and then I see your office and
now I need to take it to the next level and that's exactly what we're going
to do today with Azure. >> This is how we roll
Scott, this is how we roll. >> Absolutely. Maybe tell us about what we talked about last time on Azure Friday so folks can take
a look at that episode as well. >> Yeah, so last time when
we are on Azure Friday, we talked about Azure
Arc-Enabled Kubernetes. We did a nice demo
around Cloud and Edge, how you can we do Fleet
Management for Kubernetes, GitHub across Cloud and Edge, on-premises with some Raspberry
Pi's on these modes going on, and today we're going to talk about Azure Arc-Enabled data services. >> That's going to make
data available to everyone everywhere no matter where they
are and how they use their data? >> Yes, exactly. Let
me share my screen and we can talk about the architecture and like always, how
we like to roll. We're going to do a cool
demo and show how we can control data services
across Kubernetes. >> Sounds like a plan. >> What I'm showing you right
now is the architecture for Azure Arc-Enabled data services. To take us back, when it comes to Azure Arc-Enabled data services, what we're doing is we're
making our own data services, Azure SQL Managed Instance, and PostgreSQL Hyperscale, approachable outside of Azure. If you remember in our previous
conversation on Azure Friday, we talked about this notion
of projecting resources. While protecting resources is
taking resources that are deployed, or workloads that are
deployed outside of Azure and bring those
into our control plane, this is really going by
on the opposite side, which means that we are able to deploy SQL Managed
Instance and Postgres on Kubernetes clusters that are not actually located
physically on Azure. Today we're going
to talk about that. >> Is that a fair statement to
say that what Azure Arc provides this location transparency
to these services? >> Yeah, you can say that.
Location transparency or a way for you to bring our services to
any infrastructure really. Just being generalized about that. That's really what it brings. >> Interesting because people,
when they're doing this work, they have to think about the
Software-Defined Networking and all of the different
structures about, in this case, looking
at your example, there's these boxes and sometimes
those boxes represent subnets. Sometimes they represent
logical units. Sometimes they represent
specific nodes or pods that are running
within Kubernetes. It depends on what the boxes
and lines are representing. >> Yeah, so here, really,
what's were targeting, our deployment unit, is really Kubernetes or our
infrastructure unit is Kubernetes. What we're doing here
is we're basically using our data controller
with this as a component. This is the brain of the octopus. The data controller talks and responsible for deploying
the actual services but also reporting back to Azure because at the end you want to
get that telemetry, that billing, and
all the integration. As you can see on the left, that also enable the
native tooling prospect, meaning that the data administrator or a Kubernetes administrator
doesn't need to change the way he goes by and managing infrastructure
and these type of stuff. That's really where it
sits in that portfolio, bringing those services in a
native way to outside of Azure. >> Location transparency is a
simplistic way of looking at it. This is really a formal
unified control plane on an API that is already understood and with tooling
that it's already understood. Whether someone's using
the portal, the Azure CLI, or a Kube cuttle they have one control plane and Azure Arc makes it
transparent underneath that. >> Exactly, it's really
about how we go by and integrating with Azure
Resource Manager, correct. >> Very cool. Everything
is an Azure resource. Bring whatever makes you happy. >> Exactly. >> Now I understand. >> Yeah. Should we do the demo? >> Let's do the demo. >> Let's do the demo. What
we're going to do today, we're going to take a GKE
cluster and we're going to show the process of deploying SQL Managed
Instance on that GKE cluster. In previous Azure Friday, we had GKE cluster,
we had Raspberry Pi, and this is continuing
this messaging around you can use whatever
Kubernetes cluster that you may have but you're going to deploy a native Azure service like
Azure SQL Managed Instance. What I'm showing you right
now is a resource group. That resource group already have an Azure Arc-Enabled
Kubernetes cluster onboard it. That Kubernetes cluster is
actually a GKE cluster. To show you that this is
really a GKE Cluster, and we'll talk about the other
resources that we're seeing here, this is the Azure Arc-Enabled
Kubernetes cluster. I'll click that to see some
meta data coming in and also to show our viewers that
this is actually a GKE cluster. We also have that open in Google Cloud Platform to connect
in the dots here for our viewers. I'll come back to the GKE cluster. Let's talk about the other
resources that we have here. We have custom location, we have the data controller, and we also have a log
analytics workspace. The custom location responsible
for mapping a location, which is the Kubernetes cluster, and the namespace inside that
Kubernetes cluster to where the data service is
actually going to get deployed that's the
SQL Managed Instance. We'll see that in a second here and the data controller that
is already created. The data controller, like we
said, that's the component, the service that's responsible for connecting that
service into Azure. That's the brain of the octopus. What we're going to do now
is we're going to go to Azure Arc Center and we're going to start the process of deploying a SQL Managed Instance. This can take a while but
we're going to show how the pods gets created and then we're going to flip to an already good to go environment so we
can see how we can abuse some of those services and
actually show the value here. What we're seeing here
is Azure Arc Center. This is the place when
you can go by and deploy all your Azure Arc services
manage those, so forth. This is your starting point
when it comes to Azure Arc. What I'm going to do, I'm going to go here to SQL Managed Instances. As you can see, we have a breadth of services that we're
bringing into Azure Arc, starting with Azure
Arc-Enabled servers, Kubernetes that we talked
about, data services. We also have App Services in preview and machine
learning in preview. Very exciting stuff
going on in this space. I'll go to SQL Managed
Instance and what we will do is we're going to start the
process of creating that instance. I already have an instance
created which we are going to abuse later
on in this demo. But first, what we're going
to do is we're going to start the creation process. The deployment of
SQL Managed Instance required few parameters so
let's go over those parameters. The first thing that
I'm going to do, I'm going to select
the resource group. I'm just going to call it
SQL MI Friday just for fun. Here's the custom location
that we're referring to. That's the first mapping
that needs to be done. That custom location
represent a namespace inside that Kubernetes cluster.
We have the service type. When you are deploying
a Data Service, you need to choose what type of network stack it will
get deployed on. The way that this is done is we are mapping the type of the
Kubernetes cluster that you have. We call that profiles. In this case, we're using Google Kubernetes Engine and
that maps the storage classes, the networking, and all that. All you need to do is to choose if you want to deploy
with the load balancer, which will give you
an external IP to that data service, in our case, SQL Managed Instance or you
want to use a node port, which means that that instance
will get a private IP. The only way to get to
that instance will be from via a jump box or
something inside the VNet. For the sake of this demo, I'm
going to go with the load balancer. You can see here that
compute and storage. This is where we are going to
control the type of resources that we can allocate
to that service. Today with SQL Managed
Instance, we have two tiers. One tier that is already
in GA, General Purpose, and the second tier
that's going to go GA pretty soon, business-critical. The difference between
the two is that general-purpose is, basically, a single replica that's going to get deployed on the cluster and the business-critical is going to give you three replicas
or two to three replicas. Depends on what you want to do. That's really around
high availability. You can see here, if I'll
click the business-critical, you will see those three
replicas you can choose from. The rest is just compute allocation
and storage allocations. Like I said, we're mapping
that into the storage class of Kubernetes so your Data Service
will actually get deployed. We'll continue the deployment
process and all you need to do is just username and password
for your SQL Managed Instance. At that point, you're good to go. Now, I'm going hit the "Next" here. We'll kick it off, that deployment, we'll do a bit of validation. The first thing that
I want to show you, we're going to kick
off that deployment. At this point, I'm switching to
my jump box machine inside GCP. That machine is the
machine that I use to deploy this GK environment. To show you what is
happening in the backend, I'll go and do a kubectl get pods
and I'll use the arc namespace. That's the namespace that I
deployed the data services or I am deploying to the data services that
you are seeing right now. Those data services are
actually located here. You can see already here that the SQL MI Friday is in
container creating state. Let's add the watch command
so we can see that in action. What we did here is that
the arc controller, the data controller that we already deployed before that
demo is now sending the execution command to Kubernetes basically it's a wrapper
around kubectl commands, and what it does is send that
command and deploy the data service. You can see the SQL Managed
Instance gets deployed, so this is going to
take few minutes. >> Can I ask an
ignorant question here? >> Yeah. >> Hopefully the
audience is thinking the same thing because I just
want to make sure I understand. These right here are the pods from the point of view
of the kubectl which is talking to the GCP, they're the Kubernetes controller and Kubernetes controller
has a number of pods. But is that a Azure SQL
Instance running inside Google, no, it's a proxy to one
running inside Azure? >> Well, actually it's physically
a SQL Managed Instance that is running as pod on top of that Google Kubernetes
Engine, Kubernetes cluster. That's really what it is. It's literally taking the service
and bring that outside of Azure and gets deployed on a Kubernetes cluster that is
physically located outside of Azure. >> It's an Azure
Managed SQL Instance. >> Exactly. >> The compute of it is operating
in this Google context. >> Everything, yes, the compute and storage everything gets deployed on that Google Kubernetes
Engine Cluster, yes. >> It's not proxied
from Google to us. It's not like a virtual node, and some of the things
that had been done in the past with Kubernetes where we were you're fooling Kubernetes into thinking
that it's happening there. But you're really proxying
data somewhere else. It's running fully
within this context? >> That is correct,
yes. As you can see, the puzzle starting to get created. To answer your question, you can see that we have physical
pods and containers inside those pods that are located
on that Kubernetes Cluster. >> Real nodes, not virtual nodes. >> Nothing is virtual here. These are SQL Managed Instance
pods that gets deployed and also what it can show you is the volumes and
the storage class. >> What container registry
is pulling what image from? >> Yeah, good question.
This is coming from MCR, Microsoft Container Registry, so everything is secured, gets governance, so you're getting
those containers from MCR. We also have a disconnected
mode that you can actually download those
containers to your registry on-premises or wherever that
may be and use those images. >> That really just crystallizes
the power of arc right there. >> Yeah. You're right. For people that didn't have a chance to
actually hit that "Deploy" button. The moment that you do that, everything starts to click because
you start seeing those pods, you start seeing those volumes. As you can see here,
you actually see volumes that gets
created for that SQL. You can see here the backup volume. You can see here the data
volume and the logs. These are real storage. This gets spin up on GCP that are allocated for
that SQL Managed Instance. >> Now, I get it. >> Yeah. >> This is really just a
toolkit of awesomeness. Now that you know
that you can do this, it opens up a whole
world of opportunities. >> Right. Let me show
you something here. I want to show you how things
looks like after everything gets deployed and we have all the automation and
we have a project. We talked about it in our
previous Azure Friday, we talked about the
Jumpstart project. We'll show how that
fits into the picture. Before I'm going to do that,
I'll switch back to Azure, just to show you the
resource that got created. You can see here that this
resource is still being created and it's a real resource. You see this is an Azure
resource that got deployed. No smoke and mirrors here. This is really a SQL Managed
Instance that is now deployed on a Google
Kubernetes Engine Cluster. >> Wow. >> To maybe take it
to the next level, I wanted to show you an environment
that is fully deployed, and I wanted to show you how the operation aspects look
on this type of environment. To do that, I'm using
project, Azure Arc. We created this awesome project. It's called the Azure
Arc Jumpstart Project. As part of that project, we
create a tons of scenarios. We have automations and a lot of cool things for our own employees, to get trained on
customers, partners. It's a community
open source project. As part of that project, we also created a product. This product called ArcBox. ArcBox is really our way to provide you with a full-blown
Azure Arc environment, or whether you want to do
Azure Arc-Enabled servers, Kubernetes or data services, this gives you everything you need. So everything gets deployed
for you automatically, and we're also providing some
sprinkles on top of that, which is the operational aspect
that we're going to show. At the end of this demo, I'll show where it is in
the Jumpstart Project. What I have here is a Jumpstart
ArcBox client machine. It's similar to the GK machine. A nice bonus is that when
you are deploying ArcBox, you also getting Server
2022 that just went out. Again, just a bonus, you can start playing around with
Server 2022 right off the box. When you're deploying Azure
Arc-Enabled data services, you're also getting some
additional components with that. Let's see what are these components. I'll open another partial
session inside that machine. For this one, I'm going to query the
pods and the services that are deployed on this cluster. Now, what is this cluster? Before that we did Google
Kubernetes Engine. But actually this one is a API Cluster or Kubernetes
is based on Cluster API. Now, people that are not familiar
with the Cluster API project, what this project is
all about is really deploying an unmanaged
Kubernetes cluster in a managed environment. It's funky that way because people that used to work
with Azure are used to AKS. AWS have their own flavor with EKS, GK with Google, Tensor with VMware, but all of these deployments are a managed deployments
or managed services. Cluster API gives you
the opportunity to deploy a Kubernetes cluster
in an unmanaged fashion. The reason that you want to do
that is because sometimes you want to have more granular
control on the nerve knobs, on the deployment, how
things looks like, and the way cluster API does that is using something called providers. We have an Azure provider, you have an AWS provider,
a vSphere provider. What it does, it basically
take the notion of the infrastructure and adapt the deployment of Kubernetes
to that infrastructure. Every time that you are
deploying these cluster API, it deploys a set of
virtual machines. To show you that,
and then we'll come back to these pods and all that. Again, to connect the dots
for the audience here, I'll switch back to my portal here, and I'll go to another
resource group. This is our box Friday
resource group, and as you can see, we have
tons of resources here. We have 79 resources, gets deployed as part of our box. But what is interesting is the fact that you don't see
an AKS cluster here. Again, this is a cluster API that got onboarded as an Azure
Arc-Enabled Kubernetes cluster, which is the fundamental
building block for you to go on with data services. What we're doing is we are
onboarding that cluster API, and then we're deploying automatically those data
services on top of that, which is what we are showing
right here in this machine. All the services that you
are seeing right here are basically deployed
on this cluster API. You can see that we already
deployed that SQL managed instance. Like we did before, we
have the data controller. We also have password SQL. That's another data
service that you can deploy as part of this Azure
Arc-Enabled Data Services umbrella. PostgreSQL Hyperscale is also included and we're
deploying that for you. What I also want to show
you is these type of pods, and what are they all about? We talked about the
operational aspects. What you want to do with Azure
Arc-Enabled Data Services, we want to get you closer to
this unified operation state, were you already get all
the tools that you're familiar with the
Kubernetes ecosystem for you to feel comfortable
when you are monitoring or extending your monitoring and
operational capabilities. The metrics UI pod that you see
here and all the other ones, and the logs are actually
a Grafana and it keep on a deployments that you
are getting outside of the box with Azure
Arc-Enabled Data Services. You can see that we
already have services, load balancer services on that Kubernetes cluster that
got deployed automatically. These are the ones that we're going
to approach right now and show how we are actually monitoring and abusing that SQL manage instance. To do that, first of all, I want to show you Data Studio. As part of our box deployment, you're also getting
Data Studio deployed. Not just that, you're also getting the SQL Managed Instance
and the PostgreSQL that is deployed automatically already
configured in Data Studio. The way that we're doing this is
because Data Studio at the end of the day is a JSON based descriptor, it has a JSON based descriptor which you can control,
you can manipulate. What we did as part
of the automation, we already connected those instances and we already deployed the
Adventure Works database. Right off the box, no pun intended, you will have something
to play around with. You already have that Adventure
Works databases deployed. Again, these are SQL
Managed Instance in PostgreSQL that are deployed
physically on that cluster API. What we're going to do is
I'll show you the Grafana, and then we're going to
run a query that will simulate a stress test on that, and we're going to see how that
looks like and connect everything. Here's the Grafana dashboard. This is what you're
getting out of the box. If you're in the
Kubernetes space, Grafana, a market leader when
it comes to monitoring an operational aspect of
Kubernetes environments, so we're no stranger to that. We also have Kibana.
With the Kibana thing, what you can do is you
can do log analytics and you can do log management. Just as an example, if
I'll do here the discover, you can see that I'm getting logs
already out of this environment. What I want to show you is the out-of-the-box dashboards
that we're getting. I'll go here to Manage, and you can see that we already
have bunch of templates here. We have the Host Node Metrics, pods, basically, the Kubernetes dedicated
dashboards for this cluster. We also have Postgres and we also
SQL Managed Instance Metrics. I'll go to the "SQL
Managed Instance Metrics". These are real metrics
like live metrics that are coming from the
deployment that we did. This is basically a
relaxed environment. We're not creating any stress
here. Everything is okay. You can see everything is stable. We're running that "Refresh"
every five seconds. What I want to do is I
want to show you how I'm performing a bit of a stress on this cluster and how
does that translate to Grafana. I have here a tool, SQL Query
Stress, and what I did, I just took a query
and paste it here, so I will use that to
generate the loop. What we're doing, we're
basically running that query. As you can see, 100,000 iterations
and using four threads. Again, just creating some
stress on the system. What I will do, I will
just hit the "Go" button, and you can see that
this is starting to run. Now, what's happening
in the background is that we are basically nuking the SQL instance and creating that dense stress because that's the way we're going to show how
things looks like in Grafana. We'll come back to Grafana here, and as you can see, the
trend is starting to go up. This is just a batch request, and you will see that
across the board with other dashboards that the
trend is actually going up, so we are starting to increase
the stress on the system. The reason that I wanted to show
you this is because this is a fully comprehensive deployment
that you are getting. It's not just about
the data services, the pure data services
that we're deploying, it's also about giving
you that experience, that you can go ahead, use that and start monitoring and doing some operations on
your environment right at day 1. That's what I have to
show you today, Scott. This is pretty cool stuff
and we have more to come. >> It is deeply powerful. There's a lot to absorb here, just in the Kubernetes space, but the idea that I'm seeing
is that y'all have created a toolkit for us to
do whatever we want, wherever we want, with
whatever Cloud we want. But you're enabling us to
use the APIs, the CLIs, and the tools that were used to using and I think
that's really powerful. >> Yes, exactly. For the viewers that wants to start getting their
hands dirty with this, I wanted to give you the reference
to the jumpstart projects. I'll go back to my Edge browser, and this is the jumpstart project. The nice thing about that, Scott, is that in our
previous Azure Friday, this was just a Git repository. Today we have a full-blown
project with a lot of scenarios. If you want to go
ahead and try our box, which is what I used as part of my automation for
preparing for this demo, you have all the instructions on the architecture and
how things are built, how to use it, and
really how to start taking advantage of Azure Arc
and learning the platform. >> Very cool. Folks can check
that out at AzureArcjumpstart.IO. I am learning all about Azure Arc-Enabled Data Services
today on Azure Friday. >> Hey, thanks for watching
this episode of Azure Friday. Now I need you to like
it, comment on it, tell your friends, retweet it, watch more Azure Friday. [MUSIC]
