All Databases In Cloud Managed With a Single Service - Aiven

[Music] managing databases or any other type of stateful applications is hard at least when compared to stateless apps the more stateful applications we have the harder it becomes if managing one database one is challenging managing tens or hundreds of them is a nightmare especially if those are of different kinds of databases or applications there could be a few postgress databases a number of my sqls a couple of Prometheus is dozen R instances One open search and so on and so forth now let's add an additional complication to the story imagine that you would like to manage all those and quite a few others and that you would like to do it through a graphical user interface wow am I going crazy here didn't I say over and over and over and over again that I like everything is code and stored in git type of approach didn't I say that many times well I did say those things and I do believe in them but I also recognize that not everyone shares my preferences still I would like to be able to manage my stuff as code stored in it and synchronized through gitops so that should be a requirement as well in other words it would be great no amazing if I could manage all my data stores either through click click click click click in a console or as code and preferably for those to be managed by somebody else I don't want to manage my own stuff all in all I want manage data stores both through a graphical user interface and through code starting it so today we will explore ways to use managed open-source data stores like poql MySQL red open search fromus and many others in uh an easy way and we'll do that through Ian a s solution that offers a graphical user interface but also a kubernetes operator and terraform modules for those that uh prefer everything is code and stored in G type of approach by the way do not go to Ivan directly uh I have a special link in the description I will explain later why it is special it's not a scam I promise so uh there is a link special link in the description now let's see I in action let's create the first Ivan service now before you proceed it might be important to say what I is the short version is that it offers man Services based on open- Source data stores we can use it to manage Kafka poql MySQL open search Cassandra red grafana and so on and so forth Ian takes care of provisioning scaling and managing all of those and all we have to do is tell it what we want and what else yeah those databases or data stores can be managed in any of the big three hyperscalers and then some others as well so I will start by creating a poql database I connected uh only my Google Cloud account to it so that's the only one I can choose from if you connected AWS AER or digital oce or up Cloud I think that those are currently supported anyways you would see those as well further on I can choose one of the zones in Google Cloud select the plan and a few other options type the name and click click the button create service that's that's the one and all that is left now is to wait for a few minutes until poql is running in your favorite hyperscaler in my case that would be Google Cloud even that might not be my favorite I use all of them anyways go cloud in my case in your case whatever you want so let's fast forward until this is finished and there we go it's running I can see all the relevant information or simply click the quick connect button to get the psql command that I can use to connect to the database and that's it I'm connected and I can confirm that the database is up and running by for example listing all databases in that server by executing back slash L now I can Define additional features like uh establish Integrations Uh define tags enable protection from termination upgrade the version and so on and so forth the interesting part is the ability to create and manage all sorts of Integrations let's say that I would like to have grafana connected to that postgressql database so that I can create dashboards based on the data coming from the DB I can do that by clicking the manage Integrations button over there I can see quite a few Services I can integrate with now that list is a bit misleading because not everything can be integrated with everything actually that's wrong everything can be integrated with everything but not everything makes sense I would be happier much happier if the integration screen would show only the Integrations that make sense for a given service in this case POS SQL instead of throwing everything everything all of that at me the rest could be hidden behind a button or something like that nevertheless I can see that I can integrate with grafana so I'll choose to monit monor data in grafana and follow the instructions to create a new instance of grafana and connect it to poql now we can see that poql is sending data to grafana that's a bit misleading though because it's not yet sending anything grafana is not yet running so nothing can be sent to it and even when grafana does start operating it will be pulling data from poql rather than poq pushing it to grafana now that's me being mean and complaining about wording on the console so ignore me ignore me completely what matters is that grafana instance is being created and we will Fast Forward for a few minutes until it is up and running and there we go grafana is now up and running and I can open it by clicking the service URI link I can go back and forth between the Ivan console and grafana to copy the user paste it to grafana copy the password paste it to grafana and uh login now if I open connections and inside it data sources we can see that poql is already there I even configured grafana to fetch data from poql V created or I created earlier now that that over there is great and I have to assume that I even also created a dashboard I can use to see metrics related to that poql database so let's open dashboards and there's only the general group of dashboards with nothing inside it that's disappointing I love the idea of adding connections between services but I would expect a bit more I expected a dashboard specific to that database and maybe a generic one for all databases but I got none of that and that over there in my opinion is a lost opportunity but actually that is not the case it is not the L opportunity but rather the reflection of how Integrations are organized if I chose an integration that makes more sense like for example Prometheus so that POG metrics can be scraped and stored in Prometheus I would be able to use Ian to connect to Prometheus and grafana to create dashboards related to my post database using metric stored in Prometheus what I'm trying to say is that Integrations are great but what you get from them can be just the connection or a connection and the dashboard and a few other things and besides Integrations we can see metrics logs uh query statistics current queries users databases pools and backups for poql database that all the typical stuff that you would normally do with in this case poql or any other database now let's go back to Integrations it is great to be able to integrate one service with another even if that integration is very rudimentary and limited to connections between them however as I already mentioned there's a full list of services that does not exclude those that cannot be or shouldn't be or don't doesn't make sense to integrate them so for example if I choose open search we can see that there are no compatible end points available so message to Ivan please do not show me services that cannot be integrated with the given service it's confusing now it is possible quite possible and likely that I can integrate poql with open search but not inside the trial account and if that's the case I would expect Ivan to tell me that in any case some Integrations make more sense than others and Ivan console is a bit confusing for newes which I believe are the target audience for Ivan now that I mentioned open search let's create an instance of so create a new Service choose open search select the zone and the plan type the name click the create service and wait until the service is up and running so let's fast forward a few minutes and voila it's up and running open search is running and it is managed by Ian I can open the dashboard ignore that it is not working refresh the page copy and paste the username do the same with the password and log in and that's it it op search is up and running and I can use it for whatever I might need it for now I would probably like to see metrics from open search in grafana along side poql metric so let's integrate that one as well so manage Integrations choose the monitor data in grafana select the existing grafana instance and click the enable button and that's it that was truly truly truly easy I can confirm that it was successful by going back to grafana and opening connections followed by dat sources now there's open search outside pql and I can build a dashboard or I can import an existing dashboard because in some cases you can you have additional options there Integrations to uh add dashboards through Ivan itself now let's move away from Integrations I want to show you uh an annoying thing that made me waste my time and even open an issue with Ivan and I'll go back to I and log in to the console as if I'm coming the next day now do you see something strange over there no yes well look at that none of the services I created so far are there they're gone they're missing here's the thing Ivan has multiple consoles there's the one you get after logging in but then there are separate consoles for each hyperscaler and you will not see links to any of those in the master console you would need to know that you need to open in this case Ivan console specific to to Google cloud and then you would see all the services created in Google Cloud the same would apply to alure n AWS it's a bit silly and annoying until you figure out that the experience is split between different consoles and that nothing tells you what you have and where you have it until you open the right console that was enough clicking for me I would like to switch to Ivan kubernetes operator that is much closer to how I prefer to operate stuff but before I do that let me delete all the services I created so far and get out of psql console by typing uh back slq and that's about it the alternative way to manage ien services is through their kubernetes operator or telephone modules but I'm a kubernetes person so I will choose to show you kubernetes operator so instead instead of clicking buttons selecting options and typing something in colorful screens we can Define what we want on as not on as kubernetes manifests let's take a look at an example that will accomplish a similar result as the pogress database we created earlier over here I'm defining postgressql resource with a name and spec inside the spec I'm defining the same properties as those I Define through the console now if this would be a real world example I would probably push that manifest to get and let Argo C flux synchronize it with the cluster now I I will not do that today because mostly because I was too lazy to install Argo CD and flux but you should be doing that so I will do what should not be done and execute Cube cutle apply manually and I will feel ashamed for doing that now creation of the poq resource will not result in the database running in that cluster remember Ivan is a sus solution that offers managed databases and other services related to data so creation of that resource triggered operator to create poql database in Google cloud and manage it VI Ivan the operator enables us to manage managed resources so while waiting for posq to be up and running let's take a look at the custom resource definition or crd uh or crds that that the operator created we can see that there are a few we can use the operator to manage projects click house grafana Kafka MySQL poql and quite a few others essentially everything we can do through Ivan console probably probably everything we can do through Ian console we can do through the operator as well now let's see the status of the poql resource by retrieving all postgressql resources in the Ivan namespace it's still rebuilding I notice the same in the console what is rebuilding well I chose to create a new database server so the state should be creating or building I I don't understand why it would be rebuilding something that I'm creating for the first time anyways my brain learned to translate rebuilding to creating so I will Auto translate that word over there you can also describe that resource and see the status and the events related to that resource we can see that the status remains unknown what does that mean can't I even tell us what's going on while it's re rebuilding remember rebuilding poql I don't know if I open the console I can see that the operator is a way to define manage resources through code and that those same resources are a reflection of the information that can also be seen in the console now let me go back to my terminal and watch the posq resources in the Ivan namespace I'm watching and watching and watching and there we go the database server is now running and I can use it in the same way as I did with the one I created through the console and that's it for the demo let me press contrl C to stop watching the resources in my kubernetes cluster and delete the poq resource I created earlier now let's talk about money Ian provides open source projects related to data as managed Services there is a free plan available for postgressql MySQL R I think that those are the three and the free plan is free forever on top of those you can also get 300 bucks worth of credits to spend during 30 days actually please use the link in the description that will give you additional dollars in credits and I get an extra 25 bucks that I can use for future videos when I need to spin up a database it's a win-win situation so use the link in the description I get 25 bucks full transparency and I will use it for the next video beyond the free tiers and credits Ian pricing is easy to predict but hard to compare with the price of similar Services when they exist in hyperscalers it's hard to calculate how much something costs in Google cloud and almost impossible to do the same in AWS since the price is based on much more than the amount of memory CPU and storage there are hidden costs in all hyperscalers hence I cannot say how much more expensive is or cheaper is to other similar Services Ivan claims 340% threeyear return of investment 37% lower threeyear cost of operations 78% reduction in stuff time etc etc I cannot confirm those numbers though so I will take them with a bit with a grain of salt I Ur you to check their pricing yourself and do the calculation yourself and now we can talk about pros and cons let's go so here's the question is I even the best way to run a database and the short answer is it is not there are many many many other ways to run databases and Ian is far from being the best so is it the easiest way to run a database well it is easy but not necessarily any easier than for example running manage poql in Google Cloud so what is the value proposition of Ian the way I see it hen could be very useful if you want to manage multiple data stores especially if not all of those are available in the hyperscaler of choice for example goog Cloud does offer afka poql MySQL elastic search through elastic cloud and redis in form of I think it's called Cloud memory store or something like that for red API however it is missing Cassandra grafana m3db click house and Flink if you need a combination of both those that Google Cloud does offer and those that they don't and you would like to use manage services for all of those you would need to combine Google manage services with services from other providers and that could pose a ton of challenges and by the way everything I'm saying for uh goog Cloud equally applies to AWS and AER and especially even more for digital ocean and up Cloud none of those offer all the services that Ian does and I can see a lot of benefit in using a single provider for all of them that's where Ivan shines it offers a single place for almost all databases in data related Services you might need it does not give us all the banks and whistles that you might want but it does focus on Simplicity and needs of use and Integrations between services with that out of the way let's talk about the good and the Bad and the Ugly the everything and we'll start with cons Integrations are useful but also a bit of a mess Ian should organize Integrations in a way that only those that can be used are shown or those that are used less or make less sense to use are hidden or behind the button or moved somewhere else for example when I click an integration with posg SQL showing grafana makes a lot of sense but having open search doesn't since it's not supported anyways or at least not in the version of the account I have the second thing is that support is always by email I believe on the best effort basis that makes a lot of sense for low tier plans but I would expect a bit more for higher tiers Live support would distinguish I from other providers especially especially hyperscalers what's next what else what else kubernetes operator is relatively basic and a bit finicky if for example I specify a wrong plan in the resource definition it throws an error in the events but still claims that it started reconciliation then it claims that it created an instance and uh that doesn't work it's not created right if you make a mistake it does not understand that you made a mistake and it thinks that you're cre anyways operator is a good start but a mess speaking of the operator there is no easy to find documentation with here the specs uh actually later on I found out that it's available but then it's sometimes the link is there sometimes it's not it's the link to the docks uh for the operator is broken or non-existent doesn't work anyway documentation is not good at all when operator is concerned and finally Ivan is not the most robust or feature solution to run any single database but that is okay as I already mentioned that is not the value proposition Ian hopefully does not try to be the best provider for specific database but rather to be the provider for all of them right so if if you're not if you want to have it all you cannot have the best at any given one of them and that's okay as for the pros I already mentioned most of them so here I'll Focus only on few if you do use more than one data store and you don't want to or you do not know how to manage them directly all those data stores through your hyperscaler of choice or if the given database is not available in your hyperscaler of choice Ivan is a good solution especially if running at scale or in multiple hypers scalers the addition of the kubernetes operator is a great way to enable everything as code and storing G type of approach even though it has its flows a lot I think it's actually good I welcome it many other s providers are still still trying to wrap their heads around kubernetes and its importance for manage services another Advantage is CL Independence if I'm running my databases through Ivan I could move them from one hyperscaler to another with a click of a button if I do that I even would create a second database in the new hyperscaler make a backup restore it mirror transactions being made in one or the other and so on and so forth now that does not happen often and I would not name migrations or Independence as the top feature but it's there and it might be useful one day if you ever choose to move from one hyperscaler to another infrastructure code is supported we can use Ian's telephone provider which is okay it works okay or the kubernetes operator which leaves a lot to be desired nevertheless we are not forced to operate databases using I console which is a good thing but just as migrations not necessarily the top features since from my perspective ien is first and foremost geared towards users that prefer a graphical user interface at least for now at least initially all in all ien is a good solution for very very specific type of users if you're one of those I would recommend giving it a try thank you for watching see you in the next one cheers

Channel: DevOps Toolkit

Views: 5,462

Keywords: devops, devops toolkit, review, tutorial, viktor farcic, k8s, kubernetes, Aiven, All Databases, Apache, Apache Cassandra, Apache Kafka, Cassandra, Clickhouse, Cloud Computing, Cloud Managed, Cloud Service, DB, Database Management, Database Service, Elastic, ElasticSearch, Grafana, Influx, InfluxDB, Kafka, MySQL, OpenSearch, PostgreSQL, Postgres, Redis, database, db

Id: VhlY7kkAw7w

Length: 25min 1sec (1501 seconds)

Published: Mon Sep 25 2023