Beginners Guide To AWS ElasticSearch Service

hi guys i'm johnny cheverus welcome to my youtube channel i'm data engineer with over 10 years experience working monday to friday primarily in the financial services sector five times aws certified and i have nothing more in my free time than making videos for this very youtube channel today we're going to take a look at a beginner's guide to aws elastic search service so let's just get right into what is elasticsearch elasticsearch service is a service provided to us by aws that implements elastic search in a managed way so what is elasticsearch elasticsearch is an open source software project that lets us upload data such as geospatial strings documents and then search that data in an efficient way in a cluster of computers or a single node in today's instance think about it like it's your own personal google or personal search engine we upload data into the software known as elasticsearch this is hosted for us by aws in the elasticsearch service and then we can search for that data by specifying keywords or geospatial data we could do it on coordinates for example once that is in there and you're searching there's a couple of ways to do that so there is an api that you can just call off the console and we'll do that today and that's how you maybe integrate a custom application with elasticsearch there's also an open software ui called kibana and cabana lets us easily upload data search for data build visualizations like charts and graphs if you were doing like real-time interaction with that data so we'll take a look at kibana as well and then as i said we'll already look at that api but why use elasticsearch service well as i said it is open source elasticsearch itself and you can actually download it for free and implement it yourself but this is a bit of a pain so you have to know how to download it obviously from the open source compile it put it on the linux instance manage that linux instance look after all the infrastructure the firewalls the bpcs or it was abstract all this away and although there are options for us to kind of get more granular onto it we can't just spin up our own cluster with elasticsearch already installed and then we just get on with inserting the data uploading the data and then searching for it so it lowers that biotic entry and it abstracts the infrastructure away from the end user that really means as developers we can just get on with getting putting the data in and getting it back out and the infrastructure becomes less complex and there's automatic updates built into it but when user lasted search so i've saw this service used in numerous use cases throughout multiple businesses a lot of businesses maybe start with their logs put your logs into it from all your interactions in the aws clouds and you can start to build visual representations of that data such as bar charts pie charts line graphs you need more advanced things obviously as well it's great for hosting website data and you search through so if you have a big bank of questions and you want to search through it elasticsearch could be your answer geospatial data because obviously when you're searching geospatial data you do a lot of coordinates lookups and obviously you want a search engine possibly over a database and then elasticsearch brings that to you so you can search some coordinates and get back the name of a place quite easily versus doing that in a database and then on top of that other places for business analytics so you could bring different sources of data together put it in alaska search then build a cabana dashboard on top of that with a grass and that searches the underlying cluster for the data it needs so you can see that really it's when you need a search capability but you also need a high volume of data search service is really that go-to thing with that being said as usual with all these aws services it's a bit abstract talking like this so let's get onto the console and that's builder first the last search instance we'll upload some data and then i'll show you how to use an api to search for that data but also use the cabana inbuilt ui to look for the data as well and upload data i'll point that out right right there okay guys that's me logged back into the aws console first thing we need to do is navigate to elasticsearch service so just type in the top as usual elastic search service and we're looking the elastic search service if this is the first time you've been on the page then just click create new domain we're just going to do a development and testing uh one availability zone for when you just need an endpoint so as i said this is just our testing zone so let's leave it there just select the most recent version it's currently 7.10 for me hit next give your elastic search a domain name so i'm going to call this one movies we don't need to worry about the custom endpoint here so we can just leave that um auto changes leave as well as is data nodes we just want the smallest one so there's actually a free tier one which we will go and find and we want a general purpose t3 small i'm going to go for that should do this uh just the one node as well leave everything else as default on the data nodes storage section and click next to keep this simple i'm going to configure just full public access we want to actually access this by password so we're just going to create a master user so click offset i am and click create master user you're going to have to give this a username and password that you remember so write it down if needs be this is only a demo so i'm not going to worry too much about security here always worry about security in real life though we don't need to worry about some authentication here we don't really want to worry about cognito either right now we can just ignore that section for domain policy choose open access so run open access that just means that anyone can reach this again because it's a demo i'm not worrying too much about security it's more about getting it working and actually showing it in action so we're going to leave it fully open keep everything else at the default values and click next don't need to worry about tags click next and then once we have it here let's just double check everything 1az t3 small so it's absolutely cheap as chips ebs is in there public access is enabled we are good to go and again just make sure you've wrote down your password and confirm as you can see we've successfully created an elastic search domain it's actually loading right now so you can click this little refresh button it takes about 10 minutes usually before the cluster is fully up and ready so i'm just going to pause the video here and then we can pick it up once this goes to enable okay guys the endpoint has gone active that took about 15 minutes in total although it does say it should take 10 just be aware of that um borrow with it and just keep hitting refresh and eventually it will become available it's just one of those things have to borrow with it so the next stage is we need to load some data in the description there's a link to my github where i have put a command we're simply just going to load um one uh piece of data into this uh cluster i'll show you how to do it and then you can go ahead and you can make more as you go so the first thing we need to do is just copy this um i think the simplest thing to do for me right now is maybe just i'm just going to double click and copy it out i'm going to bring it over to a little text pad um just so you can see everything that's going on i'm just gonna make that a little bit bigger okay and i'll just pull this out a little bit more for us so there's a couple of things we need we need to replace in here one is the master user one is the password the other one is the domain um that we put in because we need to replace it with our domain so the first thing we need to do is replace that user so i just called mine admin okay so once you have your username and password in just make sure that you've still got the actual tilde in and it hasn't changed that into a quote for some reason sometimes on the text pad it will alter things next thing we need then is a domain endpoint to go in here so i'm just going to delete this out and if we go back on the elasticsearch and we take the endpoint which is right here and we just go copy and paste that's us now because i called my movies you can also just use movies there if you've called it a different domain space then you need to change that movies command okay now we have the actual command ready to go let's copy that you can run this off your terminal or your command line but to keep things simple for us and everyone follow along i'm going to use cloud shell so aws cloud shell is a little tool now that they give you to let you run commands in the cloud i just did it by hitting that little play kind of button up on the top left side the bell icon opens a new tab configures an environment so i'll pause the video here once the environment is configured i'm just going to run that command and the reason i'm doing this is just because this is a beginner's guide and obviously people might not be too familiar with command line there are issues between windows command line linux command line and a unix command line on a mac and by issues i mean differences so this way at least it's all consistent for so now i'm just going to copy and paste in that text hit enter and as you can see it successfully put in my um my piece of data that i wanted to put in there just bear that in mind um that's only one little piece of data i've put in you can put more in now if you wish just go in and start changing around the different things you see so you could change around tim burton the year the actors if you wish so that's now the data in and now we want to search that data so there's a couple of different ways to do this once do command line or we can enter a command here so i'll show you how to do that um i'll put the command on the github as well so you can go back on the github you can grab that second command now and it looks like this what you want to do again is replace that username and password right here sure that studies with the tilde and then you want a copy and paste of the domain again and i'm just doing this for speed i'm in movies leave it as movies so i'm just going to copy that command and then what you want to do is just paste it in the command line so i took a new you can take a new tab there to keep things um nice and neat so that's a new tab and then i paste it in as you can see and i hit enter and it brought me back mars attacks there at the bottom with all the other information so that's the command line way but there is a ui and the ui is also open source and it's called cabana so if we go back in the elasticsearch and you can see that you actually have your cabana url click on that and then to log in enter your username and password that you've been using all along we've already added some data you can add more data by clicking add data but i'm just going to explore my own for now we're going to want to just do global and confirm over on the left you want to hit the hamburger menu you want to go to stack management once on stack management you want to go to index patterns you want to go create index pattern then once on the index pattern we want to type in movies because we only have movies so it's just movies and then we want to click next step and we want to go create index pattern this will only take a few seconds and that's the index created and you can see that there's different things um that's picked up such as director directory keyword and then different searchables okay back under the hamburger menu you want to go to discover now okay then once on this is now your search um language so we see i can type in mars hit enter and you can see that it found mars if i take that out and hit burton you can see it didn't find burton that's everything for today guys it's a really kind of basic intro in the alaska search and cabana just loading one row and um on their cluster and then searching for it as i said you can add more data now we have cabana open up and running or alternatively can perform more searches that being said that's everything for today i'll make everything available as usual for free on my website www.johnycevers.co.uk until next time guys thanks for watching

Channel: Johnny Chivers

Views: 3,691

Rating: 5 out of 5

Keywords:

Id: KNFSJbg7i-c

Length: 12min 44sec (764 seconds)

Published: Tue May 18 2021