Elasticsearch Complete Tutorial With Python|Create Elasticsearch Index Using Python|Tutorial:1

[Music] [Music] what's up guys this is rani welcome back to our channel total technology this is tutorial one and this is going to be a brand new series and from the name it's clear that we we're going to talk about or we are going to learn about uh something on elasticsearch so i'm using the word something because i'm not sure how detail i'll be going to plan this series because actually i got so many requests so that is why i start doing this tutorial and uh this this weekend actually i'm having some uh like day off from my office so i thought of creating this tutorial but uh depending on you guys request or maybe interest or some question i'll plan the series accordingly so this is not going to be a regular series but if i get so many responses and uh based out of uh like you guys request i will uh continue this series but my primary focus is going to be graphql as i told you in the last tutorial okay so anyways so in this tutorial we're going to talk about the basics of elasticsearch again i'm not going to talk about uh very detail in the theoretical concepts of elasticsearch i believe all of you already familiar with elasticsearch that is why you are you guys are requesting me or asking me to do something with python because yes uh right now the data migration or uh uh data warehousing is one of the biggest topic in the industries everybody is migrating data from here and there from uh from a platform a to platform bin and while doing that you guys are doing a lot of like data injection data cleanup and so many other things so definitely some sort of automation job or automation scripts are required so that is why definitely python is the go-to language for them for at this moment at least so that is why i think it is really a good choice to use elasticsearch and python together and also elasticsearch and uh an api already written or in python so hopefully uh and that api can be exposed using the python library like like for elastic search within in python we have a library or module called elasticsearch you can use that to uh get set update whatever we can do using the elasticsearch api so definitely we are going to touch base upon all those use cases but as i told you everything is like there in my mind but nothing actually i planned depending on you guys request i will uh try to formulate and expedite the series so if you guys really want me to continue this series with a full pace then please send some questions and your interests and your use cases so i'll plan it accordingly okay so without further delay let's starts in this tutorial i'll be going to talk about just a little bit of elastic search and again after that how we can uh integrate or connect connect using python or definitely the installation part will be there okay but not that like code concept of elasticsearch i believe all of you are already familiar okay so first of all uh what is elasticsearch so i have only one slide for this and again it is taken from the elasticsearch official documentation page elasticsearch in the distributed and highly available open source open source search engine that is built on top of apache leucine uh it is an open source which is built in java that's available for many platforms so like by by reading this thing i believe uh most of you are not gonna understand anything same also happened with me like couple of years ago when you started learning this thing but again it's kind of a search engine which is available in the form of uh webpage uh like you can go and do something just like uh just like you open a browser and if you hit the url then you will able to see all the data and the data is coming from your uh uh like from a back end at this point just think about the back end and it is actually a built upon apache listing and all the datas are actually formatted in the structured uh unstructured json format so that is why it it's create internally announced no no sql database okay so that is why uh it's always sometimes not always sometimes it's also called a document database okay so like unlike like all other search engines also gives so many search or query related feature okay and there are multiple use cases for elastic search but i believe these three are very popular you can use elastic for multiple purpose a couple of given below suppose you are running a website that provides a lots of dynamic content for example you are running on ecommerce website and for that you are you are creating lots of api lots of uh multiple microservices database job multiple streaming jobs are there and they are running so now you want to uh capture all those logs uh with respect to each and every service each and every user clicks and everything so you can you can put everything inside your elastic search and that will work as a search engine so later on you can do some analytics or trend analysis based on uh your available data inside your elastic search for that web application means for your e-commerce application so there are some other use cases for example uh that is also very important in security or cyber security practices most of cyber security practices are actually uh using uh same stock solution so some lock collecting device or something like that so they generally ingest their lock inside the elastic search means that elk stack elastics are lock stuff in kibana elasticsearch actually process the lock from the lock is coming from the lock stack server and kibana is for the dashboard i'm not gonna go into that details actually so once the data is available uh their analytics uh team or their cyber security analytic analyst or security uh uh team can do some research or analysis based on the data available so these two are very important things and i believe most of you guys already came across all these use cases already okay so next thing is that installation so installation again i'm not gonna explain the installation this is very sorry this is very straightforward uh first of all uh uh you have to go to this website and download the elastic search uh if you i can for the sake of unlistening i can just show this thing for once okay so if you hit this url you will get this link and depending on your system or the machine you are using you can download this thing when you download this thing that will be definitely available in the form of zipper 70 format in your machine okay then the next thing i'll explain here so where is my this okay then the similarly you can download the kibana run the executable from bin fonda so let me explain how i actually download it so uh once i uh actually once i download this i actually unzip everything and after that uh these things are here if you see here okay so for kibana i got this thing and i have a bin file so what you have to do you have to execute this executable from your terminal and for elastic search also you will get a folder after unzip and there also will get one bin and you can execute similarly this thing okay so let me tell you i already running this two things over here over here so if you see here i execute this thing my kibana is running and i execute this thing my elasticsearch is running okay similarly okay here if you see here i i run this thing inside the bin this elasticsearch executable and inside here i just run the bin executable sorry the keyboard executable so similarly like elasticsearch runs on for the validation electric scratch first run from port nine two zero zero and uh kibana runs on five six zero one so once you your command prompt is up and running you have to just go to here and maybe you can validate your installation by going to this thing so if this is if if this thing is coming it means your elastic search is running congratulation okay and after that you can validate your kibana if you can if you're able to see this thing it means congratulations your keyboard is running so that's it so i'm not gonna spend a lot of time on uh the installation part so you have to go to this link download this thing unzip this thing and run the executable from the bin folder and just validate this thing don't spend much time on the installation okay simple just five minute okay so done then uh python settings so definitely we are going to talk about python so python is important so i believe all you guys are already running a machine which is already having python so next thing you have to install this uh elasticsearch using pip installer so for example uh maybe let me tell you how to install you have to right click install elastic search that's it see i already installed that is why it is saying already installed so again don't spend much time just do a uh one minute uh one not even one minute okay like maybe 30 seconds you can do it the next job is that connect to elasticsearch cluster so this is our first objective of this class how to connect with your elasticsearch cluster from your python code okay so let me minimize this thing okay so now our main job is here okay so first thing first from elastic search import elasticsearch okay then create variable yes and you can write it elastic search here okay and there you have to put all the connection objects okay sorry connection variable so it will be inside a dictionary so it should be something like host and our electric search port is running on localhost and the port will be all we know right so it should be nine two zero zero right no it should be nine two zero zero but again there is something i'm using in http and uh there is no connection security is there like no authentication is there so that is so you might get a warning okay but don't worry about that when you are using on your own machine for the rnd you can get warning but in when you are going to deploy this for your clients like elasticsearch definitely they will give you some additional security like ssl username password and that type you should use that okay and you will get all the information from the elasticsearch documentation if you search with elasticsearch python read the docs you will get everything okay so in the read the docs page you will get everything okay so again i'm not gonna spend that much time for all those things which is already documented so i'll do something interesting so that is why just i'm keeping everything simple and after that you can just do something like that here starting so it should give you a true means our connection has been established okay so let's execute this thing this thing python three and then tutorial one dot py okay so we'll get a true see i'll get a warning that security is not enabled that is fine but i'm getting warning true but if just for validation let's do one thing i let me change this port to 9800 so i'll get answer of false okay so i'll get answer of false and also that warning gone call this is not elastic search and when you are connecting to elasticsearch cluster which is not having any security enable definitely will get this warning so i give a wrong port so that is why i'm not hitting that elasticsearch cluster so there is no question of warning and there is no question of false okay so if we get this uh true in the ping so it means your elastic search is connected so our first job is done okay so let me uh copy paste this thing maybe maybe i can i will later on i can i will put all this thing here okay here don't worry so just if i don't forget that's why i'm just doing it so maybe in the later i can forget okay so next thing is that important thing is index okay so what is index so the very first thing you have to do is creating an index for whatever you are going to perform on the elasticsearch definitely we are going to perform some search or some analytics related things so before i do any analytics before that what you have to do you have to put some data data has to be available inside your elasticsearch cluster okay so now if i tell you uh go and put some data okay just forget about elasticsearch any any related any database database is very common and i believe most of you already familiar with database so if i ask you go and put some data what you do in the database like um sorry inside inside and and and and repository or whatever first you will create uh what you say that i need a database i need a table inside that i will i'm gonna put multiple data so table is something called uh uh index uh no no no table is something which is going to hold some uh some sort of information like if you are going to create a e-commerce website so probably you are going to hold multiple tables like inside your e-commerce database one is going to be like employee database or one is going to your hr database one is going to your customer database and inside the database there will be multiple tables okay same here uh very first thing you have to do creating an index everything stored is an index in in database concept everything is stored inside a database in elasticsearch everything is stored inside an index the rtms equivalent index is a database so don't confuse with the typical indexing so index means not anything indexing indexing means something a database for example in elasticsearch if you want to store your employee data then you have to create a employee index similarly if you want to create hr related data probably you can create hr index okay so same thing so index means a database which is equivalent to database okay means it is going to hold some specific data with respect to some specific type or pattern so pattern are types we'll be going to discuss later but just think about index means which is going to hold some data it's equivalent to a database okay so before we do uh let's check uh whether we have some database already index created or not so by default you will get some indexes so if you want to see how to check those indexes here or not so if you if you go to a developer tools and if you run this query you will get all the indexes see some indexes in the index already there okay how many index are there total probably one two three four five six so many indexes are there so here we're going to create some more indexes okay will create maybe initially we will create one index after that we will create two indices okay okay let's see how to do that okay so it is there so maybe i can write uh uh it should be yes dot indices dot create and after that you can write index and there you can write tutorial 1 and today's date is 16 so it should be 16 1 0 2 0 2 1 okay let's run this one so i got an earning message and i got true apart from that there is no error it means this index got created okay so let's try now that index is not here so if i hit this thing there will be one more index with this name okay so just validate our thing see it's it is created over here okay right okay so now we have seen how to create index now you see how many indexes are there so maybe i can put some comment over here setup connection maybe like this then create index display all indices okay how to do that let's do so maybe i can see interface equals to yes dot indexes dot get alias okay and maybe we can write star means everything okay and after that uh let's print this thing first so probably what will happen you will get a list and this link is going to hold all the indices okay so let's print this thing we'll get a list over here already okay the thing is that i'm running this thing again so that is why i'm getting an error because uh i'm trying to create the same index so that is why i'm getting an error so what i'll do i'll just uh skip this part for the time being okay so let's run this thing okay see here i got a list and list is going to hold all the indexes okay just to make it a bit uh beautify so what we'll do for index in in this then print hopefully this will restart all the names okay see it is actually listing down all the names one two three four all these things are there one two three one two three four five six seven eight see the same is here one two three four five six seven eight nine i believe nine is here also yes one is missing maybe i don't know why one is missing okay uh tutorial kibana tax manager okay so uh in this way you can actually create all your indexes and view the indexes okay okay so uh let me copy this code over here just hold on okay or else i will forget in the future but don't worry you guys will get this code from the github link as well okay so i don't need this one this is actually code okay so i'll update the code as well don't worry so i believe uh you guys uh enjoyed the tutorial this is a very initial tutorial so hopefully uh i will i'll get i'm expecting you guys uh send me some use cases or some some suggestion or advice so that i can plan this tutorial because uh i'm just doing this tutorial kind of on demand request so you guys have to tell me what i should cover but again something uh i i'm thinking uh if you guys are okay then i can just uh uh start from the basics like five to ten to twelve will be basic and after that i will uh create some tutorial which is actually being used in in nowadays in all the companies or in all our uh projects so that will be easier for help you but if you guys are having any specific request that also will be possible but if you want some sort of tutorial like with python or elasticsearch with some basic example then intermediate then advance and some real-world scenario that is so possible so let's see how it goes i'm expecting some feedback from all of you guys and depending on that we can plan accordingly okay so that's it guys we'll see you in the next video but before i uh conclude this tutorial guys please subscribe to our channel hit the like button share your videos with your friends family whoever you think is relevant because uh one share one like is actually going to motivate us like in a bigger aspect so this is a request please share our channel videos topics and please hit the subscribe button the bell icon so that you should not miss any notification in the future and also try to uh share with your family and friends whoever you think is relevant for our contents okay so that's it guys that's it for today we'll see you in the next video till then take care goodbye and have a nice day [Music] you

Channel: Total Technology

Views: 317

Keywords: elasticsearch, elk stack, elasticsearch automation with python, elasticsearch with python, elasticsearch complete tutorial with python

Id: w3S7mgFsC7U

Length: 21min 22sec (1282 seconds)

Published: Sun Oct 17 2021