Golang Microservices: Searching with Elasticsearch

hello my name is mario welcome to another video in today's episode i will share with you another tip for really micro services in go specifically let's talk about elasticsearch so what is elasticsearch elasticsearch is a distributed search and analytics engine if you have search about elasticsearch it's most likely you have seen the elk sort of like acronym i want to i wouldn't say acronym but it basically represents three different tools that are coming from elastic they complement each other and they basically the trifecta that elastic is sort of the company behind elasticsearch that happens to to be used uh together for doing different things so elasticsearch is the tool that we're going to be using for indexing things logstash which i mentioned previously during my open telemetry video is used for receiving logs for different applications and sending them those to elasticsearch for indexing purposes and kivana is the visualization tool that we can use for you know creating graphs some different dashboards and whatnot and mention on this because these tools are useful when we are trying to build something for analytics purposes specifically for this video we are going to be only discussing elastic search and mentioning that because elasticsearch is such a popular tool that is already uh supported by different cloud providers amazon has the manage elasticsearch and also gcp the google cloud platform has elastic cloud azure has an equivalent called elastic cloud as well so whatever cloud provider you happen to be using it's most likely already using or giving you an option to use elasticsearch so how can we use this in go well there are literally two options however those are competing options one of them will be oliver e elastic and the other one will be elastic elasticsearch the difference between the two of them is one one of them is official which will be the go elasticsearch package and the other one is unofficial which will be the one from oliver e elastic now the difference between besides that is that how long these packages have been available to the public in practice both of them are to date uh mature enough for using any production i've been using elastic which is the elastic oliver e elastic package for a while but i just recently started using the other one coming from the official company which is go elasticsearch so what would you choose if you're planning to build a new project so here's the thing what i'm thinking of choosing a package and i'm not only looking at the features but also of the i'm also looking at the possible maintenance options as well as the how long this package is going to be available for for me when i'm using it in production so one one thing that i noticed and this was two years ago when the go elasticsearch package was introduced oliver mentioned that hey it's most likely that this packs will not be see a v8 version of elasticsearch and that is something to keep in mind when we're trying to build a new microservice when we're trying to build a new product if the package that i'm going to be using is not going to be maintained in the future perhaps it makes sense to use the one that is going to be maintained and and and this doesn't happen that often but most of the times the companies uh if the company that is behind the product which in this case will be lastic if elastic is giving you an uh package and they are maintaining it is most likely again most of the times going to be going to have more support that the open source ones that there are some few exceptions but it's something to keep in mind as well so in this case what is going to be happening for this specific application of this specific demo that i want to give you sharing with you i'm not going to be using oliver e elastic but instead i'm going to be using the go elastic search again both of them are mature enough for using them in production so you cannot go wrong the biggest difference between the two of them is that the oliver one is actually giving you a more like dsl like sort of like a fluent uh api that you can use when creating your queries and the one from elasticsearch is literally everything much more manual so it's something to keep in mind as well but in the end both of them work and they do what it's supposed to be doing so what are we going to be doing next well let's go and look at the code so the code here is i made a few modifications to the existing application that we have been using for this video series again the link to the code will be in the description you feel please feel free feel free to clone it check it out and all of that so i have what i did is i built a new repository and if you remember the the video that i cover where i was calling repository dependence injection and service application and domain uh services and again i will be the link in the the the video in the description as well so i implemented that one that happens to be implementing a three different methods that will be index for updating and creating records one for up deleting records and one for searching records and the way it's going to be implemented is because the way elastic search works when you're indexing or when you're creating or updating is basically the same the same mechanism you would need to index the record is basically overwrite whatever you had before you can specify this is concrete fields when you're trying to update the different documents and whatnot but in practice it's just doing the same process all the time so if i look at the implementation that i have here there is a new type called task that is under the package elasticsearch and then i define a index method that will be like i said indexing the records when we are creating or updating any of the tasks when i'm trying to delete one of them i will be just basically that removing the document or the record from the index if i'm trying to search i will receive the values that i'm trying to use for searching purposes and just use them for for like i mentioned searching and the way i have it is that a also the open api file was updated as well and the open api file will allow us to if you haven't seen that video also i will leave it in the description and it allows us allows us to use the swagger ui to interact with the new with the new search endpoint that we're going to be using now the way it's implemented right now and it's not something i really recommend and i will be changing the this when i'm adding events and i'm when i'm going to be adding kafka producing events and consuming events but the way it is right now the way it is right now is that they have a service the application service that is uh receiving the event or writer receiving the message or receiving the the http request that is coming from the client and is actually calling the data repos the repository the the persistent data store for storing the record in the database as well as indexing the record in elasticsearch and i want to show you that so i have a create that is calling the repository that is in charge of doing the postgresql you know sql insert kind of command and then i have another one that is right here calling index which is the one indexing the record in elasticsearch and similarly there is a same step for the delete uh which is doing this the delete for the database and delete for elasticsearch and then there's another one for here for the update which is doing a sql update and there's another one doing another index as well now the search is not doing anything uh related to the persistent data store which will be postgres is literally just using the elasticsearch datastore so if we jump into the sql not sql the swagger ui you will notice if i create a new swagger i know daniel swagger a new task let's call it a new task and we can call it uh have a priority hi and they will notice that hey is the id whatever uuid if i use a word because the way it is the way that i'm doing it right now if you notice here is that i'm indexing the values that correspond to these this field which will be description priority is done and the data start and the date do i'm not currently using any of those fields for searching purposes i mean the dates but i will be doing that when i cover elasticsearch in more detail in the near future so what is going to happen is that i'm i'm going to be searching by a word a description when i'm trying to use the this new endpoint that i have here so if i call new and if i remove this tool for now i will show you how this work what is going to happen is that it will actually search for the for any task that's happened to have new in the description so in this case i created previously these two tasks that one of them is called brand new and the other one is that new task but the way it works is that in the current implementation is that if i look at the rest task endpoint which will be right here i'm receiving a we have some debugging debugging you know it's okay i have some uh a new model which represents the payload that i'm receiving in the request which happens to be all nullables so i can allow i can i don't have to pass in a description i have to pass in a priority and i don't have to pass in a is done option but still these options are going to be used when searching those properties or writers searching by those properties so if you notice in the service i'm actually just returning nothing in those cases when the values are nil but maybe you want to maybe return an error or something like that which you can specify or enforce those values when you're being receiving them so if i go back i want to show you this that this is done false and it's done false if i change the is done here and i say let's try true nothing will be returned but because nothing is done but if i try false i will you will see that the values are here right and i had another one that i created previously the important bit about elasticsearch is that um because it's so powerful the way uh depending on how you are indexing your records you can search by different options that like elasticsearch has an elastic search is massive there are a lot of options available one of the things i like doing is perhaps giving the transforming the words that you're receiving in your document or whatever you're trying to index and maybe transform those into something that the user could use for searching purposes an example would be numbers let's think uh maybe you need to the the user inputs in the description i don't know a number let's say 100 and maybe you can translate that 100 into literally 100 like the two words and the user when the user is searching one like o and e it can literally search by the description that happened to have the literally literal number one zero zero which is 100 or the word one so there are a few different transformations that happen when we're indexing the records in elasticsearch that allow allows us to give um more more options to the users when searching those records and not only that you can actually transform those and escort them depending on on how many records or how many uh values were available in the in the document that they were searching so perhaps you need to sort them hey i'm searching by a award and if the description in this case is the value that i'm using if that one is the maybe present x amount of times give them give that one the priority to the being displayed higher than the previous records so in the end elasticsearch is such a massive improvement that depending on what microservices microservices you're implementing it's most likely you're going to be having an option to search in records for searching records and elasticsearch is probably the best option available out there so elastic search is use a phenomenal tool i highly recommend it if you're planning to give your users an option for searching your records by any field that you have available not only that but you can transform those fields it allows you to uh it allows you to somehow analyze those data and maybe perhaps create some dashboards and whatnot it also allows you to create different ways to to to search different values so it not only supports by text or by you know using fuzzy words or or terms or whatever but it also supports things like a geospatial options or maybe um some more complex informations that you can need so in the end elastic search is such a powerful tool that i highly recommend you and as usual if you have any comments again the link will be the link to the repository to the repo to the code that will be in the description as well so please plea please feel free to feel free to check that out and i will talk to you next time any comment please any questions just let me know i'll talk to you next time take care

Channel: Mario Carrion

Views: 13,606

Keywords: golang, microservices, golang microservices, golang web development, golang tutorial, building microservices golang, golang microservices tutorial, golang observability, docker-compose golang, dockerhub golang, golang ecr, golang container, docker build golang, golang docker tutorial, golang beginners tutorial, golang shutdown, golang elasticsearch, golang logstash, golang elastic search, golang elk, golang searching, golang kibana

Id: ZrdbQRYst5E

Length: 14min 22sec (862 seconds)

Published: Fri Apr 16 2021