Monitor websites or servers availability with Heartbeat, Elasticsearch, Kibana setup in Windows

hi friends in this video we're going to talk about how to monitor the availability of websites or any devices with an IP address that support pinging using the elastic stack consisting of hardbeat elastic search and Keana all right let's talk about how these pieces fit together you know hardbeat is a lightweight shipping agent that monitors the websites or devices and sends that information to elastic search for storage so we take a machine and install hard bit in that and configure hard bit to monitor the availability of websites or devices then heartbit continuously monitors these devices and stores this availability information in the elastic search database so hotbit is the monitoring agent and it uses elastic search for the storage now the availability information is continuously being stored in elastic search since the data is in elastic search you can obviously use kibana to visualize and analyze the availability information present in elastic search so obviously using kibana you can create Dash boards view the raw logs of the heartbeat and do some analysis using kibana visualizations and you can even manage the index settings or the storage settings of the hardbeat index so this is how hardbeat elastic search and kibana can be used together to create a monitoring solution to monitor the up time of websites or devices so that's the architecture we are going to implement now now let's try to install heartbit first so heartbit can be downloaded and installed as a Windows background server is so you can just Google search download elastic search hardbeat or you can go to this link and this link will download the heartbe zip file and if you unpack the zip file you will find the heartbit folder with all the heartbit installation files you can copy this folder and paste it in C program files so now I have pasted my heartbit folder in C program files and I renamed it to be short like hardbeat 8.1.2 so now I have downloaded and installed hardbeat in my Windows machine now we need to to configure hard bit to connect to elastic search so obviously elastic search is the database which hard bit is going to use to store the data so if you don't know what elastic search database is and if you don't know how to install and configure elastic search and kibana in Windows I've already made a playlist on that and I will leave the link of this playlist in the description so I've got elastic search already up and running in my machine now I configure heartbit to connect to elastic search to do that in the heartbit folder there is a file called hard be. yml you can edit this file to configure hardbeat you can open it with notepad or vs code I choose to use vs code here because vs code has some syntax highlighting for yml files so in the yml file search for output. elastic search section so I got the section output. elastic search and in that write the address of elastic search so my elastic search is running on Local Host 9200 on https so I have written hosts as Local Host 9200 and protocol as https and I written the username and password of the elastic search database here and since I'm using the local certificate Ive kept SSL verification mode as none because elastic search is using a dummy https certificate but obviously if it's using a genuine https SSL certificate this is not required but in my case I had to keep SSL verification mode as none because I'm using Local Host to host elastic search so just by using this section you can configure the connectivity between hardbeat and elastic search so now save this file and close this hard bit. yml all right I have configured heartbit to connect to elastic search now for hard bit to store data in elastic search it needs to have an index template because index can be formed if there is an index template so to set up the index template of hardbat in elastic search you need to run a command so for that open a command line in the hardit folder so I'm going to open a command window here so now I opened a command window in hardbeat folder and before you run this command make sure that elastic search is running so I'm going to go to my services and I'm going to search for elastic search and I'm going to start this service actually so now my elastic search database is running now the command you need to run is heartbeat. exe space setup space minus E so basically it will set up the index template of hard bit in elastic sech so I'm going to go ahead and run this you can see index template setup is finished all right we have successfully set up the index template of heartbit in elastic search all right we have configured the connectivity between heartbit and elastic search but we did not configure the monitors till now right so each website or device you need to monitor is called a monitor the term you use is a monitor so hard bit can be configured with multiple monitors so let's try to create some monitors in heartbit so inside the heartbit folder you got a folder called monitors. D so just open this and here you got a lot of templates here and each monitor should end with a yml so in order to make it disabled they have kept an extension called do disabled so you can actually use this as a reference so let's try to create a monitor in this monitors. D folder so here I have created a blank text file called sites. HTTP yml remember the extension is not txt the extension is yml so for the sake of a demo let's try to create a HTTP monitor let's try to open it with vs code even open it with a notepad all right this is how I can define a simple HTTP monitor so the type would be HTTP the ID would be site one HTTP monitor in this case you can write any ID it should be a unique identifier and then give a readable name to that website and then keep enabled as true and give the schedule here the schedule is every 5 Seconds you know I want to monitor every 1 minute so I'm going to write 60 seconds and the host is the website you want to monitor you can write any website here for now I'm keeping the website as https example.com and you can decide if you want to do it in ipv4 or IPv6 and then the timeout for the request so obviously after 16 seconds the request will be terminated and it will be declared as unavailable so after you create a monitor here you can even create multiple monitors in the same file actually so this is a list in yml actually so this is the first list item you can just copy this and paste this here and then you can say something like site 2 HTTP monitor the name is site 2 and the schedule is every 60 seconds and let's try to actually ping Google here so google.com and save this and obviously you need administrative permissions to save the file so I'm ret drawing is admin so now I have configured two monitors in a single yml file but if you want you can even create a new yml file and create monitors in them that means you can create each yml file for each website or or you can have a group of websites in one yml file and group of websites in another yml file whatever you want ultimately in a yml file there should be a list item which will be a monitor all right for now I have created two HTTP monitors in a single yml file let's try to save this and close it so all right I have configured two websites in the yml file and now heartbit is ready to monitor two websites and ready to connect to elastic CS we have configured the connectivity and the monitors in hard bit now let's start to install the hard bit as a Windows service it's really simple actually just run the PS1 file install service hardbeat PS1 and if you have some restrictions on running the Powershell you can use this command powershell.exe execute policy and restricted file install service hard. PS1 so go to the heartbit folder and open a command window here and here run the command powershell.exe execution policy anded file install service hardit PS1 so run this and obviously it's going to ask run or not I can say run once by pressing R so it's saying permission denied because I did not use an administrative command prompt so let's try to open an administrative command prompt now so I'm going to say CMD run as administrator and now I'm going to go to the folder see program files hardbeat and now I'm going to paste the command and now let's run the command run once R and you have successfully created a service obviously I had a service already so it has deleted the service and created a new service actually you can see the status of the service just by writing SC query hardbeat because the name of the service is hardbeat and you can see the hardbeat service is present but it stopped currently all right just by a single Powershell command I was able to set up hadb as a Windows background service now let's run the hardbeat to store data in elastic search so I'm going to write SC start the name of the service is hard bit so I'm going to write hard bit and that's it hard bit is running now and now we should see data in elastic search to see data in elastic search you need kibana I've already installed kibana in my machine and it's running as a background service and I stopped it actually so let's going to start kibana again now kibana is started in my machine and now I can open kibana to access elastic search all right since my kibana service is running I'm able to see kibana in Local Host 5 601 let's set log to kibana with elastic user all right if everything goes good you can see your monitors in the observability tab so go to the observability tab and here you can see your monitors in the monitors section there are two monitors actually let's try to click on the show monitors and in the last 15 minutes it's going to show you the data you can even change this here for the last 7 days 30 days whatever and then refresh and here you got two mon s the name is site one and site 2 the URL is example.com google.com and you can see the graph here you can select only site one and this is the dashboard for site one so by default in the nework Iana versions under the observability tab you can directly see your monitors and their availability over time all right this is how you can easily view the availability information in the default observability tab of elastic search kibana so in the hard bit folder in the monitors. D folder I created sites HTTP yml file and in that I have kept my monitors but hey I have said we can even ping devices for the availability information instead of pinging websites so let's try to create an icmp monitor file so I'm going to create a new file so I created a new yml file and I would name it devices. ip. yml so let's try to open it with vs code so here let's try to create a Monitor to Monitor a device IP address so suppose you want to monitor a workstation in your Lan you want to see when it is up or down if the workstation supports the Ping the monitor will continuously ping the workstation and give you the availability information so I taken example IP address here and I'm giving the monitor type as icmp ID as my device name as device ping and schedule is every 300 seconds that means every 5 minutes it will ping the device for availability and the time out and wait and the standard stuff and since it's a list in yml file you can even add one more monitor here so I'm adding a monitor for device 2 with a different IP address and it will monitor every 300 seconds so let's try to save this and obviously I need to save as an admin and that's it now I configured hardbeat to monitor IP addresses using ping all right since I have actually added new monitors let's try to restart hardbeat service so I'm in the services app I've searched for the hardbeat service and I'll right click and stop and then right click and start now heartbit will include those two monitors also so again let's go to kibana and let's go to the observability tab and now I'm able to see four monitors obviously these two are down because I actually don't have that IP Series in my landan those were dummy IP addresses so obviously the status would be down you can see the status down has a different color so that you can easily monitor the downtime actually and you can even filter for particular monitors in the observability tab so if I want to search for only one monitor I can just write my URL here example.com and only one monitor is showing and this is the status of the monitoring now if you want to see the monitoring for the IP address just place the IP address click enter and you can see the only one IP address monitoring status here obviously you can change the time you can see for the last 7 days or the last 1 hour 15 minutes whatever you can even select custom time range just click on this click the absolute start time click the absolute end time and like like this you can even monitor for custom time range also I'm going to reset again to last 15 minutes so that's how you can easily use the observability tab in Kiana right away to monitor your monitors all right we have successfully installed hardbeat configured the monitors send the data to elastic search and used kibana to monitor the availability information interactively all right along with observability tab in kibana you can have some Community built dashboards for better visibility of the monitoring so the is this GitHub link called github.com elastic uptime CP and in this URL you can find the dashboards folder and here in the 7.x here you have a dashboard. ND Json so download this file and search for the saved objects in the elastic search kibana so I'm going to write saved objects and you got the kibana / saved objects and here import click on this and select the HTTP dashboard ndjs file and and then a lot of visualizations and settings will be imported into your kibana so remember after you install hardbeat and configure hardbat go to this GitHub repository and download this ndjs file and upload it into the saved objects so you'll get by default a lot of dashboards and visualizations imported into your kibana now let's try to see how the logs are stored in fact the index pattern is also conveniently imported from the ndjs file so you don't need to actually create index pattern also now let's try to see the raw logs on how the logs will be stored by heartbeat so I'm going to go to the left menu and inside the analytics tab go to discover you can even search in the search bar for discover and you can open the Discover page and in the index pattern select hardbeat star and now you can see the log stored by hardbeat in elastic search you can even change the timing from last 50 minutes to let's say last 1 hour and these are the logs being stored the the count of logs and here are the raw logs let's TR to zoom out so that I can see the list better and here this is the each log being stored let's expand a log and see what is stor each log is actually a Json so what hard bit does is it will monitor the site or IP address and create a log whether it's up or not and each log would be something adjacent like this here I will just search for a field name called summary. up so if the summary. up is one that means your monitor is up if the summary. up is zero your monitor is zero and you can see what monitor it is I can just search for monitor. ID you can see monitor. ID is site to http Monitor and monitor name is site 2 and it say even resolved the IP address if I search for the URL I can see the monitor was using the URL google.com so this way you can click on each document and see the detailed log created by heartbit in in Asic search and in this Keana discover page you can even see the logs like table or form I just search for a field summary summary. up click on this press button you can see the summary. up column coming here and let's try to add one more thing like monitor. name just click on this press button and you got the monitor. name and if you want to filter for only one site you can just hold on this and click on this plus button and a filter will be created here so only side two logs will be shown in list you can even click on this and edit the filter to actually change it to another monitor let's try to change it to device one ping and update filter now only device one pings logs would be visible so this way using the Discover tab of kibana you can see the raw lcks and do analysis but hey seeing the dashboard instead of raw locks would be more beautiful right so in the search bar just search for dashboards menu so I got my dashboard click on this and here since we have imported the ND Json file already we got a dashboard here hardbeat HTTP monitoring click on this and you can see useful information about each monitor being shown here so now if you want to see information about only my site 2 HTTP monitor I'm going to just over over this and click on the plus so that only side 2 HTTP monitoring information will be shown I can see right away what are the IP addresses it has been resolved to and I can see the whole monitoring status it's up all the time and see I'm seeing it for the last 1 hour I can even change it to the last 24 hours or whatever the time you want and you can see the plot since I don't have a lot of information it's showing blank now let's change this time range to last 15 minutes now you can see the up time here the green is up that means the whole time it was up actually and this is the pink times you can see the latency and all and you even have a HTTP duration heat map so you can see the distribution with time upon what is the latency of the site so in this way you will get an intuitive HTTP monitoring dashboard where you can monitor your monitors easily all right till now we have talked about how to set up heartbeat and how to see the data of heartbeat in our observability tab discover Tab and in the dashboards all right now let's talk about managing the hardbeat storage you know in elastic search the storage is basically done in indices so the data is stored in index so let's go to index management tab I'm going to search for index management and here you can see there are multiple tabs by default heartbit stores the data in a data Stream So if you don't know what an elastic data stream is I've already covered it in my elastic search introduction video so check it out for now let's go to the data stream Tab and here a data stream called hard bit is created already and if you click on that this is the index template and this is the life cycle policy and it has one indices under it let's try to click on that and here you see an index called heartbeat 8.1.2 time stamp Etc so this is the index and the data is being stored in this index and already 53 documents are stored in this index so the whole basis of index creation is index templates so let's TR to see the index templates here and search for hardbeat so there's an index template called hardbeat you can actually change the settings of your hardbeat storage Etc in this index template so let's try to click on this and let's try to go to manage edit and now you can change the index settings here so in the logistics tab I can choose whether to store the data in a data stream or a simple index so if you don't want a data stream which is immutable indices you can just turn it off and in the index settings tab you can actually change the whole settings of your index for example so if I don't want replicas since mine is a single node database I can just modify this Json and here I'm going to create a new thing called number of replicas and I'll mention it as zero now my index will not use replicas and at the end you can go to the review template and just save your templates now I can go to once again manage edit and go to the index settings and at the bottom I can see number of replicas zero so zero replicas will be created for my index and one more setting I want to mention here is if you choose not to use a data stream that means if you switch off this dat data string you'll get an error that roll or alas is not defined so in the index life cycle section you can go to the index life cycle section along with name just create a new attribute called rollover Alas and give it alas names I can just name it something like hardbeat 8.1.2 and then you can go to the review template and save the template again go to manage edit and in the index settings you can see roll over alas so that that's how you can choose not to have a data stream or you can choose some important index settings like replicas life cycle alas Etc in this edit index template tab all right you can even have data retention policy for your locks that means if you're running heartbeat for say one year and you want to return return only last 3 months you can automatically delete the data that can be possible using index life cycle policies so in the search bar just write life cycle and you can see index life cycle policies click on that and hard bit by default uses this hard bit index life cycle policy and here you can choose the data to be rolled over so there is something called hot phase warm phase and cold phase So based upon your settings your data would be rolled over from hot phas to warm phas to cold phase so here in the advance settings you already have a default if I just turn it off if each Shard size that means if each index size is more than 50 GB it will be r rolled over and a new index will be created or else if the age of the index is more than 30 days it will be rolled over and a new index will be created and then you have more settings like force merging that means sharts should be merged shrinking down sampling read only so these settings are required to optimize your storage and performance suppose I want to delete this data after a hot phase so I'm going to just click on the delete button and here you get the delete phase move data into this phase when it's by default 365 days you can just name it as 90 days that means after 90 days the data would be deleted so this way you can easily Define a life cycle policy so I mentioned delete phase 90 days old let's try to save this policy and now my data would be deleted after 90 days so it's that simple to implement a data retention policy so that's it guys we have used kibana to visualize the data and even manage the settings of the indices and also the data retention all right let's try to summarize how we set up the monitoring first we have installed and configured the heartbeat with the connectivity information and the monitors information and then we ran a command to set up an index template in elastic search and then we have started the heartbit service and then in kibana we have imported an ndjs file from GitHub and it has installed the default dashboards index patterns Etc and now you can conveniently see the dashboards you can even use the observability tab in kebana to see the up time of the monitors you can even see the raw logs generated by hardbeat in the Disco section you can use the index management page to actually change the settings of how the data will be stored by hardbeat and then using the index life cycle policy you can even Define the data retention of the hardbeat database so that's it guys this is how you can use hardbeat to monitor the availability of websites or devices you can see Ive created a block post on how to set up hardbeat in Windows for monitoring websites and servers I given you the images notes the commands the configurations so that you can copy paste and Implement in your own environment I've also given you the references to the official documentation so please check them out if you want to know deeper on how to configure heartbeat elastic search and kibana please ask questions or post your valuable feedback in the comment section hope you like this video guys thank you for watching peace

Channel: Learning Software

Views: 1,705

Keywords: learning, beginners, tutorial, server, elasticsearch, database, wnidows, setup, kibana, heartbeat

Id: sj4XF0ZRY-g

Length: 24min 16sec (1456 seconds)

Published: Sun Oct 15 2023