How to use Multiple Selenium Chrome Node with Selenium Grid with Docker compose & access in Python

all right uh hey guys and good evening one all and welcome to the video in this video i will be teaching you about selenium grid and selenium hub how you can use together with a docker container so you can spin up several instances of your selenium node or chrome node and use there i use the hubs ip address and do the scraping like a charm so what i mean by that let's see uh everything let's set up our selenium grid first of all so i'm here what i'm going to do is first of all i need to create a docker compose file so let me actually copy that so i'm in the selenium grid project new i'm going to click on file docker compose.yaml okay we have the yaml file which is pretty good now let's start start writing some code here okay so first of all we'll say uh we'll define the version of docker i'm saying that hey uh let's use the version 3.7 okay uh now what i want to do is basically i want to define my selenium node so let's define that so i'm saying a hub this is going to be a master hub which is going to listen to all the worker nodes so the image is going to be selenium hub container name automation hub i'm just giving a container name the i'm exposing port 444 444 right and now what i want to do is i want to have a worker node right so what i would do i have a worker node now okay so that's the worker node one and similarly i would have uh two more or three more worker nodes so total i would have four nodes okay that's the selenium nodes and they all depend on a hub okay so i'm using docker compose to spin up instances as you can see worker 2 so now let me collapse all of these and just show you how it looks like so as you can see i have worker 1 worker 2 worker 3 worker 4 and that's the hub that is going to drive the all the worker nodes okay so let's see how everything spins up should be pretty easy nah nothing important i mean nothing difficult i would say so cd what's the project name selenium grid okay let's do dir we have a darker compost file let's see darker compost so you give the command it would spin up all the instances for you it would spin up a hub which would monitor all the worker selenium nodes it's pretty amazing right i mean with the docker compose spin up all the infrastructure within seconds you know you could create multiple instances and stuff like that so uh let me show you what i mean by that okay so i'm gonna open chrome now uh that's my chrome now you want to head over to the ip address of 444 i guess that's the one so if you go to the 444 uh it would give you a dashboard if everything has started correctly uh hopefully i think it's still starting so here you can see selenium grid view so he allows you to monitor your multiple uh what you call that multiple worker nodes so you can see i have for four worker nodes right uh which has been configured other things what you could do i would show you some scraping also a very simple one but to you know kind of understand the concept so another thing i guess uh was something called status i guess you could also see whether the nodes are healthy or yeah hub status so you could run that command you return the json says true so you can scrape websites here let's take a very simple website you know i'm doing very simple i'm just showing you the project how everything you can build up the infrastructure right so you know let's say um let's just jokes i want to scrape some yo you want to scrape jokes basically every like whatever like every night you want to send this to some users you want to you know update people about it so you know it's easy guys it's a piece of cake um so all you got to do is copy the code we have jokes here of course so you know now if you want to write some selenium scripts i would as i said everything is in the description don't worry about anything code is there so focus on learning with me selenium scripts uh let's use this script i have here so if you see i do not need this this line right here i mean since i'm doing that there are two options basically headless headless means that do not run the chrome in the background now since we are using a docker container right uh let me comment out this one so now see uh what i'm saying is now this chrome is running in a docker container so all i have to do guys it's so amazing uh i would literally actually copy the ip address of this one i guess i guess that was how it was there i guess just want to make sure double check with on my uh code file on my master port i guess i have it so i was trying something with uh airflow as well let me just make sure the ip address okay it's w that's how we have to define it okay perfect no worries it's fine so now um okay so you would use hub and instead of how about say local oops okay and let's put that in the quotes okay so that's pretty much it right uh oopsie sorry for that so you have that in the codes now what you're saying is hey i wanna use a selenium driver which is there on a remote server i'm giving you the ip address right now it's a local host so uh since we have four instances he would do all the load balancing would route the traffic blah blah blah for you automatically let's grab the html for this website so i'm gonna grab the html for this website right here you know very easy selenium okay driver.get url uh and we're just printing printing the html um content of that so nothing fancy as i said you know so let's see if this works in in theory so we have these four worker nodes which are working for us so okay which file are we in uh i guess selenium scripts i guess yeah oh sorry oh not in grid buddy cd selenium scripts okay so now i would show you how by running remember i have the worker node is running here i'm running here now just the script.v1 python descriptor so it should open up the chrome uh driver execute uh chrome path is not defined okay one sec i know what's going on i should have commented on this one right here i don't need options for now sorry for that uh we can pass that later on it's fine so let's let's see if it works so now what's happening is it's opening the chrome on the docker container uh from all the four instances he would take one of the instances and at the same time if let's say if a second request came in right what he would do is he would route the traffic to the second grid so it would take case you want to make sure you have minimum like worker nodes are available so it's still running let's wait uh you can also come here and check if it's everything is okay on the grid looks like it's okay i'm gonna refresh this uh still true so it should be good i'm not sure if i have if i screwed up something here or not but i mean it's a very simple code via all we are doing is just connecting to the docker container here so uh wait there is something is everything okay localhost 444 wd hub i guess i mean oh yeah now we have it so we have the script data from that website so you can see it took a little while i was like ah i was like it it took a while it's fine so this is how you could uh you know run uh selenium web like selenium uh web whatever you want to call on our docker container let's just stop this otherwise it's gonna be like printing forever it's a user html so i'm gonna stop this cancel this out so yeah so you could easily connect to a docker you could easily connect to the selenium um which is running on a remote server just you use the ip address the port number wd hub and you know you can start your sweeping project here you can start scraping things out so as you can see it's pretty powerful uh use docker container use uh how many of our nodes you want you can do load balancing you can set up a swarm mode on this so that means uh if you have a swarm mode you you would have um you know you could scale up you could have you could have replicas for this so yeah that is how you do uh you know darker compose selenium selenium hub using with python um yeah so as as usual as i said the entire docker composer in the description section below the code for the basic script is also there in the description so kindly check it out and use it in your project thank you so much for watching see you guys in the next video keep smiling keep coding and goodbye

Channel: Soumil Shah

Views: 6,133

Keywords:

Id: J4BVTTBY4bk

Length: 8min 57sec (537 seconds)

Published: Wed Feb 03 2021